Elastic Robust Unlearning of Specific Knowledge in Large Language Models
Authors: Yize Sui
Jing Ren
Wenjing Yang
Ruochun Jin
Liyang Xu
Xiyao Liu
Ji Wang
Venue: NeurIPS 2025 poster ()
Keywords: LLM Unlearning; Preference Optimization; Unlearning Robustness
Primary area: deep_learning
TL;DR: A novel LLM unlearning optimization framework, namely Elastic Robust Unlearning (ERU), to efficiently and robustly remove specific knowledge from LLMs.

Abstract: LLM unlearning aims to remove sensitive or harmful information within the model, thus reducing the potential risk of generating unexpected information. However, existing Preference Optimization (PO)-based unlearning methods suffer two limitations. First, their rigid reward setting limits the effect of unlearning. Second, the lack of robustness causes unlearned information to reappear. To remedy these two weaknesses, we present a novel LLM unlearning optimization framework, namely Elastic Robust Unlearning (ERU), to efficiently and robustly remove specific knowledge from LLMs. We design the elastic reward setting instead of the rigid reward setting to enhance the unlearning performance. Meanwhile, we incorporate the refusal feature ablation into the unlearning process to trigger specific failure patterns for efficiently enhancing the robustness of the PO-based unlearning methods in multiple scenarios. Experimental results show that ERU can improve the unlearning effectiveness significantly while maintaining a high utility performance. Especially, on the WMDP-Bio benchmark, ERU shows a 9\% improvement over the second-best method, and maintains 83\% performance even under 1,000 sample fine-tuned retraining attacks, significantly better than the baseline method.

Elastic Robust Unlearning of Specific Knowledge in Large Language Models
Yize Sui', Jing Ren', Wenjing Yang', Ruochun Jin', Liyang Xu', Xiyao Liu", Wang
College of Computer Science and Technology. National University of Defense Technology
School of Computer Science and Engineering. Central South University
(suiyize18, renjing, wenjing senjinn-aang,, j jirr, xuliyang08, wj]@nudt edu. on 1xyzoewx@csu. edu. cn
Abstract
LLM unlearning aims remove sensitive or harmful information within the model, thus reducing the potential risk of generating unexpected information. However, existing Preference Optimization (PO)-based unlearning methods suffer two lim- itations. First, their rigid reward setting limits the effect of unlearning. Second, the lack of robustness causes unlearned information to reappear. To remedy these two weaknesses, present novel LLM unlearning optimization framework, namely Elastic Robust Unlearning (ERU), to efficiently and robustly remove spe- cific knowledge from LLMs. We design the elastic reward setting instead of the rigid reward setting to enhance the unlearning performance. Meanwhile, we incor- porate the refusal feature ablation into the unlearning process to trigger specific failure patterns for efficiently enhancing the robustness the PO-based unlearning methods in multiple scenarios. Experimental results show that ERU can improve the unlearning effectiveness significantly while maintaining high utility perfor- mance. Especially, the WMDP-Bio benchmark, ERU shows 9% improvement over the second- best method, and maintains 83% performance even under 1,000 sample fine-tuned retraining attacks, significantly better than the baseline method.
Introduction
With the rapid development of Large Language Models (LLMs) 13 their potential applications across various fields have increasingly become evident. This potential primarily stems from their extensive pre-trained knowledge base and exceptional generalization capabilities. However, due to the possibility that training data may contain copyrighted content, personal privacy information, and harmful speech among other undesirable elements (46), LLMs inevitably absorb some negative behavioral patterns during learning. These negative behaviors not only pose a threat to information security but may also have adverse social impacts, thereby hindering the widespread application of LLMs in real-world scenarios. Therefore, ensuring that LLMs are aligned with human values and intentions is crucial for maintaining their credibility and safety.
Since retraining models to remove specific undesirable content costly and time-consuming, re- searchers have explored methods including safe fine-tuning [7] and adversarial training (AT) [..9. However, recent studies on interpretability [[O], representation engineering [2]. and continual learning 513155 indicate that surface safety fine-tuning is difficult to ensure that LLMs are harmless in all scenarios, because this approach is not sufficient to fundamentally change the knowledge and capabilities of LLMs. Given that undesirable responses often stem from the same harmful knowledge
"Coneesonning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[6]. the concept of LLM unlearning has been proposed to directly remove such harmful knowledge from LLMs [7 T8. Currently, the basic method of LLM unlearning is to use the Gradient Ascent (GA) strategy on the forget set to realize knowledge unlearning by reversing the optimization process of gradient descent. Recently, inspired by Direct Preference Optimization (DPO) [19], Negative Preference Optimization (NPO) (20) treats the forget set as negative preference data, thereby as- signing a lower likelihood to unlearned knowledge. Unlike NPO, which only focuses on negative feedback, Alternative Preference Optimization (AltPO) (21) innovatively combines positive and negative feedback information to effectively solve the problem of decreasing model output quality.
Despite LLM unlearning has made significant progress in eliminating harmful behaviors in LLMs, it still exhibits notable limitations. Firstly, current PO-based excessively unlearning methods rely on rigid reward setting, which restricts the unlearning effectiveness to some extent [22]. Secondly, the unlearned models generated by existing methods often lack robustness, making them vulnerable to the threat of knowledge recovery. For example, unlearned models may be reactivated on unlearned dangerous knowledge by carefully crafted adversarial prompts 23 or contextual interactions (24) In
addition, Deeb and Roger [25] show that fine-tuning the unlearned model with only small number of samples can effectively restore the removed knowledge.
In this work, we propose a robust LLM unlearning optimization framework named ERU (Elastic Robust Unlearning). which aims to effectively and robustly remove specific knowledge and build trustworthy LLMs. Specifically, ERU adopts the elastic reward setting instead of the traditional rigid reward setting, achieving a more flexible balance between the reference-based reward and the reference-free reward. Subsequently, the ERU formulates the robust unlearning process max-minimum optimization problem, where the inner loop is carried out by simulating worst-case perturbations through refusal feature ablation [26]. and the outer loop focuses on removing harmful knowledge, We conduct extensive experiments on multiple LLM unlearning benchmarks such as RWKU (27]. MUSE (28), TOFU [29] and WMDP (30], covering LLMs such LLaMA-2-7B-Che 3 and LLaMA-3-SB-Instrue ERU framework.
and the experimental results fully verify the superiority of the
Our contribution can be summarized as follows: (1) We propose a novel robust LLM unlearning optimization framework called ERU, which can efficiently and robustly remove specific knowledge from L.LMs. (2) The elastic reward setting is designed achieve a more flexible balance between reference-baseee reward and reference- free reward, enhancing the unlearning effectiveness while maintaining the model utility. (3) We incorporate the refusal feature ablation into the unlearning process, which efficiently improves the unlearning robustness of the ERU. (4) Extensive experiments on multiple unlearning benchmarks and models show that the proposed ERU method has significant advantages in terms of unlearning effectiveness, utility preservation, and unlearning robustness.
2
Preliminary
In this section, we present the necessary background and closely related prior work. Additional related work is discussed in Section
2.1 Threat Model
We assume the existence of a unlearned white-box unlearned LLM of with weights 0, allowing adversaries to modify its weights or intervene in the activation space during inference. The adversary has full white-box access to 7 and attempts to recover the removed dangerous knowledge through specific methods under reasonable computational cost constraints. Our goal is to design robust unlearning method S such that the performance of π processed by S the forget objects is significantly reduced even if the ad adversary attempts to recover the forget objects, while ensuring that the utility performance of the model is not affected.
2.2 Problem Formulation
For a large language model ma trained on dataset D = (<,; vi) |i =1 1,2,.., N). we define the target distribution of knowledge to be unlearned as the forget set Dj C D. Machine unlearning aims to fine-tune ne to behave as it had only been trained on the retain set DR. DD D/Dr. Considering the high cost of retraining the model on DR from scratch, LLM unlearning has been widely studied

---

Rigid Reward Setting
Elastic Reward Setting
Refusal feature adversarial training
RFA (Pertuibttion)
log ('rll))
log (rlx)
log (
Elastic Unlearning
ERU
Robust Unlearned
Original LLM
LLM
Layer
Layer
Layer
Layer
Leyer
Figure 1: Overview of the ERU framework, which can implement robust LLM unlearning.
as effective approximate unlearning method, which only relies on Dj and a part of the retain set (D, S DR). The problem of LLM unlearning can be transformed into regularized optimization problem that balances the forget and retain objectives:
min E DED, [Br ær; 0) +A 1 (M (vr xzi 0)
(1)
Furgel loss
Retain loss
where & and f, represent forget and retain losses in generating the response y given the imput æ, and À a regularization parameter to balance them. Let Acep, (To) and AceD, (To) represent the inference accuracy of тy on Dr and Dr respectively. We expect unlearned LLM TT to exhibit low Accp, (T/) on Dj while maintaining high Acep, (m+) on D,. However, the goal of potential adversaries is to reverse low ACeD, (TT). Therefore, a robust unlearned LLM is able to resist threats from adversaries.
2.3 From PO to LLM Unlearning
Direct Preference Optimization (DPO). DPO serves as a key prerequisite incorporate LLM unlearning into the preference optimization framework. This method reparameterizes the following reward function r(z, y) through closed-form expression with the optimal policy instead of learning an explicit reward model:
(y x) Nrot (U x)
r(z, v) 3lggg
+ 3 log Z(x),
where Z(x) is the partition function, (y) 2²) represent the prediction probability of the LLM given the input-response pair (æt, 3) and Tref is the reference model. By integrating the reward formula into the Bradley-Terry ranking objective (32] the DPO can express the probability of preference data with the policy model rather than the reward model. Therefore, the objective of the DPO is:
log BBlg To (Yw x))
no (31 xx)
)]
CDPO (TO, Tret) =
(r. JDD -
Tref (yw x)) B Blo
(3)
Tref (w))
where a--) denotes the sigmoid function, B > 0 is the inverse temperature, and (x, Nar 91) is the preference pair consisting of prompts, winning responses, and losing responses in the preference dataset D.
Negative Preference Optimization (NPO). In the beginning, the LLM unlearning is implemented by gradient ascent (GA):
LGA(FO) -B(x,))~ [- log (No x))]]
prediction loss
this method always faces the risk catastrophic collapse of the forgetting process due to too fast divergence [20). To address this shortcoming, NPO make the preference optimization framework

---

applicable to LLM unlearning task. This method extends the DPO to treat forget data as negative sample. The loss function of NPO is defined as follows:
LNPO (To, Frof) (x [- log (-3
(y x Tyoe (y/ a
(5)
According to Equation and Equation (5), the gradient of GA and NPO can be expressed as:
VoLGA ED; V log To(y) x]],
(6)
V VoLNPO.S ED, [Wo(1 V log (o [=],
(7)
2n (vl=)
where (z y) n. +. can be interpreted as an adaptive smoothing weight. When the sample (vt, y) is already unlearned, we have Wa y) < 1, because of Te (y x) & Tyef (y x). so that ||VoNo,, < VooCalllz This indicates that NPO diverges more slowly than GA and thus has higher stability.
Rigid Reward Setting. In this paper, rigid reward setting is defined the of fixed and inflexible reward mechanism to guide the model's learning direction in the unlearning process of LLM. There
main types: reference-based reward and reference-free reward. The reference- based reward is based on an explicit reference model, which represents the initial state of the model before unlearning. By comparing the state of the current model with the reference model, the unlcarning degree of the current model can be measured. While the reference-free reward adopts constant offset y to replace the role of the reference model. In view of the many limitations exposed by rigid reward setting in practical applications (see Section for details), we have carefully constructed LLM unlearning framework based on elastic reward setting.
3 ERU: Elastic Robust Unlearning
In this section, we will explore in depth the proposed Elastic Robust Unlearning (ERU) framework. As shown Figure [ ERU is divided into two main parts. Firstly, we design an unlearning strategy based on elastic reward setting to overcome the limitations of rigid reward setting, Secondly, incorporate the refusal feature ablation into the unlearing process and transform the unlearing robustness problem (see Section I for details) into max- max-mmm optimization problem for solution.
3.1 Derive Elastic Reward Setting Objective
Revisit the Rigid Reward Setting, Start by revisiting the rigid reward setting. According to Equation 3,, DPO gives sastrennee-bbsee reward:
pref-based = log R
To (y x)
(8)
Tref (y x)
through comparison with the reference model. Assume (y x)) represents uniform distribution over the vocabulary for given input and replaces &eef (y x)) in the DPO loss function with it. In this way, DPO loss function can be simplified to:
cref-free (TO, U) = - - (2.00. [log a (3 (log Te (Ve xx) log (yı x)) >]]] (9)
where 7 3 (log U (yw I) log U (yi æ)) is a constant called target reward margin, proposed by B3) and achieves a transition from reference- braseedce-based reward to reference- free reward.
constant that controls the scaling of the reward difference. Consequently, this
Equation (9) can be viewed distributed and y 2 0. This ensures that the reward difference between the preferred response and the less preferred response is controlled only by the policy model ne. while making the following impossible:
special case of DPO where the reference model is uniformly
log Reef (ywr xx) log गeef (y) x)) < <,
(10)
thereby addressing the issue that the reference model weef may incorrectly distinguish between the preferred and less preferred responses. These two rewards still fall into the category of rigid reward setting.

---

Similar to the above derivation, for NPO, because the reference- sererence-based reward is the same as for DPO, it is also possible achieve the transition from reference-based reference-free reward. Based on the design idea that NPO treats the forget data as negative examples in DPO, the NPO loss function with reference-free reward can be expressed as:
Cref-free NNO
(To: U) = (x..)) [
2 log a (-8 log mo (y x))
(11)
where 7 2 the variation of target reward margin and at this point the form of y changes to:
7 = 3 (olog U (y x:)
(12)
Elastic Reward Setting. As discussed in Section B under rigid reward setting, relying solely on the reference model can lead to early-stage gradient weight smoothing ineffective, while full reliance on a constant offset may lose crucial instance distinctions. Therefore, we introduce the elastic reward setting to combine the ad adnattae of both and compensate for their shortcomings.
First, we redefine Tref as a joint reference model feef in the following form:
To(y x²) a Teet
nel (y x)) Uyy xx)
(13)
xz)
where a is hyperparameter controlling the influence of the policy model on the reference model.
When = 0, Tie is reduced to uniform distribution in the reference-free reward. When 1, takes into account the ratio between the policy model and the reference model as the NPO.
By substituting से into Equation detailed derivation.):
we get the following new objective (See Section D.I for
new (To, nee U)
log o (-3 To (y xx) Ree (3) x) )
(14)
(x log a (-3 log No (y) x) M)
where M is called the elastic reward margin and the formula is as follows:
Blog
To(yla) (e
M = //
(15)
The = 3- (y) x)) is the same reward margin parameter as Equation essentially a constant offset. Term 3 log (no (y x)/mrer(y) x)) measures the divergence in response pairs between the policy model ma and the reference model Tret, effectively capturing the instance- specific discrepancics. In order to avoid 3 log (T (y (e)/Tee(y) z)) dominating training due to scale variations, perform z-score normalization 34] on it, where μ² and o* its mean and standard deviation calculated over the training dataset. In addition, as a fixed reference during the optimization, the reference model should maintain gradients unchanged throughout the training process. For this purpose, we use the stop gradient operation to ensure, denoted as rg [:]. Combined with the above considerations and length normalization, the final loss becomes:
12). which is
LEu (no. Reet U) = E( [-3 log (u(z, y) rg [MT)
2
(16)
where u(z,y) v) = 2 log Toly
x). Equation (16). which called Elastic Unlearning (EU), can effectively slow down the rapid drop of gradient weights in the early stage caused by reference- reward (See Section [D.2 for theoretical analysis), and avoid complete reliance on reference-free reward. By incorporating the policy model and the reference model. The following sections will focus on how to enhance the unlearning robustness of EU.
Toi
elastic reward margin M, it considers the balanced influence between
5

---

3.2 Refusal Feature Ablation for Elastic Unlearning
Recent studies [35 36 have shown that introducing adversarial training in the LLM unlearning process can significantly improve the robustness of the unlearning method. However, existing methods often have high computational costs, which contrary to the original intention of LLM unlearning. Inspired by recent research on refusal feature (RF) [37 26]. we enhance the unlearning robustness of Elastic Unlearning through refusal feature ablation (RFA) to construct Elastic Robust Unlearning.
Refusal Feature Ablation. Research [26] shows that key mechanism of adversarial attacks is
to eliminate refusal features, making it difficult for the model to identify the harmfulness of the input. RFA simulates this attack behavior by directly eliminating the refusal features in the hidden representation. Following [37], given a collection of harmful prompts of tokens x = [t..... t,] E Dharmful and another set of harmless prompts
Daammlss the refusal feature of each layer E T. in L.M Ty is a one-dimensional feature linearly encoded in the residual stream, which is obtained by calculating the difference between the model's mean last-token residual stream activations h()) (x) when running harmful and harmless inputs:
Σ
(x)
(x)
(17)
HH
D Dhrmiul
D Phamieess
ZE ZED
Z DEDeniis
where Dharmful and Dammless by sampling 500 instructions from the AdvBench [38 and the Alpaca [39 datasets respectively. Furthermore, RFA is defined as an inference-time intervention that sets the refusal feature at each layer as its average activation on harmless prompts:
h (x)) - (x) ff (x)
-
Σ
At (x)
(18)
D hharmless
ZE
where f == is unit vector encoding the refusal feature direction, and h (x) - -f h (z) is
the projection that resets the value to zero along the refusal direction, and the last item is set patch the the refusal feature. In order to improve the unlearning robustness of the model, we next integrate RFA into the training process of EU.
Elastic Robust Unlearning (ERU). We describe adversarial training applied to LLM Unlearning as max-min optimization problem. This problem consists of an inner maximization process and an outer minimization process. The goal of the internal maximization process is to continuously increase the prediction probability of forgotten knowledge as much as possible through adversarial queries that may effectively bypass the model's 's limitations. The objective of outer minimization is to suppress the reemergence of forgotten knowledge, making its prediction probability as low as possible. In standard adversarial training with an -nom constraint of e, unlearned models will be trained be robust adversarial queries with perturbation ò in the input space:
min SninE (x.)) max (mo (xr 9,)) s.t. ||01|p S. c.
(19)
where is the unlearning loss function. To avoid generating adversarial samples through complex attack algorithms in the traditional adversarial training method, we take RFA an effective attack simulation during adversarial training.
We treat LLM with parameter the hidden representations that focus on the residual stream of the last token 6,, and go maps those hidden representations to output a probability distribution for sampling i.e., To (3) go H (z)). Unlike general adversarial training, refusal feature adversarial training (RFAT) is performed by simulating the perturbation of RFA:
combination of two functions, where H(x) = (h)))))
To (go o (H (x 8)). - To (go (H (x)
RHH)
(20)
Input-space pertarbation
Perturbations simulated by RFA
( (1) is the layerwise refusal feature that it needs to be dynamically updated
where Rни
[HH 1-1
every certain training steps, and H (z) Rни denotes the removal of refusal features across model

---

Table Overview of the unlearning effectiveness of various PO-based unlearning methods using the LLaMA-2- 2--B-Chha across four unlearning benchmarks. The best results are highlighted in bold, and the next best results are highlighted by underlining"
RWKU
MUSE-News
TOFU
WMDP
Method
FBy
Forget QA1
AAj
(Dy 4)
(Py
(E 591)
Privleak
ForgetD5-FQ
Forge11-FFQ1
AccBiol
AccCyber,
Original
51.9
46.8
57.5
-99.8
3.2e-16
2.1e-19
DPO
38.9
40.7
41.5
33.2
37.2
109.6
1.2e4
3,50-7
28.9
33.5
IDK
40.5
40.6
45.4
35.6
39.1
104.3
40-5
Sc-8
29.3
34.2
GA
44.5
39.6
47.3
0.0
20.8
0.05
8.1e-10
37.4
30.1
GradDiff
46.4
42.2
48.6
25.9
31.0
105.3
0.09
7.9c-3
38.6
33.5
GAgLR
41.4
44.3
-51.6
3.40-5
33.2
NPO
33.6
31.3
32.8
10.8
13.4
30.4
0.66
0.19
29.6
32.7
NPOGDR
34.8
34.7
38.1
101.3
0.44
0.24
31.8
33.0
NPO
37.6
34.5
38.5
16,6
38.6
-56.7
0.43
0.17
32.4
32.9
37.5
29.8
AdvNPO
35.9
33.2
25.2
13.7
12.8
24.6
0.63
0.26
29.9
33.2
FRU
27.1
25.5
0.73
28.4
ERUGDN
29.8
28.5
25.9
11.4
10.8
14.1
0.69
0.45
25.7
29.4
ERU KLR
31.4
27.9
27.5
15.3
0.67
0.42
27.5
30.2
layers. In an actual adversarial environment, attackers may not always succeed eliminating refusal features. Following [37], we perform RFA with probability to approximate the different degrees of adversarial perturbations encountered by the model during the training process, thus better simulating the adversarial attack scenarios in the real world. Therefore, REAT for EU to construct ERU can be expressed as:
min (a.)) max (pTg (ge HH RHH), (1 1) (ge (H (z)) .))
(21)
Following 26]. applied RFA the residual stream activations over the last 75% layer of each model to obtain the most stable fine-tuning results (see Section E.2.1 detailed settings).
Experiments
4.1
Experimental Setup
Datasets and Models. Our experiments cover unlearning tasks across four benchmarks: RWKU
27). MUSE MUSE 28). TOFU [29]. and WMDP 33) Among them, RWKU is real-world knowledge
unlearning benchmark specifically for LLM unlearning, while WMDP is designed to prevent LLMs from generating harmful content fields such biology, cybersecurity, and chemistry. For TOFU, we explore two unlearning scenarios, termed Forget05 and Forget 10, representing forget set sizes of 5% and 10%, respectively. For the MUSE dataset, primarily focus the unlearning scenario of news articles. The experiments are conducted based on the ----MAA---BBCCA- 8B-Instruct [31] More details are provided in Section E.1
( and LLaMA-3-
Evaluation Metries. Despite differences in the evaluation metrics of the aforementioned tasks, the evaluation framework can be summarized into the following three categories:(1) Unlearning Effectiveness, which measures the ability of unlearning methods to successfully remove the influence or behavior of specific data from the model. (2) Utility Preservation. which evaluates the performance of the unlearned model in executing standard tasks. (3) Unlearning Robustness, referring to the model's ability resist relearning or recovering the unlearned knowledge. The summary of the metrics on different unlearning benchmarks is shown in Table with detailed information available Section and Section
Baseline LLM Unlearning Methods. Researchers have proposed various efficient and practical
unlearning methods. To evaluate the performance of our proposed ERU framework, we select most representative preference aptimization-basee unlearning methods as baselines. First, we refer to the model without any unlearning as Original, which reflects the initial performance of the model across various tasks. In the selection of bascline methods, in addition to the GA, NPO, and DPO

---

mentioned in Section 2.3 we also included GradDiff' (a retention retention-regularizee variant of GA), the rejection-based unlearning method IDK, SimNPO 22 which relies on rigid reward setting, and the AdvNPO R5 which enhances unlearning robustness through latent-space adversarial training. Furthermore, we define new baselines (GLL NPO CDR. and NPOKLR) by integrating the GA and NPO methods with two regularizers. More details are presented in Section E..
4.2 Results of Unlearning Effectiveness Evaluation
The unlearning effectiveness reflects the extent to which unlearning methods remove specific knowl- edge from LLMs, so we aim for this metric to be as low as possible. Taalle]] shows the experimental results of various unlearning methods based on LLLMA-2-7B-CCM across Tour benchmarks. As can be seen from the table, cxisting methods generally underperform compared to ERU. Specifically, ERU achieves a 16.3% higher average metric than the second-best method (NPO) on the RWKU benchmark, and outperforms the second- method by 6% on the WMDP benchmark. This trend remains evident in the TOFU benchmark evaluation. It should be noted that the GA method has achieved the best performance in some metrics of the MUSE benchmark, indicating that it does not generate any text related to the forget set. This extreme performance comes at the cost of significant losses of model utility (as shown in Table 2). making the unlearned models generated by GA almost unusable. The PrivLeak metric in MUSE further demonstrates that existing methods generally suffer from over-unlearning or under-unlearning, while ERU performs best in this respect. In summary, ERU performs well in unlearning effectiveness, significantly superior to baseline methods. See Table 7 in Section F. for more experimental results.
4.3 Impact of Refusal Feature Adversarial Training
To evaluate the impact of refusal feature adversarial training on unlearning methods, we employ relearning attacks and various adversarial attacks to test their robustness (see Section E.2 for attack details). Specifically, we evaluate the unlearing robustness of different unlearning methods by comparing the changes in unlearning performance on the WMDP-Bio benchmark before and after the unlearned models are attacked.
For relearning attacks, given that the malicious can recover partially forgotten knowledge in unlearned models by fine-tuning irrelevant information, we fine-tune two types of data sets: (1) the retain set; (2) Wiki'ext, collection of documents Wikipedia that overlap least with dangerous knowledge. As shown in [ the experimental results show that fine-tuning with only 10 samples of the retain set can significantly restore the dangerous capability of most methods on WMDP-Bio, among which GA, DPO, NPO, SimNPO methods can restore 72.9%, 80.2%, 69.6% and 34.3% unlearning performance, respectively. Further fine-tuning with 1000 samples from the retain set, the dangerous capabilities of all methods except the AdvNPO were almost fully restored. In contrast, ERU recovers merely 8.1% % of its unlearning effectiveness when fine-tuned with 10 samples, and retains 83% of its unlcarning effectiveness even after fine-tuning with 1,000 samples, More detailed experimental results can be found in Section F.6
GA DPQ
NPO
SimiPo
AdVNPO ERU
Original
Orig PD
Post Attick
60
15
40
33
WHOP
30
25
13
Number of Retain set samples
0A
070
NP0
3ImPP0 AdVIPP
EHU
Figure 2: Knowledge recovery of different unlear- ing methods on the WMDP-BL -Bi after fine-tuning with different numbers of retain set samples.
Figure 3: Comparison of knowledge re- covery of different unlearing methods under Enhanced GCG.
8

---

For the robustness evaluation of adversarial attacks, we test the unlearned model with nine different types of attacks to induce it generate forgotten knowledge. Figure shows the degree of knowledge recovery of different unlearned models under Enhanced GCG attack, which based on an optimized set of adversarial input prefixes. Experimental results show that on the WMDP-Bio benchmark, the dangerous knowledge of most unlearned models is recovered by more than 50%, among which the DPO method has the most significant improvement, reaching 75.6%. In baseline methods, ANPO performed best, but its dangerous knowledge recovery rate was still 18.4%, compared to only 10.1 % for ERU. This comparison fully confirms that the ERU framework has significant advantages in terms of robustness. More detailed experimental results shown in Section
Table 2: Utility preservation of various PO-based unlearning methods on LLaMA-2-7B-Chal
RWKU (Uility Set)
MUSE-News
MMLU
TOrU-Forget05
TOFU-Forget10
Method
Rea7
Tru?
Faet
Flu"
KnowMemi 1)
Accuracyt
Probabiliyt
ROUGE?
Probability
ROUGET
Original
26.9
30.4
41.5
704.2
55.2
58.5
0.99
0.98
0.99
0.98
DPO
26.4
25.2
32.4
710.6
32.8
46.8
0.74
0.53
0.76
0.54
IDK
26.8
27.9
36.7
712.5
37.3
50.2
0.76
0.55
0.78
0.55
GA
25.8
30.7
40.2
207.6
48.3
0.00
GrdDDff
24.8
30.4
41.1
707.5
27.3
51.2
0.49
0.42
0.57
0148
GAKER
26.2
40.6
708.3
41.8
51.8
0.48
0.44
NPO
26.2
30.5
41.1
694.6
27.5
47.6
0.51
0.47
0.44
NPOcDa
26.5
30.4
40.8
705.2
40.5
51.7
0.56
0.55
0.65
0.53
NPOxER
31.2
50.5
0.56
0.54
SimNPO
26.3
29.4
40.5
691.3
43.5
50.2
0.56
0.54
0.72
0.53
AdvNPO
24.3
26.5
39.8
672.8
24.3
41.2
0.48
0.46
0.46
0.45
ERU
30.5
40.5
50.6
ERUGDR
26.1
30.7
41.2
707.5
47.2
52.1
0.72
0.57
0.79
0.56
KLR
40.9
4.4 Utility Preservation Analysis
effective unlearning method should keep the model as available as possible. The results in (how that all unlearning methods cause some degree of utility damage to the LLMs, with GA and AdvNPO leading to particularly significant model utility loss, the former rendering the unlearned model almost unusable. However, by introducing the utility regularization term, the utility preservation of these methods significantly improved, though still falling short of the original model. In contrast, ERU maintains high utility preservation even without the introduction of regularization terms, while the incorporation of regularization terms resulted in a better utility preservation of ERU.
Table 3: Comparison of training time of different LLM unlearning methods.
Method
NPO
SimNPO
AdvNPO
EU
ERU
Time (min)
156
92
539
125
243
4.5 Efficiency Analysis of ERU
Compared with the existing unlearning methods for robustness enhancement based on adversarial training. ERU has similar better robustness performance, but more efficient. This is because ERU does not need to perform complex gradient calculations and multiple iterative optimizations through dynamic simulation of attack algorithms (such PGD (40) find adversarial samples. Instead, directly performs ablation operations on the refusal features, significantly reducing additional forward and backward passes and greatly lowering the computational cost. Table the comparison of training time consumption of different unlearning methods in the same training environment in this paper. It can clearly seen from the table that the AdvNPO method significantly increases the time cost by enhancing the unlearning robustness of NPO through latent-space adversarial training. In contrast, the increase in time cost of the ERU method has been significantly reduced. See Section for further details.

---

5 Conclusion
In this paper, we propose the Elastic Robust Unlearning (ERU), a novel framework to enhance the unlearning capabilities of LLMs. Our framework addresses two critical limitations of PO-based unlearning methods: the rigid reward setting and the lack of unlearning robustness, By proposing the elastic reward setting, we achieve more flexible balance between reference-based reward and reference-free reward, which significantly improved the unlearning effectiveness while maintaining model utility. Furthermore, the incorporation of refusal feature ablation into the unlearning process significantly boosts the robustness of the unlearned models. Our extensive experiments across multiple benchmarks and LLMs have demonstrated the superiority of ERU in terms of unlearning effectiveness, utility preservation, and unlearning robustness.
Acknowledgments and Disclosure of Funding
This research is supported by the Natural Science Foundation of China (Nos. 62032024, 62372459), the Natural Science Foundation of Hunan Province, China (2024JK2015), the Special Foundation for Distinguished Young Scientists of Changsha (kq2209003), the Foundation of State Key Laboratory of High Performance Computing, National University of Defense Technology (202401-13).
References
[1] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint 230.::2300.00003. 2023.
[2] Aiyuan Yang. Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, al. Baichuan 2: Open large-scale language models. arXiv preprint arxiv 10305, 2023.
[3] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.092 2023.
[4] Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. arXiv preprint arXiv:2310.1377) 2023.
[5] Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. Unveiling the implicit toxicity in large language models. arXiv preprint (rXiv:2311.1739) 2023.
[6] Milad Nasr, N Carlini, J Hayase, M Jagielski, AF Cooper, D Ippolito, CA Choquette- Choo,, Wallace, F FTramer, and K Lee. Scalable extraction of training data from (production) language models (2023). arXiv preprint arXiv:2311.17035, 2017.
[7] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2rXhv:220..05862 2022.
[8] Daniel Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Benjamin Weinstein- Raun, Daniel de Haas, et al. Adversarial training for high-stakes reliability. Advances Neural Information Processing Systems, 35: 9274-9286, 2022.
[9] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman MU, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04244 2024.
[10] Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: case study on dpo and 2024.
toxicity. arXiv preprint
10

---

[11] Tianlong Li, Xiaoqing Zheng, and Xuanjing Huang. Open the pandora's box of Imss: Jailbreak- ing lims through representation engineering, arXiv preprint arXiv:2401.0622,, 2024.
[12] Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gunnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source Ilms through the embedding space. arXiv preprint 0rXi::2402.09063, 2024.
[13] Yun Luo, Zhen Yang. Xuefeng Bai, Fandong Meng, Jie Zhou, and Yue Zhang. Investi- gating forgetting in pre-trained representations through continual learning, arXiv preprint arXiv:2305.0596. 2023.
[14] Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference. arXiv preprint urXiv:239.101005 2023.
[15] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310. 16789, 2023.
[16] Zhexin Zhang, Junxiao Yang, Pei Kc, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlic Huang. Safe unlearning: A surprisingly effective and generalizable solution defend against jailbreak attacks. arXiv preprint 2rXiv:2407.02885, 2024.
[17] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li. et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.0878)) 2024.
[18] Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.10683, 2023.
[19] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[20] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404. 05868, 2024.
[21] Anmol Mekala, Vineeth Domn, Shreya Dubey, Abhishek Lalwani, David Koleezek. Mukund Rungta, Sadid Hasan, and Elita Lobo. Alternate preference optimization for unlearning factual knowledge in large language models. arXiv preprint arXiv:2409.13472 2024.
[22] Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang. Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for Ilm unlearning. arXiv preprint arXiv:2410.0716. 2024.
[23] Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from Ilms? objectives for defending against extraction attacks. arXiv preprint arXiv:2309. / 17410, 2023.
[24] Ila Shumailov, Jamie Hayes, Fleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Paper- not, Matthew Jagielski, Itay Yona, Heidi Howard, and Fugene Bagdasaryan. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai. arXiv preprint arXiv:2407.00166, 2024.
[25] Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? arXiv preprint (rXiv:2410.08827, 2024.
[26] Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda, Robust I1m safeguarding via refusal feature adversarial training. arXiv preprint arXiv:2409.2000 2024.
[27] Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint arXiv:2406. 10890, 2024.
[28] Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint 4rXiv:2407.06460, 2024.
11

---

[29] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121, 2024.
[30] Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.032. 2024.
[31] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv-2401) 2024.
[32] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, (9(3/4):324-455 1952.
[33] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
[34] S Patro. Normalization: preprocessing stage. arXiv preprint 1eXiv:/503.0644 2015.
[35] Hongbang Yuan, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Towards robust knowledge unlearning: An adversarial framework for assessing and improving unlearning robustness in large language models. arXiv preprint arXiv:2408. /0682, 2024.
[36] Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. Latent adver- sarial training improves robustness to persistent harmful behaviors llms. arXiv preprint arXiv:2407.15552 15549,, 2024.
[37] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models mediated by a single direction. arXiv preprint arXiv:2406. /717, 2024.
[38] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15045 2023.
[39] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: an instruction-folllwing llama model (2023), 2023.
[40] Aleksander Madry. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0660 060833 2017.
[41] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463-480. IEEE, 2015.
[42] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504, 2022.
[43] Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXiv:2310.20133 2023.
[44] Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023.
[45] Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. arXiv preprini 2rXiv:2309.11852. 2023.
[46] Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. Guardrail baselines for unlearning in 1lms. arXiv preprint arXiv:2403.033329 2024.
[47] Martin Pawelezyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arx.v:2310.07579, 2023.
12

---

[48] James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen, Offset unlearning for large language models. arXiv preprint 1111:22004.11045. 2024.
[49] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. 09999,9,,9,, 2019.
arXiv preprint
[50] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[51] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36, 2024.
[52] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu, Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
[53] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455, PMLR, 2024.
[54] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
[55] Jiwoo Hong, Noah Lee, and James Thome. Reference-free free monolithic preference optimization with odds ratio. arXiv e-prints, pages arXiv-2403, 2024.
[56] Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in 1lms. arXiv preprint arXiv:2402.16683. 2024.
[57] Jakub Lucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for ai safety. arXiv preprint arXhv:2409.1802. 2024.
[58] Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Jogging the memory of unlearned IIms through targeted relearning attacks. In Neurips Safe Generative Al Workshop 2024, 2024.
[59] Junkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. a-dpo: Adaptive reward margin is what direct preference optimization needs. arXiv preprint arXiv:2410.100 10148, 2024.
[60] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether thain-of-thought can solve them. arXiv preprint 1rXiv:2210.0926.1 2022.
[61] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falschoods. arXiv preprint arXiv:2109.0990 079558, 2021.
[62] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: (0tn::0555.3551 2017.
[63] Xuechen Li. Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of fnslloition-folloiinng models, 2023.
[64] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations gpt. Advances in Neural Information Processing Systems, 15:17359-17372. 2022.
13

---

[65] Yizhe Zhang, Michel Galley, Jianfeng Gao. Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maximization. Advances in Neural Information Processing Systems, 31, 2018.
[66] Yucheng Li, Frank Geurin, and Chenghua Lin. Avoiding data contamination in language model evaluation: Dynamic test construction with latest materials. arXiv preprint 1rXiv:2312.12343, 2023.
[67] Cew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pages 74-81, 2004.
[68] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschacdler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093-3106, 2022.
[69] Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos. Quantifying the privacy risks of learning high-dimensional graphical models. In International Conference Artificial Intelligence and Statistics, pages 2287-2295. PMLR, 2021.
[70] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.033000 2020.
[71] Ben Thompson and Michael Sklar. Firt: Fluent student-teachhe redteaming. arXiv preprint wxXw:24477.7447, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not need theoretical proofs and assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provide the detail experiment setting (e.g.. hyperparameters, comput- ing resource and training framework) in Section
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification: Due to some constraints, our code is currently not available for open access. Nonetheless, design is thoroughly discussed in
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//uuui guwde//ddee/ee/ wwt.:Suwwisssispoooooooooooooooi
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide a detailed description of the selection of hyperparameters, and the prompts feed into LLMs in Section E and Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: In this work, all our experimental results are averaged over multiple experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The detailed information on computer resources is shown in Section E.5
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///aarrpp...cppo/p//////aa
Answer: [Yes]
Justification: The research conducted in our paper conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: When our methods are used as intended and function properly, there is no negative social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all the papers associated with the datasets and codes used in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
The paper does not produce new assets such as datasets, but uses existing datasets.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We discussed the usage of LLMs in Section 4.1
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
21

---

Appendix
A Related Work
LLM Unlearning. The goal of unlearning is to make a trained model perform as if it were untrained on specific datasets by removing specific knowledge [41]. In the field of language models, research on unlearning has expanded into themes such as fairness, privacy, safety or hallucinations [42 43].
Due to the need for farge-scale computing costs, traditional unlearning methods are difficult to directly apply to LLMs, so a growing number of studies began to focus on the LLM unlearning
18 46 29). Yao et al. [18] were the first to define the setup and objectives of LLM unlearning. Current LLM unlearning methods are mainly divided into two categories: in-context learning-based unlearning and model optimization- based unlearning. The former, such labeled demonstrations or post-poe pesst-ssessingg filters, cannot completely remove specific knowledge from the model weights 47
18
45
488 46]. The latter minimizes correct predictions of forget objects through methods such as Gradient Ascent (GA) [8] In recent studies, Negative Preference Optimization (NPO) (20 has been proposed as promising approach. It treats the unlearning task variant of Direct Preference Optimization (DPO) [19) aiming to solve the above challenges more effectively. Our research committed to exploring the operational mechanism of PO-based LLM unlearning in depth, identifying the main limitations it currently faces, and making practical recommendations for improvement based on these findings.
Preference Optimization. The motivation for preference optimization is similar to that of LLM unlearning, which is to align LLM with human values. Viewing from the lens of preference opti- mization, we can better understand its connection with LLM unlearning. The core idea of preference optimization is derived from the reinforcement learning from human feedback (RLHF) 449 50), of
which direct preference optimization (DPO) is a typical example. As offline improvement of online preference optimization algorithms, DPO removes the need for an explicit reward model, thereby spurring the development of various reward- free offline preference objectives, including RRHF [51], SL.ic-HF (52), IPO
(53 KTO (54, ORPO 55 and SimPO 33 Recently, inspired by DPO, Zhang al. treated LLM unlearning a special case of DPO without positive samples, proposing the NPO (20). This not only establishes the theoretical connection between preference optimization and LLM unlearning, but also provides a new research perspective in this field. Following this research trajectory, we focus on improving treference-basee LLM unlearning methods. Different from the existing approaches that use rigid reward setting to assign reward values to different outputs during training, we propose the elastic reward setting that dynamically balances reference- reward and reference-free free reward, effectively combine the advantages of both and avoid their limitations.
Unlearning Robustness. Despite the effectiveness of LLM unlearning in removing harmful infor- mation from model weights, the resulting unlearned models tend to be fragile. Studies have shown that unlearned models are easily induced by adversarial prompts to regenerate deleted knowledge [35] (27. Lynch et al. (56) further found that knowledge can be extracted from both the unlearned model with high efficiency by analyzing the internal representation of the model. Furthermore. Lucki et al. (57 proposed a new method to extract potentially dangerous knowledge from unlearned models without updating the model weights. Hu et al. (58) show that access to only small number of possibly loosely related datasets can "wake up" the unlearned model and reverse the unlearning effect. These findings suggest that current LLM unlearning techniques require further improvement to enhance their robustness. Recent work (37 26 on refusal feature prove that refusal feature ablation can be used approximate the worst-case perturbation in adversarial training, and aim to extend it to the aspect of unlearing robustness to construct a robust LLM unlearing framework.
B Limitations of Rigid Reward Setting
Recent research 22 indicates that this reference-based reward setting may result in poor gradient weight smoothing. The reward formula NPO is defined as 33 log(mo (y Tre/ (y x)). Since the unlearned model is initialized identically to the reference model at the beginning of training, the
22

---

weight gradient is approximately equal to at this point (Figure 4a i..:
2 (y x))
W (2, v)
1.
(22)
To (3 x) Tee
This suggests that NPO behaves like GA in the early stages of unlearning, which may lead to over unlcarning even if the weight is reduced in the subsequent optimization process. Especially in the early stages of training, NPO tends lead to greater reduction in utility (Figure 4bp.
10a
1011
0.5
Model Utility
NPO's We
10"2
NPO
0.3
GA Retrian
10-3
10
0.1
2
4
10
9
Unlearning Epoch
Unlearning Epoch
(a) Trajectory of We for NPO over unlearning epochs.
(b) Model utility of NPO and GA across epochs.
Figure 4: Experimental phenomena of NPO and GA on forget05 dataset of TOFU, For reference-free free reward, a constant offset y is typically employed to substitute the role of the reference model. However, the limitation of this approach is that 7 remains unchanged across all training samples, failing to effectively capture the variability inherent between different data instances Completely discarding the reference model may lead to decline in model performance.
599.
C Limitations of Unlearning Robustness
While unlearning techniques effective at removing harmful knowledge from LLMs, their bustness may be weakened by methods similar to those used for safe training. For example, fine-tuning with just 10 unrelated samples can significantly reduce the unlearning ef- feet and restore the original performance of the model. In addition, malicious users may bypass restrictions with improved ad- versarial prompts to regain access to un- learned knowledge.
Original
Unleammed
Recovery
FTtttain
Taking the GCG attack 38 as an exam- ple, although the unlearned model ef- fectively defend against the GCG attack, the attack effect can be restored by small adjustments to the GCG loss function. Fig- 3 the knowledge recovery effects of six methods on unlearned models, and the results show that all methods can inval- idate unlearning to a certain extent. Cur- rent criteria for evaluating the effects of un- learning are often based on non-ad adversaraai scenarios and lack consideration of situa- tions where malicious users induce models this end, it is necessary to study new unlearn- ing methods in the adversarial context and incorporate robustness evaluation into the comprehensive evaluation system of unlearned models.
FTrForget
KL
Row
Echanced GCG
Figure 5: The unlearning robustness of NPO against knowledge recovery attacks and adversarial attacks by evaluating the accuracy on the WMDP-Bio benchmark. See (o.ction[E.2.]| for details on adversarial queries such as In-context Learning (ICL), Role Playing and En- hanced GCG.
to regenerate deleted knowledge.
23

---

D Theoretical Analysis
D.1 Derivation of ERU Loss.
The known joint reference model सee is defined as:
( (
x)) ( x
ce (y x)
(y
x)
(23)
ref
where a is the hyperparameter the influence of the reference model. Substituting कee into the loss function of NPO, new objective function is obtained:
Cnew (TT, Teef U) (x
2
log
(-8 log
To (v x) Tref (y/ σ))
(24)
B
Expanding कet further, we have:
[new (mo, Ree.. U)
TH (y x))
= -x
log a
-Blog
B
Are (y) x))
2
no (y) x²)
(25)
log a
-B
3
Uy [x)
me(y/=) Wert (vlaz)
2
To( x))
log
log (y ax)
- -- log U@V 1x) as
B
Tot (y) x))
By substituting Equation
12 we can get:
mno (ToS Ree U)
Toly x))
[z.g)>>
log o
-3 log To (y x)
3 log (vy (x) - a.
Teef (y x)
(26)
= -
log o
log To (y x)) / aB
Te x)) Tref (y)
In order to stabilize training and avoid dominant loss due to scale variations, apply Z-score normal- ization Equation
26):
new (Ta, Fee U)
To(y xx)
3 log a
-38log No (vl -33 Uty ()) - a.
Fret (3/ (x)
(27)
/ Blog
f4*
-aayy)-D
log
-Blog (v -
B
0*
D.2 Analysis of ERU training stability.
Equation (16) is the final loss function of the ERU method proposed in this paper, and its form is as follows:
[-2
2
log (u(z, y) rg [M]]
LEU (To, teet U) = B(Z,)))D,
(28)
where
B
u(x 3) = log To 2))
(29)
|yl
and M the elastic reward margin. Since uses the stop gradient operation rg [-]. the gradient of M is not updated during backpropagation,
[M] is treated as constant when calculating the gradient. We then analyze the gradient of the above loss function. First, to compute the gradient of
u(x y). we have:
Vou(a, 4)
Vo log ( x).
(30)
lyl
24

---

Define the variables:
B |y1
(x, v)
rg [M]
log Toly x)) rg [M].
(31)
Therefore, Equation
28 becomes:
[- log a (z)
(32)
B
Where a (.) is the sigmoid function and its derivative is o' (.) = (1) - Compute the
gradient of the Equation 32
with respect u(r;
2 [(x))
[- - (1 -1(1-( -0)))
2
Vu(x,y) D;
(z) - -(())
(ay))D
(33)
According to the chain rule, we have:
2
VoLEU = (a
[ o())) Vou(z.y)
(34)
Substitute Equation 30 into Equation 34
8
VoLEU (Z,,)),,
[ (1 -o(z)=
Va log Te (y x))
(35)
1
2- (1 o(2))
Ve log To (y x2)
We further transform the Equation (35
1
V VgLEU B(Z,))ED
- oz))
V log No ( 1x)
|y
(1
exp(2)
Vo log To x))
exp(2) +
|
1
B(z.))e
Ve log Te (3/ z))
exp(2) + 1
|y|
2 exp( -2)
Vo log To z²)
(36)
exp( ()) + lyl
2 exp( [v log ( (x) rg [M])
Vo log To x))
exp Fo log ( x)) + rg [M]) +
yyl
2 exp(rg (M]) (To m)) (6)
(x,)eD
Vo log TO xx)
lyd
exp(rg [M]) (T x)]
+
(x
(%, Vo log no(y x)]
where
- exp(rg [M] [To x)]
W/ y)
(37)
exp(rg [M] [TO (y [)] adaptive smoothing weight similar Wo (z, y) in NPO. It be seen that Wz y) distribution depends on the specific forget data sample, and it shows a stronger correlation with the response length ly]- This correlation effectively circumvents the situation described in Equation (22). thus ensuring that weight smoothing maintains its effectiveness in the early stages of the learning process.
E Experiment Details
E.1
Experimental Setups for the Unlearning Task
E.1.1 RWKU
RWKU evaluates the unlearning effect by the following metrics: (1) FB (Fill-intthe-Bland evaluates the memory ability of the model for unlearned target knowledge by filling in the blanks, and requires
25

---

the model to complete incomplete sentences based on the given context or facts. (2) QA (Question Answer), which evaluates the model's ability utilize knowledge in practical applications by constructing questions related to unlearned objectives. (3) AA (Adversarial Attack) evaluates the unlearning effect of the model the face of induction through nine different types of adversarial attacks, and tests whether the forgotten knowledge is easy to be reactivate. We use ROUGE-L. scores to measure the agreement between the model predictions and the true answers.Lower scores indicate better unlearning effects.
In addition, the evaluation metries of the benchmark for the utility performance of the unlearned model include the following aspects: (1) Rea (Reasoning Ability), which evaluate Big- Bench- Hard thouhht (COT) prompts with 3-shot examples and report the Exact Match (EM) score. (2) Tru (Truthfulness) evaluates whether the model becomes dishonest after unlearming with MCI [6 and reports 6-shot accuracy scores. (3) Fac (Factuality), which evaluates whether unlearning negates the original knowledge of the model by evaluating factuality on TriviaQA 62 and reporting the F1 score. (4) Flu (Fluency), which uses the instructions in AlpacaEval [63, and report the weighted average of bi-gram and tri-gram entropies [64 65 to measure the generation quality LLMs.
[60]. Take
The existing LLM unlearning methods usually need to construct a forget corpus, but since RWKU only provides unlearning targets, these methods can not be directly applied to RWKU. To this end, we synthesize the forget corpus by guiding the original model to generate text descriptions related to unlearning targets. The following shows specific prompt template and an example of the generated data.
[System]
(intro)
You know everything about (unlearning target).
[User]
Write paragraph {unlearning target).
Example 1: Prompt template that generate factual text descriptions related to unlearning target.
[System]
{intro}
You know everything about (unlearning target}.
[User]
Please generate question about funlearning target) based on your
rich knowledge [unlearning target}.
Example 2: Prompt template that generate question related to unlearning target.
Hyperparameter Setting. We conduct experiments on RWKU benchmarks based on LLaMA-2- 7B BB-Chat and LLaMA-3-8B--nsttmn
averaging results from 100 unlearning targets using single-target unlearning setting. The training epochs of all unlearning methods uniformly set to 3, and the learning rate is selected for each method in the range of le-8 8 to le-5 through grid search. Similarly, Our ERU conduct a grid search for in the range [0.5, 1.0] and a in the range [Se-2, 0.2]. We use AdamW with 20 step warm-up during training. Other hyperparameters strictly follow the study Settings of Jin et al. [27]
E.1.2 MUSE
MUSE involves two types of text data, news articles and books 44 The former is selected as the benchmark in this paper, which adopts BBC news articles [66] collected after August 2023. All articles are randomly into forget set D,, retain set D, and holdout set D,, In the MUSE benchmark, three key metries are proposed to evaluate the effectiveness of unlearning, namely VerbMem (no verbatim memory), KnowMem (no knowledge memory) and PrivLeak (no privacy disclosure). These metrics, from the perspective the data owner, evaluate that the model does not retain any information related to the forget set after unlearning.
26

---

VerbMem. This metric requires that unlearned model Ti should not exactly replicate any details from the forget set, in other words, text fragments generated by the unlearned model should remain significantly different from the content of the forget set. Specifically, by inputting the first tokens of a text forget set to the unlearned model, and the model is required to continue writing. The comparison between the generated continuations and the real continuations is then quantified using the ROUGE-L Fl score [67]. The formula is as follows:
VerbMem(z D)
Σ DEDr
ROUGE (Tj) (mnTa) 11111)
(38)
[Dr
where f (x(aa) is the model-generated continuation, and 1[+1; is the real continuation. Ideally, the unlearned model should produce something very different from the real VerbMem, which case the VerbMem score should approach O.
KnowMem. This metric requires that unlearned model z/ should not be able answer questions related to forget set after removing specific data. Specifically, by generating question-answer pairs related to forget set, and evaluating the ability of the model to answer these questions. The average of ROUGE scores of all question-ansee answer pairs is used to quantify how well the model remembers the forget set knowledge. The formula is as follows:
KnowMMem(Hd. Dr) = DD
ROUGE (zb (1).a) .))
(39)
(q.a)ED;
PrivLeak. This metric requires that the unlearned model should not leak any information about whether the forget set was ever used for training. Specifically, to accurately measure privacy leakage, a loss-based membership inference attack (MIA) method called Min-K% Prob [[5) is employed and the standard AUC-ROC score [68] 669 is calculated to distinguish members from non-members. The formula as follows:
AUC (T). Di, Db) AUC (ottrain Dr, Dn)
PrivLeak =
(40)
AUC [T petrrain ;D;, Dn)
where Tg retrran represents the LLM that is retrained by directly eliminating the forgotten set. Ideally, the unlearned model should have PrivLeak score close to O. If the PrivLeak score deviates significantly from O, it indicates that the model has privacy leakage problem. A significant positive or negative PrivLeak indicates over or under-unlearmi respectively.
Hyperparameter Setting. For MUSE-NewS, we use LLaMA-2-74 1-MMA-2-7B-CAA BBChat and LLaMA-3-8B- Instruct for our experiments. The original model is available directly from the benchmark. For ERU, we train 10 epochs at a learning rate le-5, Also, we conduct a grid search for the range [0.5, 1.0] and in the range [5e-2, 0.2]. The hyperparameters of other unlearning methods strictly follow the settings detailed by Shi et al. (28).
E.1.3 TOFU
TOFU contains information from 200 different fictional authors, each consisting of 20 question- answering pairs. Among them, part of the author information is defined as the forget set, which is the target of the model unlearning. The benchmark contains three unlearning configurations corresponding to 1%, 5%, and 10% of fictitious authors named TOFU01, TOFU0S, and TOFU10. pairs from the fictional authors. To assess
The retain set consists of the remaining guestion-answenn unlearning performance, we use the FQ (Forget Quality) metrics, which measures the unlearning effect by comparing the similarity between the unlearned model and the model trained only on the retain set. The core metric of Forget Quality is the Truth Ratio, which measures the ratio of the probability that the model generates a correct answer the probability that generates an incorrect answer when answering questions in the forget set. Specifically, the Truth Ratio is calculated by the following formula:
Appn EneAm P(a /)
Reruth
(41)
P(a ( 2/1al
27

---

where ä is rewritten version of the correct answer, and à is a set of similarly formatted answers that are actually wrong, both of which can be generated using GPT-4. Forget Quality is evaluated using the Kolmogonv-Smirnov test (KS test) to compare the Truth Ratio distributions of the two models: One an unlearned model, and the other is a model that only trained on the retain set. Because the retain model has never been exposed to the data in the forget set, it is consider the gold standard for assessing the unlearning effect. KS test calculates the p-value by analyzing the cumulative distribution function (CDF) of the Truth Ratio distribution of the two models. If the p-value is high (usually greater than 0.05), it indicates that the unlearned model and the retain model cannot be distinguished, that is, the unlearning effect is good. Conversely, low p- p-value indicates poor unlearning effect.
Hyperparameter Setting. For all baseline methods, we refer to the settings of the previous study 20). fixed the batch size 32, set the learning rate le-5, and fine-tune the target LLM by 10 epochs with the AdamW optimizer. During the training process, the first epoch uses linear warm-up learning rate, and the subsequent epoch uses a linearly decaying learning rate. We take LLaMA-2- 7B-Chat as the initial model and fine-tune it with 5 epochs on the TOFU to obtain the original model with learning rate setting consistent with the baseline method. For the NPO method, we perform grid search optimization for the hyperparameter in the range [0.05, 0.2]. The training parameters of ERU method are the same as those of NPO, but the grid search for parameter a is added the range of [5e-2, 0.2]. Other unlearning methods and evaluation processes strictly follow the experimental settings proposed by Maini et al.
E.1.4 WMDP
The WMDP benchmark contains 1,273 multiple-choice questions related to knowledge of hazardous biology and 1,987 multiple-choice questions related to cybersecurity. The benchmark uses a zero- shot question-answer format and determine the final answer by comparing the highest logit values for options A, B, C, and D. WMDP uses QA accuracy as a quantitative measure of unlearning performance, which we label AccBio and AccCyber in this paper. An example of multiple choice question is as follows:
Which of the following encoded factors help viruses neutralize host chemokines during infections?
B. Orf7a from Blan-Bacc Barr virus
A. BARF1 from Epstein-Bei
C. Nef from human immunodeficiency virus-1 (HIV-1)
D. Glycoprotein from human herpesvirus (HHV-5)
Answer:
Example 3: An example multiple choice question about biology.
In addition, unlearning method requires the elimination of dangerous knowledge while ensuring the retention of general knowledge. To assess whether the model retains general knowledge after implementing unlearning, we report accuracy on subject-specific areas in MMLU (70)
Hyperparameter Setting, For the WMDP benchmark, we select LLaMA 2-7B-Chat and LLaMA- the original model and perform the experiment only the WMDP-BA Bio and WMDP. Cyber subsets. During the experiment, we use the forget set consisting biosecurity and cybersecurity knowledge, and retain set of unrelated text. For NPO, a grid search in the range of learning rate [2.5e-6, 5e-6] and /3 [5e-2, 7.5), and set up 3 training epochs. The training parameters of ERU method are the same those of NPO, but the grid search for parameter is added in the range of [5e-2, 0.2]. The parameters of other ynlearning methods are according Lucki et al. [57].
plain text related to conduct
E.2 Experimental Setups for Unlearning Robustness
Since malicious users can access unlearned knowledge by circumventing restrictions, more rigorous adversarial query and relearning attacks need be considered when evaluating unlearning methods.
28

---

Therefore, we compare the unlearning performance of the unlearned model before and after ten unlearning recovery attacks on the WMDP benchmark evaluate unlearning robustness. This section details the experimental setup for refusal feature adversarial training of EU and the specific configurations of relearning attacks and adversarial queries used to verify the unlearning robustness.
E.2.1 Refusal Feature Adversarial Training
Table shows the hyperparameter settings we adopted when performing refusal feature adversarial training on the EU. Consistent with [26]. we only use 32 pairs of positive and negative samples each time the RF updated, and only train for epoch in each round of the max- max-miiimum optimization process to avoid overfitting of the model. Finally, we simulate the adversarial perturbation with probability p 0.5.
Table 4: Hyperparameters used for refusal feature ad adversarial training.
Hyperparameter
LLaMA 2-7B-Chat
Learning rate
2e-5
2e-5
LoRA rank
128
128
LoRA alpha
32
32
Gradient clipping
1.0
1.0
Batch size
32
32
RFA layers
[8,32]
[8,32]
RF update training steps
4
4
E.2.2 Relearning Attack
Fine-Tuning. We use finetuning to recover forgotten knowledge in the LLMs to evaluate the low-rank adaptive (LoRA) techniques to
robustness of the unlearned method. Specifically, fine-tune unlearned models on two datasets: (1) retain set. (2) WikiText, a collection of Wikipedia documents with the least overlap with hazardous knowledge. hyperparameters are follows: Epochs is 3, the learning rate is 2e-4, the learning rate scheduler linear, and the warmup ratio is 5e-2. The weight decay le-2 and the optimizer adamw. See Table 5 for other parameter setting.
Table 5: Hyperparameters used for finetuning for relearning attack.
Hyperparameter
settings
Target modules
all
Lora rank
128
Lora alpha
16
Gradient accumulation steps
Batch size
Maximum sequence length
1024
E.2.3 Adversarial Query
Enhanced GCG. The GCG method (38] does not perform well against unlcarning methods because focuses on finding adversarial prompt suffixes that can induce the model output a specific target string, which results in dangerous knowledge as prefix being detected by the unlearned model first. Once the unlearning method identifies context containing dangerous concepts, it introduces persistent noise into the residual stream, which prevents the model from recovering the unlearned knowledge. In contrast, Enhanced GCG (57] focuses on optimizing the adversarial prompt prefix, which is designed to prevent the model from recognizing dangerous knowledge at the initial stage. This method adopts a moving target to improve the internal representation loss in FLRT 71 For
details, please refer to the study of Lucki et al. [57].
29

---

Prefix Injection. Prefixing the question with some directive statement guides the model to follow specific instructions to answer the question.
[Original statement]
What is the capital city of France?
[Adjusted statement]
Please provide the capital France?
the following queation: What is the
Example An example of Prefix Injection.
Role Playing. Set the model to play a specific role, such scholar, expert, etc., and have it answer questions from the perspective of that role.
[Original statement]
What is the capital city of France?
[Adjusted statement]
As a historian, can tell me about the significance of the French Revolution?
Example 5: An example of Role Playing.
Reverse Query. Querying the unlearning target based on target-related information, ensuring that the answer is the target itself.
[Original statement] What is the capital city of France?
[Adjusted statement]
The Eiffel Tower is a famous landmark which European city?
Example 6: An example of Reverse Query.
In-context Learning. By inserting an example of question-and-answe pair relevant to the goal before the question, the model is guided to answer the question in the expected direction.
[Original statement]
What is the capital city of France?
[Adjusted statement] Q: In which country is the Eiffel Tower located? A: France.
Now, can you tell me capital France?
Example 7: An example of In-context Learning.
Synonym Manipulation. Replace key words with synonyms or near-synonyms in the question to test whether the model can still understand and answer the question.
[Original statement]
What is the capital city of France?
[Adjusted statement]
What is the seat of government of France?
Example 8: An example Synonym Manipulation.
30

---

Background Hint. Before asking the question, give some background information related to the target to provide additional contextual clues to the model.
[Driginal statement]
What is the capital city of France?
[Adjusted statement]
France, home to the famous Louvre Museum and known for its romantic atmosphere, has its capital city that is also a major fashion hub. What is the capital of France?
Example An example of Background Hint.
Affirmative Suffix.
Adding affirmative phrases after the question elicit positive answers.
[Original statement]
What the capital city of France?
[Adjusted statement] The capital city of France is Paris, isn it?
Example 10: An example of Affirmative Suffix.
Cross Lingual. Using language other than the main language of the model to pose the problem increases the difficulty and complexity of the problem.
[Original statement]
What is the capital city of France?
[Adjusted statement]
Quelle est la capitale de la France?
Example 11: An example of Cross Lingual.
E.3
Regularization Utility Preservation
Most LLM forgetting methods are not specifically designed to maintain model utility. To this end, some studies have explored regularization strategies aimed at improving the performance of retain set. These strategies mainly include gradient descent and KI. divergence minimization on retain set.
E.3.1 Gradient Descent on D, (GDR)
GDR directly trains the model to maintain the utility performance of the unlearned model by performing standard gradient descent optimization the retain set. The formalization is as follows:
min LGDR Lunteamn EEWD, [log (Io (3) xx)]]
(42)
where Lunteam the selected unlearning method. For example, GradDiff and IDK in the experiment combinatorial methods that combine GA/DPO methods with GDR, respectively.
E.3.2 KT. Divergence Minimization on DD (KLR)
KLR maintains the utility performance of the model by minimizing the Kullback-Leible (KL) divergence between the probability distribution predictions the unlearned model and the reference model on the retain set. The formalization is as follows:
min LKL Lunleam EzwD, [KL (Tref (: [x) || mo (+|x))]
[x)]]
(43)
In our experiment, we combine NPO and GA with KLR to form the combined baseline method N POKLR and GAKLE.
31

---

Table 6: Summary of unlearning effectiveness, utility preservation and unlearning robustness metrics
across different unlearning benchmarks. The arrows indicate the directions for better performance.
Benchmark
Unlcarning Effectiveness
Utiity Preservation
Unlearning Robustness
Rea (Reasoning Ability) f
FB (Fill-in-the- Blank) J QA (Question Answer)
Tru (Truthfulness)
RWKU
Fac (Factuality)
AA (Adversarial Attack) +
Flu (Fluency) f
VerbMem on D, 4
MUSE
KnowMem on D 4
KnowMem on D,
PrivLeak {E -56,5%)) 5%))
Forget0S-FQ f Forget1O-FQ f
TOFU
AccBio (accuracy biological) AccCyber (accuracy cyber)
AccBio (accuracy on biological) + AccCyber (accuracy on cyber) J
WMDP
Accuracy on MMLU f
E,4
Summary of the metries across different unlearning benchmarks.
Table summarizes the corresponding metrics for evaluating the three capabilities of the LLM unlearing method in the four benchmarks involved in experiments.
E.5
Computational Cost
Our experiment to evaluate the unlearning effectiveness, utility preservation and unlearning robustness were conducted with two A100 GPUs. The time consumption of the experiment is mainly affected by factors such the scale of model parameters, the size of the data and the length of a single sample. Overall, the time consumption is completely acceptable. Taking the TOFU benchmark with 5% forget size an example, the ERU training on 4 hours, which is significantly shorter than the retraining of the model. Without refusal feature adversarial training, the training time takes only 2 hour under the same conditions with only EU operations. It is particularly critical that ERU does not introduce any additional inference time cost.
Instructt takes approximately
F Additional Experiment Results
In this section, we present additional experimental results in detail. Specifically, Table reports more experimental results on unlearning effectiveness and utility preservation using RWKU benchmarks on the LLaMA 3-8t-Instruct model. Figure 6 shows the effect of different hyperparameter a values on ERU unlcarning effectiveness two different LLMs (LLaMA 2-7B-Chat and LLaMA 3-8B- Instruct). Table presents ERU ablation studies to analyze the impact of each component on system performance by removing key components in the ERU, such refusal feature adversarial training, elastic reward margin and length normalization operation. Figure 7 provides more results from the relearning attack experiment. These experiments evaluate the performance of various methods the WMDP- and WMDP-Cyber datasets under different number of sample fine-tuning Figure S shows the results of more adversarial attack experiments. These experiments cover variety of adversarial attack methods, such as Enhanced GCG, Prefxx Injection, Role Playing, etc., and evaluate the impact of these attacks on different methods the WMDP-Bio benchmark.
F.1 More Experiments Results about Unlearning Effectiveness and Utility Preservation
We further report experimental results on the (-emA-3--B--ae Instruct, using RWKU benchmark. focusing on unlearning effectiveness and utility preservation. As shown in Table 7 ERU always maintains the optimal performance in the forgetting effect. In terms of utility preservation, performs better than general baseline methods (such as GA, NPO) and is similar to regularized baseline methods.
32

---

Table 7: More experiments Inssttrttttt The best results highlighted in bold, and the next best results are highlighted by underlining " "
RWKU
Method
Forget Set QAL
Utility Set Fact
FB
AA
Rea
Trut
Flut
Original
85.7
73.2
75.3
42.1
36.2
53.4
706.2
DPO
46.9
38.7
41.5
41.3
32.8
26.2
715.8
IDK
48.5
40.6
50.4
42.0
34.9
46.2
719.3
GA
39.3
31.4
47.7
40.4
36.4
49.6
709.5
GradDiff
44.6
35.2
51.8
39.6
36.0
50.4
710.2
GAKLR
51.2
43.5
58.7
41.0
35.8
53.3
704.1
NPO
33.6
22.3
24.8
41.0
36.3
54.4
698.2
NPOGDR
37.8
24.7
28.1
40.3
37.1
51.8
708.3
NPOKLR
38.6
24.5
28.5
41.1
35.6
53.2
704.8
SimNPO
35.2
22.8
24.3
36.2
32.8
50.8
685.8
AdvNPO
33.4
23.6
19.7
40.5
35.4
48.2
679.3
ERU
31.2
21.1
18.5
41.5
36.2
52.4
710.5
F22 ERU with Different Value
In order to explore the influence of hyperparameter LLMs, and the results are shown Figure 6 When a is set to 0, the model loss degenerates to Equation (II.. With the increase of value, although there differences in the corresponding values when the two LLMs achieve the optimal performance, both of them significantly improve the unlearning effectiveness. This phenomenon confirms that the selection of hyperparameter has an important effect on model performance and needs to be optimized according to specific scenarios.
ERU, adjust its value two different
30
30
WMDD (iDD-Biotfccll))
28
10.1
Value
Value
(a) Experiment on LLaMA 2-7B-Chat.
(b) Experiment on LLaMA 3-8B Instruct.
Figure 6: The impact of different a unlearning effectiveness across two LLMs.
F.3
Ablation Studies
All key designs in ERU have important roles. To deeply analyze the influence each component on system performance, we perform ablation studies by removing key components from the ERU. As shown in T removing refusal feature adversarial training makes the unlearned model almost unable to handle retrain attacks as the previous methods. The removal of the clastic reward margin
33

---

decrease unlearning effectiveness by 15%. The experimental results fully verify the importance of each component ERU.
Table 8: Ablation study of various methods and their performance in terms of unlcarning effectivenes, utility preservation and unlearning robustness on LLaMA-2-7B-Chin RFAT and EMR represent refusal feature adversarial training and elastic reward margin, respectively. In ablation experiments with RFAT removal, fine-tuned retraining attacks below 10 samples unrelated to the unlearned target are used as the evaluation condition for unlearning recovery attacks. In the experimental results, the red value indicates the worst result.
Unlearning Effectivenes
Utility Preservation
Unlearning Robustness
Method
RWKU
MMLU Accuracyt
WMDP
FB4
QAL
AA4
AceBio
AccCyber,
Original
51.9
46.8
57.5
58.5
63.2
42.8
ERU
29.2
27.1
25.5
50.6
24.8
28.4
w/o RFAT
29.6
28.4
35.8
52.3
51.4
37.9
w/o ERM
33.4
31.2
32.5
51.5
26.5
30.4
F,4 Discussion the Robustness of ERU against Adaptive Attacks
We systematically weakened the ability of REAT in the following two ways to simulate the degree of damage this mechanism caused by different adaptive attacks.
Firstly, it is mentioned 3.2 that we use probability p to perform RFA to approximate the different degrees of adversarial perturbations encountered by the model during the training process. Therefore, reducing p will decrease the chance of the model being exposed to the "worst-case perturbation" weakening the robustness gain. Unlike setting p to 0.5 in the paper, set p [0.4, 0.3, 0.2] respectively here to weaken the ability of REAT.
In addition, in our paper, following the research of Yu et al. 26 we applied RFA to the last 75% layers (layers [8,32]) the model obtain the most stable fine-tuning results. Therefore, changing the layer where the RFA is applied will also weaken the robustness gain of REAT. We are in the following experiments respectively set of RFA application layer to [12,32], [16,32], [20,32], [24.32].
We discuss the performance recovery of various methods on the WMDP-Bio after fine-tuning with different numbers of retain samples. The experimental results shown in the following table.
Table 9: Experimental results on the robustness of ERU to adaptive attacks.
Method
0 Sample
Sample
10
Sample
Sample
100 Sample
250
Sample
500
Sample
1000 Sample
FRU(
29.3
32.3
ERU(==0.4)
24.6
28.6
29.5
30.2
31.8
34,3
36.5
38.8
ERU(g=0.4)
24.9
29.5
34.0
36,2
37.4
38.4
41.7
42.5
ERU(layers [12.32])
24.6
26.3
28.4
31.7
31.5
31.4
33.5
34,2
ERU(layers [16.32]
24.9
28.0
30.2
31.4
33.2
35.2
37.1
ERU(layers [20,32]
24.7
29.8
30.8
33.3
36.6
38.4
38.6
ERU(layers (24.32])
24.4
30.4
33.4
37.5
37.2
40.5
41.7
41.9
ERU
24.8
25.1
26.2
28.7
27.8
30.6
31.0
30.7
It can be from the experimental results in the table that ERU can still maintain certain degree of unlearning robustness after taking some special measures to break the protection to different degrees.
F.5 Statistical Significance Tests
To strictly evaluate the performance differences between the method we proposed and the baseline method, we conducted statistical significance test on the metrics in the three evaluation dimensions of unlearng performance. A statistically significant result, indicated by p-value less than 0.05, would confirm that performance improvement of our proposed methods meaningful and consistent. To compare these distributions, we employ the Wilcoxon Signed Ranks Test. We use bootstrapping to generate multiple samples from the original dataset through resampling with replacement. For each bootstrap sample, calculate both metries for both the proposed and baseline methods, resulting
34

---

in distributions of metric values for each method. After conducting the significance analysis, all p-value for the two models (LLaMA-2-7B-- MUSF-News, TOFU, WMDP) are significantly below 0.05 when comparing each of our proposed methods against the baseline methods.
across four datasets (RWKU,
F.6 More Relearning Attack Experiment Results
55
50
25
10
1000
1000
Number set samples
Wikkitext samples
(b)
orgw
46.000 0rgeal
VMMP (yarrArcCyorr)
WMDP- Syber(ArrCCybee)
3.8
38-
16
36
244
34
28
28
5
10
50
100
1000
10
500
1000
Number of Retain sot samples
Number of Wikitext samples
(c)
(d)
Figure More Relearning Attack Experiment Results.
35

---

F.7
More Adversarial Attack Experiment Results
(a) Enhanced GCG
(b) Prefix Injection
(c) Role Playing
53
UONDRR
(d) Reverse Query
(c) In-context Learning
(f) Synonym Manipulation
Doggnal
por Fi.-hicckk
gial
KPO
6.70
MP0
(g) Background Hint
(h) Affirmative Suffix
(i) Cross Lingual
Figure 8: More Adversarial Attack Experiment Results.
G
Limitations
Despite the significant improvements in unlearning effectiveness and robustness of the proposed ERU framework. future work needs to consider two limitations. First, the experimental validation is conducted on models with up to 8B parameters (e.g., LLaMA-2-7B-- Instruct). While these results are promising, the scalability and performance ERU on larger scale language models (e.g., 70B+ parameters) remains unexplored, behavior due their increased parameter size and complexity. Second, ERU assumes a white-bon box setting where adversaries have full access to model weights and activations. However, in practical scenarios involving black-box models (c.g., API-based LLMs), the applicability of ERU might be constrained, as its reliance internal feature ablation and gradient-basee optimization may not translate seamlessly. Addressing these limitations can further enhance the multi- functionality of the framework in actual deployment.
and LLaMA-3-8BB
larger models may exhibit different
36


Reviews:
[{"review_id": "CtgJLLRnXa", "reviewer": "Reviewer_gzkG", "summary": "This paper proposes Elastic Robust Unlearning. This method combines an \"elastic reward\" which regularizes the unlearning algorithm by adding a parameter to vary the influence of the reference model, with the technique of refusal feature adversarial training to make the model more robust to relearning and jailbreaking attacks.", "questions": "- Why is TOFU not included in the utility analysis in 4.4? It does not seem to appear in the appendix either.\n- The ablation study in table 8 is a useful study but the results are presented in a confusing way. Why is unlearning effectiveness evaluated on a different dataset compared to unlearning robustness? Ideally the ablation would be on a single consistent benchmark.\n- How do the regularized ERU models (in table 2) perform on the forget tasks (table 1)? This is key to understanding the forget/retain tradeoff - otherwise the comparisons are apples and oranges.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The method comprehensively covers unlearning effectiveness as well as robustness, incorporating recent techniques to improve performance on both.\n- The evaluation thoroughly covers several benchmarks that are currently state of the art.\n- The proposed method of an \"elastic\" reward is interesting and intuitive, to interpolate between the reference-based and reference-free setting.\n\nWeaknesses:\n- The results seem generally inconclusive. ERU does not do uniformly better than other methods across benchmarks, and as such it seems to be another point on the forget/retain tradeoff curve rather than pushing the Pareto frontier forward.\n- The elastic reward is interesting but the reasoning for it seems fairly heuristic. NPO itself is a version of regularized gradient ascent, and the elastic reward seems to be an alternative regularizer that incorporates the reference model. The arguments for why this should lead to better unlearning seem to be based on a fairly hand-wavy explanation of training dynamics (Appendix B). As such it's hard to predict when this method will be effective.\n- Some results seem cherry picked: why is ERU regularized not included in Table 1? Why is WMDP evaluated only on bio and cyber, but not chemistry? Why is TOFU not included in Table 2? Additionally, presenting forget and retain in separate tables rather than side by side makes it difficult to determine which methods perform the best.", "was_revised": true, "final_justification": "I appreciate the authors' response. Due to the concerns listed in my responses, I have maintained my score.", "rebuttal": "Dear reviewer gzkG\n\nThank you for a thoughtful and constructive review. We are pleased to hear your positive assessment of the novelty of our work and think our \"elastic\" reward is interesting and intuitive. We hope to address your concerns and questions in our response below.\n\n###  **Weaknesses:**\n\n> **Weakness (1)** : The results seem generally inconclusive. ERU does not do uniformly better than other methods across benchmarks [...]\n\nMost baseline methods (except advNPO) only deal with the two-dimensional trade-off of unlearning effectiveness and model utility (such as GA sacrificing utility for unlearning effectiveness). By jointly optimizing three key dimensions, namely unlearning effectiveness, utility preservation and unlearning  robustness, ERU achieves a coordinated improvement among them and advances the Pareto boundary. ERU has significant advantages in any dimension:\n\n`Unlearning Effectiveness`: The results in table 1 of the paper show that ERU significantly outperforms suboptimal methods on three of the four datasets (RWKU,TOFU,WMDP). Only in the MUSE-News dataset is it inferior to GA. However, it should be pointed out that GA achieves this advantage at the cost of significantly reducing the model's utility.\n\n`Utility Preservation`:  It can be seen from the results in table 2 of the paper that the original ERU outperforms the baseline unlearning method without regularization in most evaluation metrics, and some metrics even approach or exceed the methods enhanced by regularization. Taking the KnowMem metric as an example, ERU's score (43.2) surpassed $GA_{KLR}$ (41.8), $NPO_{GDR}$ (40.5), and was close to $NPO_{KLR}$(46.4). Particularly, when ERU also incorporates the regularizer, its utility maintenance ability is significantly improved, specifically manifested as the outstanding improvement of $ERU_{KLR}$ on the MMLU dataset and $ERU_{GDR}$ on the MUSE-News dataset.\n\n`Unlearning Robustness`: As can be seen from the results in Figure 2 and Figure 3 of the paper, ERU significantly outperforms the suboptimal method in terms of Unlearning Robustness. For example, under 1,000 samples of fine-tuned retraining attacks, ERU can still maintain 83% unlearning performance, while the suboptimal method advNPO can only maintain 72%.\n\nFurthermore, in terms of time efficiency, ERU saves approximately half of the training time compared to the advNPO method, which also falls within the category of adversarial training.\n\n> **Weakness (2)** : The elastic reward is interesting but the reasoning for it seems fairly heuristic. NPO itself is a version of regularized gradient ascent [...]\n\nThe elastic reward we proposed is not a heuristic design but a principle-based solution to address the limitations of the Rigid Reward Setting (Appendix B): reference-based reward can lead to instability in early training, while reference-free reward lose instance-specific signals.   Therefore, we redefined the reference model as the joint reference model, unifying the reference-based and reference-free rewards (Equation 13).   Then, after substituting it into the objective function of NPO, we can derive the objective function of EU (Appendix D.1).    We derived the adaptive adaptive smoothing weight from the objective function of EU through gradient analysis (Equation 37) and theoretically proved that it can avoid early-stage gradient weight smoothing ineffective (Appendix D.2).\n\n> **Weakness (3.1)** : Some results seem cherry picked: why is ERU regularized not included in Table 1? Why is WMDP evaluated only on bio and cyber, but not chemistry?\n\nThank you for your valuable suggestions! We promise All experiments follow established protocols from RWKU, MUSE, TOFU, and WMDP benchmark. Since our experimental setup on WMDP followed simNPO [1], we only covered bio and cyber. In order to address concerns, we now supplement the regularized ERU experimental results and the experimental results of the chemistry on WDMP benchmark as follows:\n\n|Method|RWKU|RWKU|RWKU|MUSE-News|MUSE-News|MUSE-News|TOFU|TOFU|WMDP|WMDP|WMDP|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n||FB↓| QA↓ |AA↓|VerbMem↓ |KnowMem↓ |PrivLeak |Forget05-FQ↑| Forget10-FQ↑|AccBio↓ | AccCyber↓|  AccChem↓| \n|Original|51.9 | 46.8|  57.5| 58.3| 63.7| -99.8| 3.2e-16| 2.1e-19| 63.2| 42.8|52.4|\n|DPO|38.9 | 40.7 | 41.5| 33.2| 37.2| 109.6| 1.2e-4| 3.5e-7| 28.9| 33.5|35.2|\n|IDK|40.5|  40.6 | 45.4| 35.6| 39.1| 104.3| 4e-5| 5e-8| 29.3| 34.2| 34.8|\n|GA|44.5 | 39.6|  47.3| **0.0**| **0.0**| 20.8| 0.05| 8.1e-10| 37.4| 30.1|31.4|\n|GradDiff|46.4|  42.2|  48.6| 25.9| 31.0| 105.3| 0.09| 7.9e-3| 38.6| 33.5|32.7|\n|GA_KLR|46.8 | 41.4|  44.3| 27.4| 58.6| -51.6| 0.11| 3.4e-5| 37.9| 33.2| 33.5|\n|NPO|33.6 | 31.3| 32.8| 10.8| 13.4| 30.4| 0.66| 0.19| 29.6| 32.7| 31.8|\n|NPO_GDR|34.8|  34.7|  38.1| 13.2| 48.6|101.3| 0.44| 0.24| 31.8| 33.0| 32.2|\n|NPO_KLR|37.6|  34.5 | 38.5| 16.6| 38.6| -56.7| 0.43| 0.17| 32.4| 32.9| 32.4|\n|SimNPO|34.2 | 31.8 | 37.5| 12.6| 11.3| 14.9| **0.97**| `0.45`| 28.6| 29.8|31.7|\n|AdvNPO|35.9 | 33.2|  **25.2** | 13.7 | 12.8 | 24.6 | 0.63 | 0.26|  29.9 | 33.2| 32.8| \n|ERU|**29.2**| **27.1**| `25.5`| `10.4` |`9.2` |**12.3**| `0.73`| **0.48** |**24.8**| **28.4**|**27.2**| \n|ERU_GDR|29.8| 28.6| 25.9| 11.4 |10.8 |`14.1`| 0.69| `0.45` |`25.7`| `29.4`|28.6| \n|ERU_KLR|`31.4`| `27.9`| 27.5| 11.6 |11.2 |15.3| 0.67| 0.42 |27.5| 30.2|`28.2`| \n\nFrom the table, we can see that although the regularizer will slightly hurt the ERU unlearning effectiveness of ERU (this trend is consistent with other methods), it can still maintain good performance. The best method is **bolded**, and the second-best method  in `highlight`.\n\n> **Weakness (3.2)** : Why is TOFU not included in Table 2? \n\nThank you for pointing out this oversight. We now supplement the experimental results of the TOFU dataset in the utility preservation dimension as follows:\n\n|Method|RWKU|RWKU|RWKU|RWKU|MUSE-News|MMLU|TOFU-Forget05|TOFU-Forget05|TOFU-Forget10|TOFU-Forget10|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | \n||Rea↑ |Tru↑ |Fac↑ |Flu↑| KnowMem↑|Accuracy↑|Probability↑|ROUGE↑|Probability↑|ROUGE↑|\n|Original|26.9 |30.4| 41.5| 704.2|55.2|58.5|0.99|0.98|0.99|0.98|\n|DPO|26.4 |25.2 |32.4|`710.6`|32.8|46.8|`0.74`|0.53|0.76|0.54|\n|IDK|26.8|27.9| 36.7|**712.5**|37.3|50.2|**0.76**|0.55|`0.78`|0.55|\n|GA|25.8| 30.7 |40.2 |707.6|0.0|48.3|0.00|0.00|0.00|0.00|\n|GradDiff|24.8  |30.4  |`41.1`|707.5 |27.3 |51.2|0.49|0.42|0.57|0.48|\n|GA_KLR|26.2 |29.8 |40.6 |708.3|41.8|51.8|0.48|0.44|0.53|0.49|\n|NPO|26.2 |30.5|`41.1`| 694.6 |27.5 |47.6|0.51|0.47|0.46|0.44|\n|NPO_GDR|26.5 |30.4| 40.8| 705.2|40.5|51.7|0.56|0.55|0.65|0.53|\n|NPO_KLR|26.3|**31.2**|40.9| 703.8|`46.4`|50.5|0.56|0.54|0.71|0.55|\n|SimNPO|26.3 |29.4| 40.5| 691.3|43.5|50.2|0.56|0.54|0.72|0.53|\n|AdvNPO|24.3 |26.5| 39.8 |672.8|24.3|41.2 |0.48|0.46|0.46|0.45|\n|ERU|26.2 |30.5| 40.5| 708.8 |43.2 |50.6|0.59|`0.56`|0.74|0.53|\n|ERU_GDR|26.1 |30.7| **41.2**| 707.5 |**47.2**|`52.1`|0.72|**0.57**|**0.79**|**0.56**|\n|ERU_KLR|**26.6** |`30.8`| 40.9| 708.6 |44.2 |**53.4**|`0.74`|**0.57**|`0.78`|`0.55`|\n\n\nFrom the table, we can see that the regularizer can consistently enhance the model utility of ERU and outperforms the suboptimal method. The conclusion is consistent with the description in the paper.\n\n> **Weakness (3.3)** :Additionally, presenting forget and retain in separate tables rather than side by side makes it difficult to determine which methods perform the best.\n\nSince our original intention was to discuss the performance of the unlearing method from three dimensions respectively, to avoid confusion, the corresponding experimental tables and images are also presented separately. To present the advantages of ERU more clearly, we will present the comprehensive results in the appendix.\n\n###  **Questions:**\n\n>  **Question (1):** :Why is TOFU not included in the utility analysis in 4.4? It does not seem to appear in the appendix either.\n\nPlease refer to our response to **Weakness (3.2)**.\n\n>  **Question (2):** The ablation study in table 8 is a useful study but the results are presented in a confusing way. [...]\n\nGiven that in the discussion of unlearing robustness in Section 4.3, we demonstrated the unlearing robustness of each method by evaluating  the change of unlearing effectivenes on WMDP, we have followed this metric here. We apologize for any confusion this may have caused. To address your concerns, we now supplement the experimental results of unlearing robustness on RWKU as follows:\n\n|Method|Unlearning Effectiveness|Unlearning Effectiveness|Unlearning Effectiveness|Unlearning Robustness|Unlearning Robustness|Unlearning Robustness|Utility Preservation|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n||RWKU|RWKU|RWKU|RWKU|RWKU|RWKU|MMLU|\n||FB↓|QA↓|AA↓|FB↓|QA↓|AA↓|Accuracy↑|\n|Original|51.9|46.8|57.5|51.9|46.8|57.5|58.5|\n|ERU|29.2|27.1|25.5|29.2|27.1|25.5|50.6|\n|w/o RFAT|29.6|28.4|35.8|44.4 (15.2↑)|39.6 (12.5↑)|46.8 (21.3↑)|52.3|\n|w/o ERM|33.4| 31.2|32.5|29.8| 28.6|26.1|51.5|\n\nFrom the table, we can see that after removing the refusal feature adversarial training (RFAT), the unlearing effectiveness of the unlearned model is significantly impaired (FB:29.2$\\rightarrow$44.4, QA:27.1$\\rightarrow$39.6, AA:25.5$\\rightarrow$46.8), which highlights the importance of RFAT.\n\n>  **Question (3):** How do the regularized ERU models (in table 2) perform on the forget tasks (table 1)? This is key to understanding the forget/retain tradeoff [...]\n\nPlease refer to our response to **Weakness (3.3)**.\n\n### **Concluding remarks.**\n\nWe would be grateful if you could let us know whether our explanations have addressed your concerns. Please let us know if you have any other questions or concerns.\n\n### **References.**\n\n[1] Fan , et al. (2024) Simplicity prevails: Rethinking negative preference optimization for llm unlearning."}, {"review_id": "JAcilCQyfm", "reviewer": "Reviewer_NnGx", "summary": "This paper introduces Elastic  Robust Unlearning (ERU), a novel framework designed to remkove specific knowledge from LLM more effectively and more robustly than the PO-based unlerarning algorithms like DPO, NPO, or the variants of NPO. They have two key designs: 1. the elastic reward setting, by using a reference model value combined between the uniforms distribution and the original model's output. This is in contrast with the rigid reward setting in prior works that either uses a uniform distribution as the reference value or use the original model's prediction probability. 2. They apply a refusal feature ablation (RFA)-based adversarial training procedure during the training process to simulate adversarial removal of the refusal featurein the model's hidden activation and formulate the robust unlearning as a max-min optimization problem. The experiments on multiple unlearning benchmarks, such as TOFU, WMDP, MUSE, show that ERU outperforms PO-based algorithms by a good margin.", "questions": "See the Strength and Weakness Section.", "limitations": "See the Strength and Weakness Section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper evaluates ERU across multiple axes—effectiveness, utility preservation, and robustness. Experimental results are favorable across multiple benchmarks compared to the most widely used methods, such as GA and NPO, or their variants. This is convincing and shows the effectiveness of the designed algorithm.\n\n2. The combination of the RFA-based adversarial training is novel and improved the robustness of unlearning algorithms. Incorporating RFA into the unlearning loop is a clever way to approximate worst-case adversarial perturbations without costly inner-loop PGD. By randomly ablating learned refusal features in the residual streams with probability p, ERU achieves robustness comparable to latent-space adversarial training but at a lower computational cost.\n\nWeakenesses and Questions:\n\n1. The novelty of this elastic reward setting: Given that ERU’s elastic reward formulation collapses to the rigid reference (only using uniform distribution or only using reference models) when alpha = 0 or alpha = 1, why should elastic and rigid reward methods be considered fundamentally distinct classes rather than simply two points on the same spectrum? In particular, since every rigid scheme can be recovered by fixing \nalpha at an extreme, what principled argument or theoretical insight supports treating ERU as a qualitatively new paradigm?\n\n2. You did some ablation studies about the effects of two main components of ERU: the elastic reward design and the RFA-based adversarial training in the appendix, but this ablation is only on the RWKU dataset for comparing the effectiveness. I wonder how they perform on the other three datasets.\n\n3. Figure 6 (PS: It does not have a caption) shows the effect of alpha and shows that setting alpha away from zero will improve the performance. It seems that the optimal alpha highly depends on the model (what about across datasets? What are the optimal alpha?) Is there anyway to explain this? Do you have any more intuition about what the hyperparameter alpha is and what it controls? And the most importantly, in what case would you expect a larger optimal alpha?", "was_revised": true, "final_justification": "Thanks the authors for their responses in detail. I read your response and I really appreciate it, especially for clarifying your contribution in more detail and providing more ablation studies to show the strength of the method, so I raise my scores. I also read other reviewers' responses and find them helpful.", "rebuttal": "Dear reviewer NnGx\n\nThank you for your positive evaluation of  the novelty of our work and the comprehensiveness of our experiments. We are pleased to address your concerns.\n\n###  **Weaknesses and Questions:**\n\n> **Weakness and Question (1)**: The novelty of this elastic reward setting [...] why should elastic and rigid reward methods be considered fundamentally distinct classes rather than simply two points on the same spectrum? [...]  what principled argument or theoretical insight supports treating ERU as a qualitatively new paradigm?\n\nThank you for your profound insights into the elastic reward settings in the ERU. You have correctly noticed that when α = 0, the elastic reward setting of ERU has been transformed into reference-free reward. However, what we need to clarify is that when α = 1, the elastic reward setting of ERU does not completely collapse into reference-based reward. At this time, the influence of uniform distribution $U (y | x)$ still exists (Equation 13 of the paper). \n\nTherefore, what we want to emphasize is that the elastic reward setting of ERU is not merely a simple hyperparameter interpolation. Instead, starting from the limitations of the rigid reward setting (Appendix B), and addressing these limitations by dynamically balancing the complementary advantages of reference-based reward and reference-free reward (Appendix D.2). The elastic reward setting is precisely the new category defined to distinguish this rigid setting. ERU converts rigid reward setting into a continuous and adjustable spectrum, in which α controls the degree of influence between reference and non-reference components, a paradigm that was lacking in previous work.\n\n*****\n\n> **Weakness and Question (2)**: You did some ablation studies about the effects of two main components of ERU [...] I wonder how they perform on the other three datasets.\n\nThank you for your insightful observation regarding the scope of our ablation studies.   We appreciate the opportunity to address this limitation and provide comprehensive cross-dataset ablation results below:\n\n`Ablation results of Unlearning Effectiveness`\n|Method|RWKU|RWKU|RWKU|MUSE-News|MUSE-News|MUSE-News|TOFU|TOFU|WMDP|WMDP|WMDP|\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n||FB↓| QA↓ |AA↓|VerbMem↓ |KnowMem↓ |PrivLeak |Forget05-FQ↑| Forget10-FQ↑|AccBio↓ | AccCyber↓|  AccChem↓| \n|Original|51.9|46.8| 57.5| 58.3 |63.7 |-99.8| 3.2e-16| 2.1e-19 |63.2| 42.8|52.4| \n|ERU|**29.2**|**27.1**| **25.5**| `10.4` |**9.2** |**12.3**| **0.73**| `0.48` |**24.8**| **28.4**|**27.2**| \n|w/o RFAT|`29.6`|`28.4`| `35.8`| **10.2** |`9.3` |`21.7`| `0.71`| **0.49** |`26.5`| `28.3`|`29.8`| \n|w/o ERM |33.4|31.2| 32.5| 12.3 |11.1 |18.4| 0.64| 0.44 |28.2| 29.4|30.5| \n\n`Ablation results of Utility Preservation`\n\n|Method|RWKU|RWKU|RWKU|RWKU|MUSE-News|MMLU|TOFU-Forget05|TOFU-Forget05|TOFU-Forget10|TOFU-Forget10|\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| \n||Rea↑ |Tru↑ |Fac↑ |Flu↑| KnowMem↑|Accuracy↑|Probability↑|ROUGE↑|Probability↑|ROUGE↑|\n|Original|26.9 |30.4| 41.5| 704.2|55.2|58.5|0.99|0.98|0.99|0.98|\n|ERU|`26.2`|`30.5`|`40.5`|`708.8`|`43.2`|`50.6`|`0.59`|`0.56`|`0.74`|`0.53`|\n|w/o RFAT|**27.1**|**32.1**|**41.5**|**710.5**|**44.3**|**52.3**|**0.62**|**0.57**|**0.76**|**0.54**|\n|w/o ERM|25.7 |29.8| 40.2| 709.8 |42.5 |51.5|0.57|0.56|0.72|0.53|\n\n`Ablation results of Unlearning Robustness`\n\n|Method|RWKU|RWKU|RWKU|WMDP|WMDP|\n|:--|:--:|:--:|:--:|:--:|:--:| \n||FB↓|QA↓|AA↓|AccBio↓|AccCyber↓|\n|Original|51.9|46.8|57.5|63.2|42.8|\n|ERU|**29.2**|**27.1**|**25.5**|**24.8**|**28.4**|\n|w/o RFAT|44.4|39.6|46.8|51.4|37.9|\n|w/o ERM|`29.8`|`28.6`|`26.1`|`26.5`|`30.4`|\n\nBy analyzing the above three tables, it can be seen that after removing the two core components of ERU, it shows significant differences in the three dimensions of unlearning (Unlearning Effectiveness, Utility Preservation, Unlearning Robustness). The best method is **bolded**, and the second-best method in `highlight`.\n\nSpecifically, removing RFAT does not significantly affect Unlearning effectiveness, but improves utility Preservation(Rea: 26.2 $\\to$ 27.1, KnowMem: 43.2 $\\to$ 44.3), but causes a near collapse of robustness (FB:29.2 $\\to$ 44.4, AccBio:24.8 $\\to$ 51.4).  This indicates that there is an inherent contradiction between adversarial training and model utility, and seeking a balance between them is the key to construct a robust unlearning mechanism.\n\nThe removal of the elastic reward setting directly weakened the Unlearning Effectiveness (FB: 29.2$\\to $33.4, QA: 27.1$\\to $31.2), confirming the core supporting role of this component for Unlearning Effectiveness.  At the same time, due to the failure of early-stage gradient weight smoothing (see Appendix D.2 for details), its removal also impairing the performance of Utility Preservation.\n\n*****\n\n> **Weakness and Question (2.1)**: Figure 6 (PS: It does not have a caption) shows the effect of alpha and shows that setting alpha away from zero will improve the performance.\n\nThank you for the reminder. To clarify, Figure 6 illustrates the impact of different $\\alpha$ (the hyperparameter controlling the influence of the reference model in our elastic reward setting) on unlearning effectiveness across two LLMs: LLaMA-2-7B-Chat and LLaMA-3-8B-Instruct.\n\n*****\n\n> **Weakness and Question (2.2)**: It seems that the optimal alpha highly depends on the model (what about across datasets? What are the optimal alpha?) Is there anyway to explain this?\n\nYour observation that the optimal α depends on the model is correct and consistent with our findings. We explain this below.\n\nAccording to the loss function of NPO (Equation 5 in the paper), as unlearing proceeds in the expected direction, the prediction probability of the current model $\\pi_\\theta$ for the knowledge to be forgotten will continue to decrease and gradually deviate from the prediction probability of the reference model $\\pi_{ref}$ for this part of the knowledge. This is the basic principle of reference-based reward for unlearing. Correspondingly, the reference-free reward(Equation 12 in the paper) directly replaces the \"ruler\" role of the reference model in a rigid way by uniformly distributing $U (y | x)$.\n\nThe parameter $\\alpha$  in the elastic reward setting we proposed is precisely the valve that balances and regulates these two \"Ruler\". When $\\alpha$ decreases, the elastic reward setting tends towards reference-free reward. Conversely, it tends towards reference-based reward. Therefore, if users have sufficient trust in the reference model and believe that it can accurately provide a large enough prediction probability for forgetting knowledge during the unlearing process, it is beneficial to appropriately increase $\\alpha$ to bias the reference-based reward. This is why in **Figure 6(b)** of the paper, with the increase of $\\alpha$, the Unlearing Effectiveness is better (with a lower accuracy rate in the forget set). On the contrary, when the reference model is incompetent (LLaMA 2-7B-Chat vs. LLaMA 3-8B-Instruct), it is necessary to reduce reliance on the reference model and lean more towards reference-free rewards (Figure 6(a) in the paper).\n\nIn addition, your concern about the influence of different datasets on the optimal α is also worth delving into. We have supplemented the experiments on multiple benchmarks as follows:\n\n`LLaMA-2-7B-Chat`\n|Method|RWKU|RWKU|RWKU|TOFU|TOFU|\n|:--|:--:|:--:|:--:|:--:|:--:|\n||FB↓| QA↓ |AA↓|Forget05-FQ↑ |Forget10-FQ↑| \n|α=0.00|33.8| 31.1|35.6| 0.92 |0.47 |\n|α=0.02|32.0| 30.8|31.8| 0.85 |0.45 |\n|α=0.05|29.2| 27.1|25.5| 0.73 |0.48 |\n|α=0.10|30.9| 27.4|27.3| 0.69 |0.46 |\n|α=0.15|31.3| 28.1|28.5| 0.68 |0.42 |\n|α=0.20|31.5| 28.2|26.7| 0.65 |0.45 |\n\n`LLaMA-3-8B-Instruct`\n|Method|RWKU|RWKU|RWKU|TOFU|TOFU|\n|:----|:--:|:--:|:--:|:--:|:--:|\n||FB↓| QA↓ |AA↓|Forget05-FQ↑ |Forget10-FQ↑| \n|α=0.00|33.1| 23.4 |23.1| 0.62| 0.49 |\n|α=0.02|33.4| 22.6 |21.6| 0.59| 0.48 |\n|α=0.05|32.2| 22.8 |20.9| 0.68| 0.50 |\n|α=0.10|32.5| 22.4 |19.2| 0.73| 0.49 |\n|α=0.15|31.8| 21.6 |19.4| 0.78| 0.52 |\n|α=0.20|31.2| 21.1 |18.5| 0.84| 0.56 |\n\nFrom the table, we can see that the trends of most datasets are basically consistent with the aforementioned analysis. However, there are differences in specific instances (such as Lama-2-7B-chat in RWKU), which may be due to the high prediction probability of the model for forget knowledge in this dataset. Therefore, we conclude that although both the model and the dataset can influence the optimal value of α, the essence still depends on the prediction accuracy of the model for forget knowledge.\n\n*****\n\n> **Weakness and Question (2.3)**: Do you have any more intuition about what the hyperparameter alpha is and what it controls? And the most importantly, in what case would you expect a larger optimal alpha?\n\nAs we replied in **Weakness and Question (2.2)**, when the reference model can output a relatively high prediction probability for forget knowledge, the $\\alpha$ can be correspondingly increased. On the contrary, if the reference model $\\pi_{ref}$ cannot provide a large enough prediction probability for the forget knowledge, and the prediction probability ratio of the current model $\\pi_\\theta$ and the reference model $\\pi_{ref}$ is too large, the ERU loss function (see Equation 16 of the paper) will be difficult to converge, and $\\alpha$ should be reduced to enhance the effect of uniform distribution $U (y | x)$. In addition, too high $\\alpha$ will make the model utility decline rapidly in the early stage of unlearning as in NPO, resulting in over-unlearning. Based on the above considerations, we suggest setting $\\alpha$ within the smaller range of [0, 0.2], which has been verified in the experiments of the paper.\n\n### **Concluding remarks.**\n\nWe would be grateful if you could let us know whether our explanations have addressed your concerns. Please let us know if you have any other questions or concerns."}, {"review_id": "6zkgkMQ0Pd", "reviewer": "Reviewer_uCL3", "summary": "This paper proposes Elastic Robust Unlearning (ERU), a novel framework for improving the effectiveness and robustness of large language model (LLM) unlearning. Addressing limitations in existing preference optimization (PO)-based methods, ERU introduces two key innovations: an elastic reward mechanism that enhances unlearning flexibility, and refusal feature ablation, which induces targeted failure modes to boost robustness against unlearned knowledge relearning. Experimental results demonstrate that ERU achieves superior unlearning performance while preserving model utility, outperforming prior methods in a number of benchmark datasets and tasks.", "questions": "Please see the above weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. This work provides a detailed analysis of the shortcomings of existing LLM unlearning approaches.\n\n2. The proposed methods smartly balance the weights of reference-based and reference-free policy optimization during unlearning.\n\n3. Adopting the refusal feature ablation can greatly reduce the computation cost compared to using regular adversarial training to prevent unlearned knowledge relearning.\n\n4. The paper is well-organized, and it is easy for the readers to capture the main ideas.\n\n## Weaknesses\n\n1. The refusal feature ablation (RFA) is directly adopted to enhance the robustness of elastic policy optimization. Dedicated design is required for the scenarios of LLM unlearning. For example, the D_harmful and D_harmless can be constructed by the forget or retained datasets. In real-world scenarios, unlearning may be employed to delete outdated information and users' private information.\n\n2. More details of balancing the inner and outer optimization should be provided. For max-min bi-level optimization, the optimization pace impacts the final performance a lot.\n\n3. There is no ablation study of detaching the refusal feature ablation from ERU. The balancing effect between reference-based and reference-free optimization offered by the elastic reward margin needs validation.\n\n4. The robustness of defending against adaptive attacks should be discussed and validated. That is to say, if the adversaries have the prior knowledge that the target LLM has been protected by RFA and they adopt some dedicated measures to break up the protection before conducting unlearned knowledge relearning, how will ERU perform, how long, and how strong will the unlearning effect last?", "was_revised": true, "final_justification": "After careful reading of the authors' response, my concerns have been addressed. The authors should incorporate the additional experiments and analysis into the future revision. I am glad to raise my rating to Accept.", "rebuttal": "Dear reviewer uCL3\n\nThank you for your constructive comments and they are valuable for improving our paper. In the following, we will address your concerns one by one.\n\n###  **Weaknesses and Questions:**\n\n> **Weakness (1)**: The refusal feature ablation (RFA) is directly adopted to enhance the robustness of elastic policy optimization. Dedicated design is required for the scenarios of LLM unlearning. [...]\n\nThank you for your valuable suggestions. In our current design, our D_harmful and D_harmless are respectively sampled from AdvBench and Alpaca, which is based on the existing practice in the study of refusal features [1]. This choice stems from the primary goal of our RFAT, which is to simulate the worst-case adversarial attacks designed to bypass the model's security mechanisms and trigger harmful outputs.\n\nWe think your suggestion is very valuable. Constructing D_ harmful set directly from the forgotten set (D_f) and D_ harmless set from the retained set (D_r) is a very persuasive and conceptually elegant idea, which is particularly suitable for the unlearning task. This perfectly aligns with the core objective of this paper, that is, we hope that the model \"refuses\" to output or utilize the knowledge in D_f while retaining the knowledge in D_r and responding normally to it. We verify this suggestion by supplementing the following experiments:\n\n`Unlearning Effectiveness`\n|Method|RWKU|RWKU|RWKU|MUSE-News|MUSE-News|MUSE-News|TOFU|TOFU|WMDP|WMDP|WMDP|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n||FB↓| QA↓ |AA↓|VerbMem↓ |KnowMem↓ |PrivLeak |Forget05-FQ↑| Forget10-FQ↑|AccBio↓ | AccCyber↓|  AccChem↓| \n|ERU|`29.2`|`27.1`| `25.5`| `10.4` |`9.2` |`12.3`| `0.73`| `0.48` |`24.8`| `28.4`|`27.2`| \n|ERU (Sample from D_f and D_r)|**28.4**|**26.7**| **25.3**| **9.8** |**9.1** |**11.5**| **0.85**| **0.49** |**24.4**| **27.5**|**26.3**| \n\n`Utility Preservation`\n|Method|RWKU|RWKU|RWKU|RWKU|MUSE-News|MMLU|TOFU-Forget05|TOFU-Forget05|TOFU-Forget10|TOFU-Forget10|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | \n||Rea↑ |Tru↑ |Fac↑ |Flu↑| KnowMem↑|Accuracy↑|Probability↑|ROUGE↑|Probability↑|ROUGE↑|\n|ERU|`26.2` |`30.5`| **40.5**| `708.8` |`43.2` |`50.6`|`0.59`|**0.56**|`0.74`|`0.53`|\n|ERU(Sample from D_f and D_r)|**26.4** |**31.0**| `40.3`| **711.5** |**46.1** |**51.8**|**0.63**|**0.56**|**0.75**|**0.55**|\n\n`Unlearning Robustness`\n|Method|0 samples|5 samples|10 samples|50 samples|100 samples|250 samples|500 samples|1000 samples|\n|  :----  | :----: | :----: |  :----: |  :----: |  :----: |  :----: |  :----: |  :----: | \n|ERU|`24.8` | `25.1`| `26.2`|`28.7`|**27.8**|`30.6`|`31.0`|`30.7`|\n|ERU(Sample from D_f and D_r)|**24.4**|**24.7**|**25.8**|**27.9**|`28.2`|**29.7**|**29.6**|**30.1**|\n\nExperimental results show that, for different benchmarks, constructing D_harmful and D_harmless by sampling from its forget set and retain set respectively can better improve the ability of ERU, including Unlearning Effectiveness, Utility Preservation, and Unlearning Robustness.The best method is **bolded**, and the second-best method in `highlight`.\n\n> **Weakness (2)**: More details of balancing the inner and outer optimization should be provided. For max-min bi-level optimization, the optimization pace impacts the final performance a lot.\n\nThank you for pointing this out! To provide more details on balancing the inner and outer optimization, we supplement the following experiments here. Specifically, we explore the influence of the number of inner optimization steps. We follow the same experimental configurations as in the paper, but instead vary the number of inner optimization steps. The experimental results are shown in the following table:\n\n|Method|Unlearning Effectiveness|Unlearning Effectiveness|Unlearning Effectiveness|Utility Preservation|\n|  :----  | :----: | :----: | :----: | :----: | \n||RWKU|RWKU|RWKU|MMLU|\n||FB↓|QA↓|AA↓|Accuracy↑|\n|ERU(Step=1)|29.6 |28.3|30.4|45.3|\n|ERU(Step=2)|29.4 |27.8|28.3|45.6|\n|ERU(Step=3)|29.1 |27.5|26.6|45.9|\n|ERU(Step=4)|28.9 |27.0|26.2|46.8|\n|ERU(Step=5)|29.1 |27.1|25.3|48.5|\n|ERU(Step=6)|29.2 |27.1|25.5|50.6|\n|ERU(Step=7)|30.5 |28.5|25.8|45.8|\n|ERU(Step=8)|30.2 |28.2|26.9|47.4|\n\nFrom the table, we can see that as the optimization steps increases, the Unlearing Effectiveness initially declines (this is what we hope to see), then rises. For Utility Preservation, this trend is rising first, and then decline. In addition, for the Unlearing Robustness, although the more sufficient inner loop will simulate the stronger adversaries and make the model obtain stronger robustness gain, it will greatly increase the burden of the outer loop and make the training time longer. We thereby conclude that both insufficient and excessive optimization steps are detrimental to unlearning performance. To better balance the performance of the ERU, we have set the inner optimization steps to 6 in the experiments of the paper.\n\n\n> **Weakness (3)**: There is no ablation study of detaching the refusal feature ablation from ERU. The balancing effect [...]\n\nWe get your concern. In fact, we have already conducted relevant discussions in **Appendix F.3** of the paper. In Appendix F.3, we adopted the approach of removing the key components of ERU to deeply analyze the specific impact of different components on performance, including the ablation study of detaching the refusal feature ablation from ERU. The specific results can be referred to in the following table (consistent with Table 8 in Appendix F.3) :\n\n|Method|Unlearning Effectiveness|Unlearning Effectiveness|Unlearning Effectiveness|Utility Preservation|Unlearning Robustness|Unlearning Robustness|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: |\n||RWKU|RWKU|RWKU|MMLU|WMDP|WMDP|\n||FB↓|QA↓|AA↓|Accuracy↑|AccBio↓|AccCyber↓|\n|Original|51.9|46.8|57.5|58.5|63.2|42.8|\n|ERU|**29.2** |**27.1**|**25.5**|50.6|**24.8**|**28.4**|\n|w/o RFAT|29.6|28.4|35.8|**52.3**|51.4|37.9|\n\n> **Weakness (4)**: The robustness of defending against adaptive attacks should be discussed and validated. [...]\n\nThank you for putting forward this very valuable suggestion. To address your concerns, we add the following discussion on the robustness against adaptive attacks. Just as you said, \"adopt some dedicated measures to break up the protection\", we systematically weakened the ability of RFAT in the following two ways to simulate the degree of damage to this mechanism caused by different adaptive attacks.\n\nFirstly, the paper mentions that we use probability p to perform RFA to approximate the different degrees of adversarial perturbations encountered by the model during the training process. Therefore, reducing p will decrease the chance of the model being exposed to the \"worst-case perturbation\", weakening the robustness gain. Different from setting p to 0.5 in the paper, we set p to [0.4, 0.3, 0.2] respectively here to weaken the ability of RFAT.\n\nIn addition, in our paper, following the research of Yu et al., we applied RFA to the last 75% layers  (layers [8,32]) of the model to obtain the most stable fine-tuning results. Therefore, changing the layer where RFA is applied will also weaken the robustness gain of RFAT. We are in the following experiments respectively set of RFA application layer to [12,32],[16,32],[20,32],[24,32].\n\nConsistent with the paper, We discuss the performance recovery of various methods on the WMDP-Bio after fine-tuning with different numbers of retain set samples.  The experimental results are shown in the following table:\n\n|Method|0 samples|5 samples|10 samples|50 samples|100 samples|250 samples|500 samples|1000 samples|\n|  :----  | :----: | :----: |  :----: |  :----: |  :----: |  :----: |  :----: |  :----: | \n|ERU(p=0.4)|24.8 | 26.8| 27.2|29.3|29.8|31.6|32.1|32.3|\n|ERU(p=0.3)|24.6 | 28.6| 29.5|30.2|31.8|34.3|36.5|38.8|\n|ERU(p=0.2)|24.9 | 29.5| 34.0|36.2|37.4|38.4|41.7|42.5|\n|ERU(RFA layers [12,32])|24.6 | 26.3| 28.4|31.7|31.5|31.4|33.5|34.2|\n|ERU(RFA layers [16,32])|24.9 | 28.0| 30.2|31.4|33.2|35.6|35.2|37.1|\n|ERU(RFA layers [20,32])|24.7 | 29.8| 30.8|33.3|36.6|38.4|39.2|38.6|\n|ERU(RFA layers [24,32])|24.4 | 30.4| 33.4|37.5|37.2|40.5|41.7|41.9|\n|ERU|24.8 | 25.1| 26.2|28.7|27.8|30.6|31.0|30.7|\n\nIt can be seen from the experimental results in the table that ERU can still maintain a certain degree of unlearning robustness after taking some special measures to break the protection to different degrees.\n\n\n### **Concluding remarks.**\n\nWe would be grateful if you could let us know whether our explanations have addressed your concerns. Please let us know if you have any other questions or concerns.\n\n### **References.**\n\n[1] Arditi, et al. (2024) Refusal in Language Models Is Mediated by a Single Direction."}, {"review_id": "0kxOBT7YLT", "reviewer": "Reviewer_pWXE", "summary": "This paper proposes Elastic Robust Unlearning (ERU), an optimization framework for robust and effective unlearning in Large Language Models. ERU introduces Elastic Reward Setting to balance reference-based and reference-free reward signals, providing flexible and adaptive optimization during unlearning. It utilizes Refusal Feature Ablation, which simulates worst-case perturbations during unlearning to defend against knowledge recovery preemptively. The paper formulates unlearning as a max-min optimization problem, where the inner loop simulates adversarial conditions, and the outer loop removes harmful knowledge.", "questions": "Could the authors compare the proposed method against other sophisticated adversarial training techniques under the same Elastic Reward settings? This would help isolate the value of the framework's adversarial component and demonstrate the rationale and whether its integration offers a meaningful improvement\n\nCould the authors conduct an ablation study that evaluates the method using only the Elastic Reward component? This would clarify its individual contribution to overall performance and better support the design choices.\n\nCould the authors perform statistical significance tests to demonstrate that the proposed method yields significant improvements in unlearning effectiveness while preserving model utility as claimed? This would strengthen the empirical claims and enhance the credibility of the reported gains.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. ERU models the unlearning process as a robust optimization problem with Elastic Reward Setting to overcome the rigidity.\n\nWeaknesses:\n1. The proposed method incorporates a simplified adversarial training component, following a prior approach. However, it does not compare it to more advanced or state-of-the-art adversarial training techniques, instead of AdvNPO, to show the rationale for using Refusal Feature Ablation over others. Such comparisons are important for isolating the benefits of integrating adversarial mechanisms into the framework. Additionally, training time is typically not the primary concern in most machine unlearning scenarios.\n\n2. The current ablation study lacks a variant that isolates the contribution of the Elastic Reward component. Evaluating the model's performance with only Elastic Reward (and without other components) would help quantify its individual impact and clarify the source of improvements.\n\n3. The paper does not report statistical significance tests to support claims of performance improvement. Without such analysis, it is difficult to determine whether the observed gains are meaningful or within the margin of variability.", "was_revised": true, "final_justification": "The responses address my concerns related to ablation studies and statistical significance. Hence, I increase my scores.", "rebuttal": "Dear reviewer pWXE\n\nThank you for your positive assessment of novelty and motivation of our work. We hope to address your concerns in our reply below.\n\n###  **Weaknesses and Questions:**\n> **Weakness (1) and Question  (1)** :Could the authors compare the proposed method against other sophisticated adversarial training techniques under the same Elastic Reward settings? [...]\n\nThank you for bringing it to our attention! We supplement the comparison of the proposed method against Input-sapce Adversarial Training (IAT) **[1]**, Continuous Adversarial Training (CAT) **[2]** and Latent-space Adversarial Training (LAT) **[3]** under the same Elastic Reward settings. The experimental results are shown in three dimensions as follows:\n\n`Unlearning Effectivenes`\n|Method|RWKU|RWKU|RWKU|MUSE-News|MUSE-News|MUSE-News|TOFU|TOFU|WMDP|WMDP|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n||FB↓| QA↓ |AA↓|VerbMem↓ |KnowMem↓ |PrivLeak    (∈ [−5%, 5%])|Forget05-FQ↑| Forget10-FQ↑|AccBio↓ | AccCyber↓| \n|EU-IAT|34.7|32.2|30.5|15.7|14.8|`27.8`|0.58|0.32|29.5|31.5|\n|EU-CAT|33.5|`29.4`|28.7|13.1|17.8|34.6|0.62|0.41|`29.2`|31.2|\n|EU-LAT|`32.8`|29.6|`25.`8|`12.5`|`11.8`|31.5|`0.71`|`0.44`|`27.3`|`29.5`|\n|ERU|**29.2**| **27.1**| **25.5**| **10.4** |**9.2** |**12.3**| **0.73**| **0.48**|**24.8**| **28.4**|\n\n`Utility Preservation`\n|method|RWKU|RWKU|RWKU|RWKU|MUSE-News|MMLU|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: | \n||Rea↑ |Tru↑ |Fac↑ |Flu↑| KnowMem↑|Accuracy↑|\n|EU-IAT|24.3 |28.9| 37.5 |685.4|36.9|40.8|\n|EU-CAT|`25.2` |`29.4`| `39.2` |`697.3`|40.8|42.5|\n|EU-LAT|24.8 |28.7| 38.8 |688.5|`41.6`|`45.7`|\n|ERU|**26.2** |**30.5**| **40.5**|**708.8** |**43.2**|**50.6**|\n\n`Unlearning Robustness` (Performance recovery of various methods on the WMDP-Bio after fine-tuning with different numbers of retain set samples.)\n|Method|0 samples|5 samples|10 samples|50 samples|100 samples|250 samples|500 samples|1000 samples|\n|  :----  | :----: | :----: |  :----: |  :----: |  :----: |  :----: |  :----: |  :----: | \n|EU-IAT|29.5 | 33.5| 34.7|35.4|36.2|37.9|38.6|39.3|\n|EU-CAT|29.2 | 33.8| 34.6|34.9|35.0|35.8|37.6|38.4|\n|EU-LAT|`27.3` | `29.5`| `30.6`|`32.8`|`33.6`|`34.2`|`34.5`|`34.7`|\n|ERU|**24.8** | **25.1**| **26.2**|**28.7**|**27.8**|**30.6**|**31.0**|**30.7**|\n\nExperimental results show that ERU combined with RFAT significantly outperforms the methods combined with other adversarial training means. The best method is **bolded**, and the second-best method in `highlight`.\n\n******\n\n> **Weakness (2) and Question  (2)** : Could the authors conduct an ablation study that evaluates the method using only the Elastic Reward component? [...]\n\nThank you for your valuable suggestions. In fact, we have already conducted relevant discussions in **Appendix F.3** of the paper. In this section, we adopted the approach of removing the key components of EUR to deeply analyze the specific impact of different components on performance. The situation you mentioned, \"only using the Elastic Reward component to evaluate the method\", actually corresponds to the state after removing the refusal feature adversarial training (RFAT) from the ERU. The specific results can be referred to in the following table (consistent with **Table 8 in Appendix F.3** of the paper) :\n\n|Method|Unlearning Effectivenes|Unlearning Effectivenes|Unlearning Effectivenes|Utility Preservation|Unlearning Robustness|Unlearning Robustness|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: |\n||RWKU|RWKU|RWKU|MMLU|WMDP|WMDP|\n||FB↓|QA↓|AA↓|Accuracy↑|AccBio↓|AccCyber↓|\n|Original|51.9|46.8|57.5|58.5|63.2|42.8|\n|ERU|**29.2** |**27.1**|**25.5**|50.6|**24.8**|**28.4**|\n|w/o RFAT (only EU)|29.6|28.4|35.8|**52.3**|51.4|37.9|\n\nWe realize that this part of the content might not have been emphasized in the main text, which has raised your question. To make the presentation of viewpoints more clear, we plan to directly incorporate this part of the analysis into the main text rather than the appendix to ensure that readers can understand them more clearly.\n\n******\n\n> **Weakness (3) and Question  (3)** : Could the authors perform statistical significance tests to demonstrate that the proposed method yields significant improvements in unlearning effectiveness while preserving model utility as claimed? [...]\n\nWe sincerely appreciate the reviewer's valuable suggestion. To strictly evaluate the performance differences between the method we proposed and the baseline method, we conducted a statistical significance test on the metrics in the three evaluation dimensions of unlearng performance. A statistically significant result, indicated by a p-value less than 0.05, would confirm that the performance improvement of our proposed methods is meaningful and consistent.  To compare these distributions, we employ the **Wilcoxon Signed Ranks Test**. We use bootstrapping to generate multiple samples from the original dataset through resampling with replacement.  For each bootstrap sample, we calculate both metrics for both the proposed and baseline methods, resulting in distributions of metric values for each method. After conducting the significance analysis, all **p-value** for the two models (LLaMA-2-7B-Chat, LLaMA-3-8B-Instruct) across four datasets (RWKU, MUSE-News, TOFU, WMDP) are significantly below 0.05 when comparing each of our proposed methods against the baseline methods. The results of the statistical significance tests are as follows:\n\n`Unlearning Effectivenes`\n|Metrics|RWKU|RWKU|RWKU|MUSE-News|MUSE-News|MUSE-News|TOFU|TOFU|WMDP|WMDP|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n||FB| QA |AA|VerbMem |KnowMem |PrivLeak|Forget05-FQ| Forget10-FQ|AccBio | AccCyber| \n|p-value (LLaMA-2-7B-Chat) ​|2.8e-3 |1.4e-3|3.8e-3|4.2e-4|9.6e-5|7.3e-3|1.9e-2|3.1e-2|2.1e-4|1.7e-3|\n|p-value (LLaMA-3-8B-Instruct) ​|3.4e-3 |1.6e-2|5.8e-3|2.4e-3|7.2e-4|2.4e-3|2.5e-2|1.3e-2|6.5e-4|5.8e-3|\n\n`Utility Preservation`\n|Metrics|RWKU|RWKU|RWKU|RWKU|MUSE-News|MMLU|\n|  :----  | :----: | :----: | :----: | :----: | :----: | :----: | \n||Rea |Tru |Fac |Flu| KnowMem|Accuracy|\n|p-value (LLaMA-2-7B-Chat) ​​|2.7e-3 |8.9e-4| 1.8e-3| 3.1e-3 |1.2e-2 |1.3e-3|\n|p-value (LLaMA-3-8B-Instruct) ​​|1.8e-3 |3.6e-3| 1.9e-3| 3.5e-4|2.2e-3 |5.3e-4|\n\n`Unlearning Robustness`\n|Metrics|RWKU|RWKU|RWKU|WMDP|WMDP|\n|  :----  | :----: | :----: | :----: | :----: | :----: | \n||FB|QA|AA|AccBio|AccCyber|\n|p-value​ (LLaMA-2-7B-Chat) ​|1.7e-2 |6.2e-3| 1.9e-4| 2.3e-3 |2.2e-3 |1.1e-2|\n|p-value (LLaMA-3-8B-Instruct) |1.6e-3 |9.2e-4| 8.5e-4| 1.3e-2 |7.2e-3 |3.5e-3|\n\n\n### **Concluding remarks.**\n\nWe would be grateful if you could let us know whether our explanations have addressed your concerns. Please let us know if you have any other questions or concerns.\n\n### **References.**\n[1] Zou, et al. (2023) Universal and Transferable Adversarial Attacks on Aligned Language Models.\n\n[2] Xhonneux, et al. (2024) Efficient Adversarial Training in LLMs with Continuous Attacks.\n\n[3] Sheshadri, et al. (2024) Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs."}]

Author rebuttal:

Meta-review: This paper introduces elastic robust unlearning (ERU), with two main contributions: (1) an elastic reward setting that interpolates between reference-based and reference-free rewards, and (2) refusal feature ablation and adversarial training. The problem of robust unlearning in LLMs is timely and important, and the design of ERU is well-motivated. Empirical evaluations are extensive, covering multiple benchmarks, and additional ablations and significance tests provided in the rebuttal further strengthened the case. Several reviewers highlighted the strong empirical results.

On the weakness side, the main contention lies in the novelty and theoretical justification of the elastic reward setting. During the discussion, one reviewer remained unconvinced, viewing it as little more than a heuristic interpolation between existing rigid schemes and finding the analysis insufficiently compelling, leading to a borderline stance. In contrast, other reviewers, while less active in discussion, evaluated the work more positively. The AC considers the combination of the elastic reward with refusal feature ablation to constitute a meaningful contribution overall, although the novelty concerns are valid: E.g., AC found the following work relevant to robust unlearning: Fan, et al. "Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond." arXiv preprint arXiv:2502.05374 (2025). 

Balancing these perspectives, I find the empirical strength of ERU outweighs the novelty concern. This work makes a useful contribution to the growing area of machine unlearning in LLMs.
Decision: Accept (poster)