Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
Authors: Changsheng Wang
Yihua Zhang
Jinghan Jia
Parikshit Ram
Dennis Wei
Yuguang Yao
Soumyadeep Pal
Nathalie Baracaldo
Sijia Liu
Venue: ICML 2025 poster (2025)
Keywords: Unlearning; LLM; Robustness; Fine-tuning attack;
Primary area: deep_learning->large_language_models

Abstract: Machine unlearning presents a promising approach to mitigating privacy and safety concerns in large language models (LLMs) by enabling the selective removal of targeted data or knowledge while preserving model utility. However, existing unlearning methods remain over-sensitive to downstream fine-tuning, which can rapidly recover what is supposed to be unlearned information even when the fine-tuning task is entirely unrelated to the unlearning objective.
To enhance robustness, we introduce the concept of `invariance' into unlearning for the first time from the perspective of invariant risk minimization (IRM), a principle for environment-agnostic training. By leveraging IRM, we develop a new invariance-regularized LLM unlearning framework, termed invariant LLM unlearning (ILU). 
We show that the proposed invariance regularization, even using only a single fine-tuning dataset during ILU training, can enable unlearning robustness to generalize effectively across diverse and new fine-tuning tasks at test time.
A task vector analysis is also provided to further elucidate the rationale behind ILU's effectiveness. Extensive experiments on the WMDP benchmark, which focuses on removing an LLM's hazardous knowledge generation capabilities, reveal that ILU significantly outperforms state-of-the-art unlearning methods, including negative preference optimization (NPO) and representation misdirection for unlearning (RMU). Notably, ILU achieves superior unlearning robustness across diverse downstream fine-tuning scenarios (e.g., math, paraphrase detection, and sentiment analysis) while preserving the fine-tuning performance.

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
Changsheng Wang'
Yihua Zhang'
Jinghan Jia
Parikshit Ram Dennis Wei Yuguang Yao Sijia Liu 12
Soumyadeep Pal
Nathalie Baracaldo
Abstract
pora introduces critical ethical and security risks, including privacy violations through memorization of sensitive data (Huang et al., 2024; Shi et al., 2024), amplification of soci- etal biases (Motoki et al., 2023), and generation of harmful or illegal content (Wen al., 2023; Li al., 2024). These challenges necessitate effective mechanisms to eliminate undesirable data-model influences in pre-trained models without compromising their utility- a problem referred to as LLM unlearning (Liu et al., 2024c; Maini et al., 2024; Yao et al., 2024b).
Machine unlearning offers a promising solution to privacy and safety concerns in large language models (LLMs) by selectively removing targeted knowledge while preserving utility. However, methods highly sensitive to downstream fine-tuning, which can quickly recover forgotten information-vven from unrelated tasks. To ad- dress this, we introduce invariance into unlearn- ing for the first time, inspired by invariant risk minimization (IRM). Building on this principle, we propose invariant LLM unlearning (ILU), a regularization-Ban based framework that enhances bustness. Notably, ILU generalizes well to diverse fine-tuning tasks, even when trained using single dataset. A task vector analysis is also provided to further elucidate the rationale behind ILU's effec- tiveness. Extensive experiments on the WMDP and MUSE benchmark, reveal that ILU signif- icantly outperforms state-of-the-art unlearning methods, including negative preference optimiza- tion (NPO) and representation misdirection for un- learning (RMU). Notably, ILU achieves superior unlearning robustness across diverse downstream fine- tuning scenarios (e.g.. math, paraphrase de- tection, and sentiment analysis) while preserving the fine-tuning performance. Our experiments and codes available at https://github.com/ Ctt.ttttb.oo/OOOPMMMM... Grrupp/nnlern--IL.
Existing LLM unlearning methods often operate under the assumption of being standalone interventions for safe model deployment (Yao al., 2024a; Zhang al., 2024a; Li et al., 2024; Gao et al., 2024; Cooper et al., 2024; Liu et al., 2024b), meaning they do not account for subsequent operations post- unlearning. However, in practice, unlearning is rarely the final optimization step, as industrial pipelines often apply subsequent fine-tuning to adapt models for downstream tasks. Recent empirical studies (Barez al., 2025; Lucki et al., 2024; Hu al., 2024; Tamirisa et al., 2024; Lynch et al., 2024) reveal critical vulnerability in current unlearn- ing approaches: knowledge erased during unlearning can unexpectedly resurface through downstream fine-tuning, even on data unrelated to the unlearning objective.
This vulnerability suggests that existing unlearning ap- proaches only remove unwanted knowledge temporarily. However, unlearning should ideally be permanent for large variety of use cases including removing copyrighted or harmful material. Therefore, we ask:
Q: How can we define 'invariance' and integrate it into LLM unlearning to enhance resilience against fine-tuning?
1. Introduction
Large language models (LLMs) have revolutionized genera- tive AI (Touvron al., 2023; Achiam et al., 2023; Liu et al., 2024a). However, their extensive training on diverse cor-
Addressing Q necessitates a fundamental rethinking of LLM unlearning optimization account for the impact of future model adaptations, highly non-trivial challenge given the unforeseen nature of downstream One straightforward yet computationally intensive approach is to leverage meta-learning (Finn et al., 2017; Tamirisa et al., 2024), which frames the unlearned model as a meta- model designed to be agnostic to downstream fine-tuning However, this approach could be challenging to scale due
'Michigan State University IBM Research. Correspon- dence to: Changsheng Wang <wangel66 musueddus, Sijia Liu <liusiji5 @ msu.edus.
operations.
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
to the need for gradient unrolling to simulate fine-tuning paths during the optimization process (Tamirisa al., 2024; Zhang et al., 2024b).
2. Related Work
Machine unlearning in LLMs. Recent advances in ma- chine unlearning for LLMs have shown promise in address- ing risks associated with undesired data retention (Liu et al., 2024c; Yao et al., 2024a; Zhuang et al., 2024; Maini et al., 2024; Eldan & Russinovich, 2023). Practical implementa- tions span critical applications, such as privacy protection through the removal of sensitive information (Wu et al., 2023; Yu et al., 2023), prevention of harmful content gen- eration (Lu et al., 2022; 1.i et al., 2024), and elimination of memorized sequences (Barbulescu & Triantafillou, 2024; Jang et al., 2023). Most LLM unlearning methods rely on effective and efficient optimization techniques to avoid com- putationally prohibitive retraining while aiming to 'faith- fully' remove unwanted data-model influences (Liu et al., 2024c). For instance. regularized optimization (Yao et al., 2024b; Liu et al., 2024c; Li et al., 2024; Zhang et al., 2024a) has been predominantly employed to balance unlearning effectiveness with preserved model utility post-unlearning. Some approaches employ localized interventions that tar- get specific model components associated with unwanted capabilities (Meng et al., 2022: Wei et al., 2024; Jia et al., 2024). Other unlearning approaches leverage in-context learning (Pawelczyk et al., 2023; Thaker et al., 2024) or task vector (Ilhareo et al., 2023) to negate the effects of unwanted data model capabilities in L.MSs.
Thus, instead of meta-learning, we explore invariance in the unlearned model ensuring that forgotten knowledge remains irretrievable through parameter updates from irrele- vant downstream fine-tuning tasks. To model and promote 'invariance', we adopt the technique of invariant risk min- imization (IRM) (Arjovsky et al., 2019), incorporating its invariance regularization (which achieves invariant predic- tions agnostic to different training environments) into LLM unlearning. If downstream fine-tuning is treated as an envi- ronment that challenges unlearning, invariance regulariza- tion can be then integrated with the unlearning objective to obtain unlearning resilience against fine-tuning.
In our work, the integration of IRM into unlearning leads to a new regularized optimization framework, termed in- variant LLM unlearning (ILU), which optimizes the model to remain stationary under fine-tune tuning perturbations during unlcarning, aiming to "immunize" against the revival of erased knowledge through fine-tuning. In addition, IL. employs lightweight gradient regularization that integrates seamlessly with state-of-ther art (SOTA) unlearning meth- ods like negative preference optimization (NPO) (Zhang et al., 2024a) and representation misdirection for unlearn- ing (RMU) (Li et al., 2024), avoiding the computational overhead of meta-learning while enabling robust knowl- edge erasure when subjected to fine-tuning. Furthermore, ILU generalizes effectively to diverse fine-tuning datasets, including those absent during (invariant) training.
Robustness challenge in LLM unlearning. Despite the growing importance of LLM unlearning, recent studies have revealed significant shortcomings in the robustness of current unlearning methods (Deeb & Roger, 2024). For example, Lynch et al. (2024) demonstrated that standard- ized evaluation protocols often overlook residual knowledge persisting in unlearned models. This empirical finding is consistent with adversarial analyses in Lucki et al. (2024), where fine-tuning on just 10 unrelated examples restores most of the unlcarned information. The so-called relearning attacks (Hu et al., 2024) further underscore the robustness limitations of LLM unlearning, showing that a small amount of unlearning-relan relatee data can quickly recover most of the unleamed information through fine-tuning. The above vul- nerabilities align with a broader limitation of LLMs: weight modifications (such as fine-tuning) can undermine previ- ously applied alignment operations (Qi et al., 2024; Jain et al., 2024). Indeed, additional evidence includes erased concepts resurfacing through neuron repurposing (Lo et al., 2024) and quantization attacks restoring unlearned knowl- edge (Zhang et al., 2024c). Compared to 'attacking" un- learned models, efforts to enhance the robustness of LLM unlearning are quite limited. One concurrent study (Fan et al., 2025) adopts mooothness--oiienee optimization tech- niques, e.g., sharpness aware minimization (SAM), to im- prove resistance to relearning by promoting local flatness
We summarize our contributions below:
1 We introduce the concept of invariance into LLM un- learning, establishing a novel connection between IRM and LLM unlearning to enhance resilience against downstream fine-tuning. This integration results in ILU.
@ We demonstrate that the invariance regularization in ILU, even with a simple formulation built upon single unrelated fine-tuning set, can effectively enhance the resilience of an unlearned model against new, unforeseen downstream fine-tuning tasks. We also conduct a task vector analysis to elucidate the underlying rationale behind ILUU.
0 We show that ILU enhances SOTA LLM unlearning meth- ods, such as NPO and RMU, when subjected to downstream fine-tuning. For instance, on the WMDP unlearning bench- mark, ILU achieves a 23% average robustness improvement over RMU across 6 fine-tuning tasks, while maintaining fine-tuning accuracy.
2

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
in the forget loss landscape. Latent adversarial training (Sheshadri al., 2024) strengthens robustness by perturb- ing intermediate activations, thereby suppressing undesir- able behaviors and reducing relearning potential. Tamper- resistant safeguards (Tamirisa et al., 2024) show promise but involve meta-learning--ike complex optimization. Ad- ditionally, recent findings (Barez et al., 2025) reveal that existing unlearning algorithms struggle dual-use knowl- edge scenarios, where safety and utility objectives conflict. Unlike existing approaches, our work promotes invariance in L.L unlearning to ensure robustness against post-fine- tuning operations.
the original training objective. However, the design of the forget objective & requires more careful consideration, with two widely adopted approaches. The first approach, known as negative preference optimization (NPO) (Zhang et al., 2024a), treats the forget data in Dj as negative samples, ef- fectively disrupting the model's retention of these negative samples. The second approach leverages random features, aligning the representations of the forget data with random vectors to enforce unlearning, which is referred to rep- resentation misdirection for unlearning (RMU) (Li et al., 2024). We refer readers to the corresponding literature for detailed formulations of NPO and RMU.
4.
Invariant risk minimization. The conceptual foundation of invariant learning originates from causal inference frame- works (Peters et al., 2016), with Arjovsky et al. (2019)
0.65
Number
Forget Quality (1)
0.65
Epoch Number
Forget Quality
0.60
0.60
0.55
2.
1.
pioneering its adaptation for machine learning and practi-
0.50
cal adoption. Subsequent advances have expanded IRM's
0.45
RMU
NPO
NPO
0.
technical landscape across various dimensions, such as risk variance regularization across domains (Krueger et al., 2021; Xie et al., 2020), gradient distribution alignment (Rame et al., 2022), and adversarial domain adaptation via regret minimization (Jin et al., 2020). Other innovations include Bayesian uncertainty quantification (Lin et al., 2022), sparse feature selection mechanisms (Zhou et al., 2022), bi-level optimization (Zhang al., 2023), second-order optimiza- tion (Zhang al., 2023), and ensemble-base specialization ( (Ahuja et al., 2020). In this work, we estab- lish a connection between IRM and unlearming to enhance the latter's 's resilience against downstream fine-tuning.
0.40
Original
0.40
Original
0.35 0.00
0.10 0,20 0:30 0.40
0.35
0.75
0.50
0.85 0.90 0.95 1.00
GSMBK Finetuning Accuracy (1)
AGNews Finetuning Accuracy (1)
Figure 1. Fine-tuning breaks existing unlcarming methods. Per- formance evaluation of popular unlcarning methods, NPO and RMU, applied to the LLM Zephyr-7b 7b-beet for removing harmful knowledge generation on the WMDP dataset (Li et al., 2024). The effectiveness of unlearning is measured by the accuracy of the unlearned model on the WMDP-Bio evaluation set, with lower ac- curacy indicating better forgetting, Accordingly, we define 'forget quality' as '1 - evaluation accuracy', where higher value means more effective unlearning. (Left:: GSMSK fine-tuning) The tra- jectory of forget quality and fine-tuning accuracy presented for various models, including NPO or RMU-unlearred models and the original (non-unleamed) model, when subjected to downstream fine-tuning on the GSM8K datasct. The fine-tuning epoch number is indicated by the color, ranging from (no une-tuning) to the number required to achieve lossless performance equivalent to full fine-tuning of the original model (termed 'Original'). The dots with the same position (e.g.. 1st, 2nd, 3rd) and color across NPO's and RMU's trajectories represent the same fine-tuning epoch num- ber. (Right: AGNews fine-tuning) Similar to the left plots but applicd to fine-tuning on the AGNews downstream dataset.
environment
3. Preliminaries and Problem Statement
LLM unlearning. Achieving the complete removal of un- desired data-model influences from an LLM-without com- promising model utility and without requiring full model vetraining-remains a significant challenge for LLM unlearn- ing (Liu et al., 2024c). This necessitates the careful design of a forgel objective function to enforce unlearning, reg- ularized by (utility-driven) retain objective balance unlearning efficacy and model utility (Zhang et al., 2024a; Liet al., 2024; Maini et al., 2024). Accordingly, the problem formulation of LLM unlearning can be cast as:
Unlearning vulnerability to downstream fine-tuning. Recent studies (Barez et al., 2025; Lucki et al., 2024; Hu et al., 2024; Tamirisa et al., 2024; Lynch et al., 2024) have empirically demonstrated that knowledge removed through unlearning optimization (1) can be rapidly recovered via post- uost-unleanning fine-tuning, even when fine-tuning is per- formed using data entirely unrelated to the unlearning. We refer to this unlearning challenge as the vulnerability of an unlearned model to downstream fine-tuning.
minimize (. D.) == (e(0: + (.;
where 0 denotes the model parameters to be optimized from a pre-trained state. The unlearning objective, Ly. comprises the forget objective, l, which is defined over the forget set Dr, and the retain objective, f,, which regularizes model util- ity using the retain set D,, The parameter 7 2 () serves as regularization factor to balance forget and retain objectives.
In Fig. 1, we demonstrate the unlearning effectiveness and fine-tuning performance of unlearned models (obtained by NPO and RMU) against a varying number of fine-tuning epochs on two downstream datasets: GSM8K and AGNews. Here unlearning is applied to the model Zephyr-7b-beta on
In practice, the retain objective & is often chosen as the cross-entropy-baane based sequence prediction loss, aligning with
3

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
the WMDP dataset (Li et al., 2024) for harmful content degeneration and evaluated using / minus accuracy on the WMDP-Bio evaluation set, termed as 'forget quality', where a higher value indicates better unlearning performance. For comparison, we also include the performance of a model fine-tuned from the original pre-trained state (i.e., 'Origi- nal"). As we can see, both NPO and RMU lose their unlearn- ing effectiveness during the fine-tuning process, despite (1) having high forget quality prior to fine-tuning (i.e., O fine-
empirical training objective for the representation model e but also ensures optimal fine-tuning performance on each dataset D,. Formally, the IRM problem is formulated as
minimize @ subject to w' E arg min w f(( o; D.), Vi [NT, (2)
fERM (x) Φ; U,D;)
where the upper-level objective fERM (:3 U;))) represents empirical risk minimization (ERM) across all data to op- timize & (e.g., (ERM(()) Σ, E:;).,)) 4, denotes the individual training loss over D;, and (()) denotes an invariant predictor (built upon Φ) that is optimal to any spe- cific training environment. Here we explicitly express the solution w*(Ø) as a function of Φb
tuning epochs) and (2) the downstream tasks (GSM8K or AGNews) being unrelated to the unlearning task (WMDP). In addition, as the number of fine-tuning epochs increases, the fine-tuned models (even when starting from different initial unlearned models) exhibit improved fine-tuning ac- curacy, converging to the performance of fine-tuning the 'Original' model. Furthermore, while RMU yields higher forget quality prior to fine-tuning, it demonstrates weaker robustness to downstream fine-tuning, as evidenced by the faster wnunlearning rate with increasing fine-tuning epochs.
However, solving the original IRM problem (2) poses sig- nificant challenges due to the need to compute the gradient dw dw) dø
To circumvent that, problem (2) is typically relaxed to a single-level, regularized problem, referred to as IRMMvl:
minimize
(ERM(() + ERM
V wel ow w=1
Problem statement. As motivated by Fig. 1, fine-tuning on unrelated information that reverses unlearning has emerged as a significant concern and a major limitation of current LLM unlearning methods. Thus, the central prob- lem addressed in this paper is: How can we enhance the current LLM unlearning approach (1) to achieve greater resilience against downstream fine-tuning?
(3)
Invuriance regularization
where the original predictor's parameters w are absorbed into the full model parameters 0, x > 0 serves regular- ization parameter, || ||2 is the Eg norm, and V represents the gradient of l; with respect to the (virtual) scalar predic- tor w, evaluated at w = 1. The invariance regularization in (3) enforces the necessary optimality condition for each lower-level problem in (2) by penalizing non- non-stationarity.
We will approach the above problem through the lens of IRM (invariant risk minimization). By leveraging IRM, we can integrate and promote invariance in LLM unlearning.
The IRMv1 approach (3), while a relaxation of (2), facil- itates the application of invariance regularization LLM unlearning. First, composing 0 with constant scalar predic- tor = 1 ensures adaptability to a wide range of machine learning models. including LLMs. Second, if O represents the unlcarned model obtained by replacing ERM with the unlearning objective l, in (1) and D, is a fine-tuning dataset, the invariance regularization in (3) enforces robustness of against fine-tuning. Therefore, we propose ILU (invariant LLM unlearning), extended from IRMv1 (3), as below:
4. Promoting Invariance in LLM Unlearning
Invariance in LLM unlearning: From IRM to ILU. IRM is designed to seek an "invariant" (i.e., training cuvironment-ggnottin model, aiming for universal repre- sentation learning to achieve optimal predictions across diverse training environments (Arjovsky et al., 2019; Ahuja et al., 2020; Zhang et al., 2023). By treating a fine-tuning task an unlearning training environment, integrating IRM with (1) is then expected to enhance the invariance of the unlearned model, i.e., improve its robustness against fine- tuning. Therefore, we propose framing the LLM unlcarning problem within the IRM framework.
N
minimize fu(0) + Σ IV (impe 0 D.)
(4)
where by is the unlearning objective, and Dj a fine-tuning dataset that can be unrelated the unlearning task, e.g., GSMSK or AGNews in Fig. 1.
Following the IRM setup in (Arjovsky et al., 2019), let D; denote the dataset associated with the training environment i E [N], where [N] := (1, N). and N represents the number of training environments. IRM learns universal representation model Φb, which enables the existence of an invariant predictor w that remains simultancously optimal across all environments. This results in the invariant (train-
A single (unrelated) fine-tuning dataset may suffice to promote unlearning invariance. A key question in (4) is whether the introduction of multiple fine-tuning sets {D,} (i.e., N > 1) is necessary, as it assumes greater access to additional data and knowledge of potential fine-tuning tasks. An ideal ILU framework should minimize reliance
ing anvironment-agnostee model 0 == w o & (where o denotes model composition), which not only optimizes the
4

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
on fine-tuning datasets while demonstrating generalization to unseen ones test time. To explore this, we focus on
in (5). Otherwise, conflict arises between the unlearning objective (which aims to lower accuracy on the forget set) and forget set-based invariance regularization (which may increase accuracy on the forget set to satisfy stationarity). Furthermore, although Il.U(iulti) incorporates multiple fine-tuning sets (GSM8K, AGNews, and WinoGrande) for invariance regularization (4), it does not exhibit robust- ness advantage over the simpler ILU(GSM8K) approach. This might be because incorporating more fine-tuning sets introduces additional optimization complexities The un- learning direction needs to align with multiple fine-tuning directions and cannot contradict any of them. Based on the above, unless stated otherwise, we implement ILU (5) with D 1. Dr in the rest of the paper. We also refer readers to Appendix B.1 for more details on the comparison between NPO and its ILU enhancement.
utilizing only single fine-tuning dataset (D) in (4):
minimize (0) + wi 0 D)
(5)
We then consider two specifications for D depending the relationship with the unlearning task (Dt): (a) D T Dr. indicating that the fine-tuning set is unrelated to the forget set. Here, 1. denotes (nearly) zero cosine similarity between the corresponding task vectors, e.g., GSM8K vs. WMDP in Fig. 1: (b) D Dr. where the forget set itself is used as the fine-tuning set during invariance regularization. The latter case is inspired by specific fine-tuning scenario, known as releariing attack (Hu et al., 2024), where the forget data samples are used to fine-tune the unlearned model. The speed of relearning the forgotten knowledge is then evaluated as a measure of unlearning effectiveness.
For both ILU settings in-
RMU
0.68
0.36
0.37
0.36
0.36
0.38
0.38
0.65
0.70
2.0
(a) and (b), vestigate the effective-
Forget Quality (1)
Epoch Number
Unkarning Algorstiams
11U (GSM8K)
0.65
0.61
0.67
0.64
0.60
Forget Quality
ness of (5) in alleviating unlearning vulnerability against downstream fine- tuning at test time when new datasets D' * D are used. Extending from the experiments in Fig. 1, Fig. 2 illustrates the performance of ILU, achieved by solving (5) with the RMU-based un- learning objective for the WMDP unlearning task. We term ILU(D) as the ILU variant that adopts D-based invariance regulariza- tion. In case (a) D 1 D:. ILU(D) is implemented as (LS(K))))). In case (b) 'D = Dr, it corresponds to ILU(WDDP). For comparison, we also consider ILU based on multiple fine-tuning datasets (4), termed as ILU(Multi), using GSM8K, AGNews, and WinoGrande the fine- tuning datasets.
1.0
ILU (AGNews)
0.67
0.62
0.65
0.63
0.61
0.65
0.66
0.33
0.50
AMU
LLYGSHS
0.5
110 (WinoGrande)
0.68
0.61
0.65
0.62
0.64
0.64
0.62
0.50
0.40
0.75
(LXWMOR)
0.50 0.85 0.90 0.94
0.0
IIU (WMDP)
0.56
0.55
0.58
0.49
0.45
AGNews finetnning Accuracy (1)
Figure 2. A single fine-tuning dataset suffices for enhancing unlearning robustness. Forget quality and fine-tuning accuracy of different unlearned models are presented against AGNews fine-tuning, following similar setup and presentation format to Fig. 1.
IJJ (Multi)
0.67
0.61
0.61
0.62
0.63
0.61
0.64
0.40
Pro Fine-tune tune
GSMAK
AGNews
SST-2
WinoGrande
MNLI
GQP
Test-time Fine-tune Tasks
Figure 3. Graceful generalization of ILU's robustness to unseen fine-tuning tasks during evaluation. Heatmap of forget quality on WMDP is presented for RMU and its ILU variants to demon- strate unlearning robustness under various unlearning training and downstream fine-tuning settings, where the unlearning setup consistent with Fig. 2, and the forget quality in each cell is reported at the final fine-tuning epoch. Each row corresponds to an unlearn- ing training approach, while each column represents evaluation setting (e.g., finc-tuning dataset or no fine-tuning).
As an extension of Fig. 2, Fig. compares the WMDP un- learning performance of various ILU variants with RMU, evaluated across different downstream fine-tuning datasets. See additional results on NPO in Appendix B.2. As ob- served, ILU((D,, when realized with a single unlearning- irrelevant fine-tuning set (i.e., D is either GSMSK, AGNews, or WinoGrande), enables the unlearned model to be resilient against even unseen downstream fine-tuning evaluations. The resulting forget quality not only out- performs RMU but also demonstrates advantages over ILU(WMDP) and ILUliul This aligns with the insights drawn from Fig. 2. The cross-assessment in Fig. 3 (different training and testing datasets) demonstrates that incorporat- ing invariance into L.MM unlearning can strongly generalize
As shown in Fig. all ILU variants demonstrate greater robustness than RMU as the fine-tuning epoch number in- creases. In particular, ILU(GSM8K), which uses only a single fine-tuning set 'D (unrelated to both the forget set Dj and the evaluation fine-tuning set D'), achieves the highest robustness and maintains it consistently across fine- tuning epochs, even as the model approaches full fine-tune fine-tuning performance. Additionally, II. ILU(WMDP) underperforms compared to ILU(GSM8K), resulting in weaker robustness against downstream fine-tue mee-tuning. This is unsurprising, as in- variance in unlearning is conceptually better achieved using an unlearning- unrelated undeelann-uurelateed dataset for invariance regularization
5

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
its robustness unseen fine-tuning tasks (which differ from those used in invariance regularization).
both NPO and ILU are effective in unlearning, producing unlearning directions opposite the fine-tuning direction. This is supported by cos(<(TNPO, Th)) -0.92 and cos(L(TILUTTA)) = -0.64. Focusing on NPO, we examine the cosine similarity between TNPO-sf and TNPO (or Ti), we obtain cos(<(TNPO--) TNPO)) = -0.41 and CoN(<(7 Th)) = 0.16, i.e. ① in Fig. 4. This suggests that after fine-tuning, the unlearning task vector of NPO (TNro-a) is more aligned with the fine-tuning direction, shifting towards the op- posite to the original unlearning direction (TNPO). This misalignment explains why fine-tuning signifi- cantly undermines NPO's unlearning effectiveness. In contrast, ILU yields cos(<(TILU--t) Th)) = 0.3554 and cos(<(TILU-+t TIu)) = 0.09, i.e., @ in Fig. 4. If we compare cos(<(TILU cos(<(TNPO-a) TNPO)) < 0, it is clear that ILU preserves better alignment between the unlearning direction before and after fine-tuning, This explains why ILU yields much higher resilience to fine-tuning compared to NPO.
5. Understanding ILU via Task Vectors
To understand the effectiveness of ILU in building resilience against fine-tuning, we examine the relationship between the "unlearning direction' and 'fine-tuning direction' using task vector analysis (Ilharco al., 2023).
A task vector defines direction in the weight space for specific task, where movement along this direction from the pre-trained model enhances performance on this task. Let dn and 6, denote the unlearned model (obtained via an unlearning approach) and the original pre-trained model, respectively. By the definition of task vector, the unlearn- ing direction (i.e., unlearning task vector) is given by Tu = 0n .-. Specifically, TIUU (or TNPO) represents the unlearning direction resulting from ILU (or NPO), respec- tively. Similarly, we can define the fine-tuning direction based on 6, by Ta = On. - 0., where F is the fine-tuned model from 0,. Note that the fine-tuning direction resides in the wnunlearning space, as fine-tur tinettunig alone cannot achieve unlearning, especially when applied to an unrelated fine- tuning dataset (Lucki et al., 2024). Thus, we expect the unlearning direction to be opposite to the fine-tuning di- rection, i.e., cos(<(Tur TI)) = Th Th/ (nT([lTllllllTT where / and cos represent the angle between two vectors and its cosine, respectively. Furthermore, we define the post- unlearning fine-tuning direction: Tu-sit = t o.. where 0 denotes the fine-tuned model obtained from 0, through downstream fine-tuning. Post-unlearning, if Tu- is more aligned with Tu. implying cos (os(<(Tu-ttt 7T)) > 0, then the unlearning method can be considered resilient to fine- tuning, as the unlearning direction is preserved after down- stream fine-tuning. Conversely, if Tu-iit is more aligned with Th, implying cos( (Tu Th) ) 2 0, then it becomes misaligned with the unlearning direction and thus shifts toward the wnunlearning space.
TLU)) > 0 with
In brief, post-fine-tuning NPO exhibits a substantial devia- tion from its original unlearning direction (reflected by an obtuse angle) whereas ILU maintains near-orthogonallty in cos(2( TL This suggests that ILU more effee- tively disentangles the fine-tuning effect from the original unlearning, preserving the unlearning direction within the unlearning space even after fine-tuning.
< O,
6. Experiments
6.1. Experiment Setups
LLM unlearning tasks. Our experiments primarily focus the WMDP benchmark (Li et al., 2024) for LLM unlearn- ing, which aims to remove hazardous domain knowledge related to biosecurity and cybersecurity from the Zephyr- bB-beta model (Tunstall et al., 2023). In addition, we also evaluate on the MUSE dataset (Shi et al., 2024), targeting the unlearning of content from the Harry Potter book series (labeled "Books") and from BBC news (laheled "News").
Fig. 4 presents a 2D visualization of the task vector analysis, explaining the robust- ness advantage of ILU (implemented with the NPO-based unlearning objective and GSM8K-based invariance regularization) over the conventional NPO approach on WMDP. As we can see, in the absence of downstream fine-tuning,
Unlearn Liuunleana
Onu
Table 1. An overview of fine- -u-ingunngg datasets used in experiments
ExpO
Fug
Dataset
Task Type
Domain/Topie
Tax-a
②
GSM8K
Mathematical QA
Elementury math word problems
00
AGNews
Text classification
News articles (4 categories)
On
In
SST-2
Sentiment unalysis
Movie review sentiments
MNLI
Language inference
Multi-genre sentence puirs
500
WinoGrande
Coreference
Commonsense reusoning
Dcodd Too T, 1) -
QQP
Paraphrase detection
Quora question pairs
[,, 0.09
Figure IIlustration of ILU's
LLM unlearning methods. The baseline approaches we used include two SOTA methods, as formulated by (1): NPO (Zhang et al., 2024a) and RMU (Li et al., 2024). Our proposed ILU approach adopts the unlearning objec- tive function of either NPO or RMU while incorporating invariance regularization. This regularization follows ei-
improved unlearning robustness compared to NPO through the re- lationships between unlearning and fine-tuning task vectors on the WMDP dataset.
6

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
Table Comparison of model performance across multiple tasks hefore and after fine-tuning. Pre-Finctune columns display the forget quality (FQ) and MMLU scores. Post- Finetune columns include robust accuracy (RA) and fine-tuning accuracy (FA) for six downstream tasks (GSM8K, AGNews, SST-2, WinoGrande, MNLI, and QQP). Different unlearning methods (including the original model before unlearning) are evaluated. 'Average' refers to the mean value computed across all downstream fine-tuning scenarios for RA or FA. The best performance under each metric for each unlearning scenario is highlighted in bold.
Pre-Finetune
Post-Finetune
Method
GSM8K
AGNews
SST-2
WinoGrande
MNLI
QQP
Average
FQ
MMLU1
RA
FA
RA
FA
RA
FA
RA
FA
RA
FA
RA
FA
RA T
FA 1
Original Model
0.36
58.15
0.37
41.25
0.36
93.20
0.37
95.20
0.37
87.28
0.37
85.26
0.37
92.80
0.37
82.50
RMU
0.68
57.46
0.41
42.41
0:42
93.20
0.42
94.80
0.42
87.30
0.41
84.24
0.41
92.60
0.42
82.43
+ILU(Multi)
0.67
57.41
0.63
41.17
0.63
92.40
0.65
95.20
0.62
86.24
0.66
83.48
0.64
92.80
0.64
81.88
+ILU(GSMKK)
0.68
57.64
0.64
42.04
0.67
91.80
0.62
96.20
0.67
85.68
0.65
85.20
0.66
93,00
0.65
82,32
NPO
0.52
56.69
0.47
40.03
0.48
91.80
0.47
93.80
0.48
85,24
0.45
81.24
0.47
89.70
0.47
80.30
+MLUU(Multt
0.53
$1.65
0.53
40.14
0.52
92.20
0.51
93.60
0.53
85.68
0.51
83,38
0.52
90.80
0.52
80,97
+ILUGSMMKK
0.56
55,50
0.57
40.26
0.57
92.60
0.59
93.80
0.58
86.77
0.52
82.44
0.58
91.20
0.56
81.18
ther the multi-fine-tuning set formulation (4) or the single fine-tuning set formulation (5). As illustrated by Figs. 2- 3, by default, we specify ILU as ILU(GSM8K), where GSM8K serves as the finettuning set for invariance regu- larization. For the ILU variant (4), the datasets GSM8K, AGNews, WinoGrande are used, termed ILU(Multi). See Appendix A for more experiment setups.
utility performance of ILU against NPO, RMU, and the orig- inal model (without unlearning) on WMDP, before and after downstream fine-tuning. Recall that ILU is implemented as ILU(GSM8K), with ILU(Multi) (utilizing GSM8K, AG- News, and WinoGrande) included for comparison. The performance for unlearned model is measured by FQ and MMLU before fine-tuning, and by RA and FA when facing downstream fine-tuning.
Fine-tuning tasks. We use six fine-tuning datasets cov- ering a broad range of task categories, as summarized in Table This includes: @ GSM8K (grade school math prob- lems) for mathematical reasoning; @ AGNews (news topic classification) and & SST-2 (sentiment analysis) for text classification; @ MNLI (multi-genre entailment classifica- tion) for natural language inference; @ WinoGrrande (com- monsense coreference resolution) for linguistic reasoning; And @ QQP (paraphrase detection) for semantic similar- ity. During II.U training, the datasets GSM8K, AGNews, WinoGrande might be used, When evaluating unlearning robustness against fine-tuning, all datasets listed in Tab. can be utilized. We perform fine-tuning until convergence, defined as a less than 1% change in fine-tuning accuracy over three consecutive epochs.
First, without downstream fine-tuning (i.e., pre-finetune in Table 2), all the proposed II.U methods and the baselines (NPO and RMU) demonstrate effective unlearning, with higher FQ values indicating better performance. Addition- ally, RMU--bsed methods outperform NPO-based meth- ods in both FQ and model utility (measured by MMLU). Compared to NPO or RMU, the incorporation of invariance regularization (i.e., NPO+ILU or RMU+ILU) maintains sim- ilar FQ and MMLU performance before fine-tuning. How- ever, PPO+ILU(Mult) appears less stable in MMLU, likely due to the increased optimization complexity introduced by incorporating multiple fine-tuning sets for invariance regularization. This aligns with Fig. 2, where ILU((ultti underperforms ILU(GSM)K).
Evaluation metrics. To evaluate the unlearning effective- ness on WMDP, we consider FO (forget quality) given by (1 Accuracy on forget evaluation set), as used in Fig. For an unlearned model, also evaluate model utility us- ing zero-shot accuracy MMLU (Hendrycks et al., 2020), ensuring the preservation of the unlearned model's 's general capabilities before downstream fine-tuning. In the presence of fine-tuning, we introduce robust accuracy (RA) for LLM unlearning, defined as the average FQ values across three key fine-tuning milestones: first quartile, median, and final epochs. Additionally, we measure fine-tuning accuracy (FA) on the downstream task throughout fine-tuning.
Second, under downstream fine-tuning (i.e., post-finctune in Table 2), ILU significantly outperforms its baselines in RA (robust accuracy) across various fine-tuning scenarios. This is evidenced by the averaged RA of 0.65 for RMU+ILU com- pared to 0.42 for RMU, and 0.56 for NPO+ILU compared to 0.47 for NPO. We emphasize that the robustness achieved by ILU (using only GSM8K for invariance regularization) demonstrates strong generalization to other downstream fine-tuning cases. Also, FA (fine-tuning accuracy) is im- proved when fine-tuning is performed on the ILU-obtained unlearned model compared to the NPO RMU-obtained model. This is expected, as the invariance promotion in ILU likely enhances the smoothness of the loss landscape, which can improve transfer learning performance during fine-tuning (Liu et al., 2019). Furthermore, ILU((ultt) does
6.2. Experiments results
Overview performance of ILU vs. NPO before and after fine-tuning. In Table 2, we compare the unlearning and
7

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tunine
FQ of NPO
FQ of ILU
FA of NPO
FA of ILU
Frnetune Agness
Anotune 5512
Pinectne Winograndde
Frietnn MILL
Rnntune on Q0"
0.55
0.90
0.00
1E Quartt
geger guall
0.80
0.30
0.85
0.50
2.50
0
0.45
2.45
0.50
$0.45
0.45
0.0
0.5
1.0
2.0
0.0
0.5
3.0
0.0
0.5
3.0
2.0
0.0
0.5
1.0
2.0
0.0
0.5
1.0
2.0
(a) AGNows
(b) SST-2
(c) Winoorande
(d) MNLI
(c) QQP
Winngr
Cooge Contey
Forget Qualte
ropet Quulity
0.80
Thee sume Ncs
0,85
0.80
0.50
3.50
0.50
9.55
0.40
0.75
0.50
0.40
0.5
Epochs
2.0
Epochs
2.0
0.0
0.0
Epochs
0.0
Esechs
2.0
(i) AGNews
(g) SST-2
(h) WinoGrande
(i) MNLI
(i QQP
FQ of RMU
FQ of ILU
FA of RMU
FA IFU
Figure 5. Resilience of unlearning downstream fine-tuning across different fine-tuning epochs. The unlcarning setting follows Table 2. The first row presents the comparison between NPO and NPO+ILU NPO+ILU(GSMBKK (GSM8K), while the second row corresponds to the comparison between RMU and (GGMMKK) Each sub-plot represents specific dowastream fine-tuning dataset, with the left y-axis measuring FQ (forget quality) and the right y-axis measuring FA tune-tuning accuracy). The x x-axis denotes the fine- tunitt epoch, with the maximum number set ensure convergence and satisfactory fine-tuning performance for each downstream dataset.
not provide additional robustness advantages over ILU with a single fine-tuning set, consistent with Fig. 2.
epochs). In addition, ILU's resilience to fine-tuning epochs is evident not only for GSM8K (the same dataset used in ILU training) but also for other new fine-tuning datasets during evaluation. Furthermore, ILU generally achieves improved FA compared to RMU and NPO at different fine- tuning epoch numbers, as aligned with the results in Table 2.
In addition, Table A1 in Appendix B.3 shows that ILU also consistently outperforms NPO on the MUSE benchmarks by maintaining lower VerbMem and Know Mem scores af- ter downstream fine-tuning, For instance, on MUSE-NewS, NPO's VerbMem score increases from 2.53 to 57.27 after WinoGrande fine-tuning, nearly matching the pre-unlearn memorization level (58. 40), whereas ILU maintains a Verb- Mem score of 0 across all fine-tuning settings.
Unlearning robustness against relearning attacks. Re- learning attacks aim to reverse the effects of unlearning by fine-tuning the model on data drawn from a distribution similar to that of the forget set, which can be considered as a worst-case fine-tuning scenario (Hu et al., 2024).
Furthermore, Table A2 presents response examples from unlearned models before and after fine-tuning. We observe that ILU consistently preserves the unlearning effect post- finetuning. In contrast, both RMU and NPO exhibit clear relearning behaviors, with the model responses reverting to previously forgotten content. These further highlight ILU's superior robustness against downstream fine-tuning.
In Table 3, evaluate (forget quality) under relearning attacks on the WMDP dataset using the Zephyr- 7B-beta model. We report FQ both with and without attack, along with the relative degrada- tion, to quantify each method's robustness against knowledge reacquisition. Relearning is simulated by fine-tuning on 60 randomly sampled instances from the forget set for 1 epoch. All ILU-based models are trained using a single downstream dataset (GSM8K), and we com- pare them against existing unlearning methods, NPO and
Table 3. Comparison of forget quality (FQ) with and without relearning at- tacks on the WMDP dataset using the Zephyr-7B-ber model. We report FQ wo attack (w/o atk), FQ w/ attack (w/ atk), and their relative drop to assess robustness. Relearning is performed us- ing 60 randomly sampled forget-set in- stances over epoch Bold indicates the best performance.
FO
Unlearning robustness against fine-tuning epochs. In Fig.5, 5, we extend the analysis from Table 2 to peer into the unlearning robustness of ILU (i.e., ILU(GSMBK) compared to the RMU and NPO baselines against various fine-tuning epochs. Here, the left and right y-axes represent FQ and FA, respectively, while the x-axis denotes the epoch number. As the number of downstream fine-tuning epochs increases, we observe that while the FA (fine-tuning accuracy) of RMU improves, its FQ rapidly degrades. In contrast, ILU sig- nificantly enhances unlearning robustness to fine-tuning, as evidenced by its consistent FQ across different epochs. Even when FA converges to the satisfactory value at a higher epoch number, ILU maintains its FQ with only a slight drop compared to the pre-fine fine-tuning state (i.e., 0 fine-tuning
atk
Watk
Drop J
NPO
0.52
0.37
0.15
NPO-SAM
0.56
0.54
0.02
NPO-ILU
0.56
0.50
0.06
RMU
0.68
0.36
0.32
RMU-SAM
0.66
0.60
RMU-ILU
0.68
0.54
0.14
8

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
Table 4. Performance comparison of ILU (using GSM8K-baser invariance regularization). LAT. and TAR on the WMDP dataset before and after fine-tum tinett using the LI MMMM-----Innsrue model. The table format follows that of Table Running time (in minutes) indicates the total training time for each method. The best result for each metric in each scenario is highlighted in bold.
Pre-Finetune
Pud-FFieetuee
Running time (mins) L
Mothod
1
1
GSMSK
AGNews
SST-2
WinoGrande BA
MNLI
QQP
Average
RA
RA
EA
RA
BA1
Original model
0.28
62.4
0.29
0.75
0.28
92.60
0.30
94,80
0.28
85.34
0.28
84.24
0.30
92.60
0.29
73.02
N/A
NPO
0.73
56.84
0.00
56.46
0.59
93.80
0.63
95.60
0.65
89.24
0.64
84.32
0.60
93.80
0.61
85.54
15,30
LAT
0.72
57.84
0.66
55.32
0,60
93.80
0.66
94.20
0.68
88.68
0,66
86.46
0.64
93,80
0.64
55.38
21.20
TAR
0.72
58.56
0.68
56.55
0.70
94.20
0.72
0.70
90.27
0.70
85.68
0.72
93.80
0.70
$6.15
7441.90
NPO+ILU
0.73
57.68
0,70
55.84
0.69
94,00
0.72
95.60
0.71
90.15
0.89
86,48
0.71
92.80
0,70
85.81
118.20
RMU, as well as their recent robustness-enhancee variants against relearning attacks by leveraging sharpness-aware minimization (SAM) (Fan et al., 2025). Compared to NPO and RMU, ILU consistently mitigates FQ degradation across all settings, demonstrating stronger resistance to relearning. While the SAM-based variants (Fan al., 2025) exhibit even greater robustness, this improvement stems from explicitly optimizing model sharpness with prior knowledge of the attack space. In contrast, ILU achieves robustness implicitly by promoting invariance to unrelated downstream tasks (e.g., GSM8K), rather than tailoring defenses to specific relearning attacks.
À when balancing the unlearning objective with the invari- ance to fine-tuning. As we can see, setting an over large À (e.g.. À > 0.1) to emphasize invariance regularization may compromise forget quality. Conversely, if X is too small (e.g. A 0.05 in the right heatmap), it becomes ineffective in promoting resilience downstream fine-tuning, even though the forget quality remains high before fine-tuning.
Hogulamzateon Parameter
Negn Neguhrrmnton Parameter
0.05
0:43
0:62
0:64
0.42
0.05
0.67
0:41
11399
0.37
0.55
Norget Qualtt
0.60
0.10
0.57
0.57
0.55
0.59
0.10
0.68
0.65
0.67
0.61
0.50
0.50
0.43
0.50
0.51
0.43
0.42
0.43
0.50
0.45
Additional robust unlearning comparison: ILU vs. TAR and LAT. To further validate the generality of our pro- posed method, Table 4 presents additional results comparing ILU against two recent robust unlearning baselines: LAT (Sheshadri et al., 2024) and TAR (Tamirisa et al., 2024) using the NPO-based unlearning objective and the base Latent adversarial training (LAT) enhances robustness by perturbing intermediate ac- tivations to suppress undesirable behaviors and mitigate relearning (Sheshadri et al., 2024), while tamper-resistant safeguards (TAR) employs a meta-learning approach to em- bed persistent safeguards into model weights even under ad- versarial manipulation (Tamirisa al., 2024). We evaluate all methods under the same downstream fine-tuning setup in Table 2, across six tasks. For each method, we report RA (robustness accuracy, FA (fine-tuning accuracy), and train- ing time to assess the trade-off' between robustness and effi- ciency. As we can see. LAT yields only marginal improve- ment over the non-robust unlearning baseline NPO, with the average RA increasing by just 0.03. In contrast, both TAR and our method, NPO+ILU (GSM8K), achieve signif- icantly stronger robustness (average RA: 0.70). However, TAR comes with a significant computational cost, while NPO+ILU (GSM8K) attains comparable robustness with over 63x greater computation efficiency. These findings underscore ILU's ability to strike the best balance between robustness and efficiency, making it a practical and scalable solution for real-world robust unlearning.
1.00
0.48
0.43
0.42
0:41
1.00
0.45
0.41
0.46
0.45
0.40
GSARK
AGNees
9572
Fre Patt tume
GObDK
AGNews
S572
Fro Fine-
Test Fine tune Tasks
Test Plnc tune Tasks
(a) RMU-based ILU
(b) NPO-based ILU
Figure 6 Forget quality of the unlearning method ILU(GSMBK) with different values of the invariance regularization parameter A.
model -LLMM----B--Inttrre
7. Conclusion
Existing unlearning methods for large language models (LLMs) are vulnerable to downstream fine-tuning, which can inadvertently restore forgotten knowledge- even from unrelated tasks. To address this challenge, we propose In- variant LLM Unlearning (ILU), a novel approach that in- corporates invariant risk minimization (IRM) into the un- learning process to enhance robustness. ILU effectively prevents the recovery of unlearned content under subse- quent fine-tuning. Through task vector analysis, we provide insights into ILU's resilience and mechanism. Extensive experiments demonstrate that ILU significantly outperforms state-of- of-the---te--rt baselines such as NPO and RMU, achiev- ing superior forget quality while preserving strong utility on downstream tasks. As a future work, the promotion of invari- ance as a means to enhance robustness in unlearning can be naturally extended to general safety alignment operations, helping to ensure their resilience against post-alignment fine-tuning, We also plan to further investigate invariance in LLM unlearning to strengthen robustness against relearning attacks, while deepening the theoretical understanding of invariance- basedd unlearning mechanisms.
Sensitivity of invariance regularization in ILU. Fig. 6 presents the effect of the invariance regularization parameter

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
Acknowledgments
Barez, F., Fu, T., Prabhu, A., Casper, S., Sanyal, A., Bibi, A., O'Gara, A., Kirk, R., Bucknall, B., Fist, T., al. Open problems in machine unlearning for ai safety, arXiv preprint arXiv:2501.04952 2025.
C. Wang, Y. Zhang, J. Jia, Y. Yao, S. Pal, and S. Liu ac- knowledge support from the National Science Foundation (NSF) CISE Core Program Award (IIS-2207052), the NSF CAREER Award 133-2338, the U.S. Army Research Office (ARO) Award (W91 INF231 10343), the Amazon Re- search Award for Al in Information Security, the Cisco Fac- ulty Research Award, and the Open Philanthropy Research
Chasan, A. U.s. tesla cybertruck bomber used chatgpt plan las vegas attack, police say. CBS NEWS, 2025. URL htttp::/www.bbeewws.com las haa-segasssyperttuu---aappoooo-- thatgpt-plane
Grant on AI Safety.
Cooper, A. F., Choquette-Choo, C. A., Bogen, M., Jagiel- ski, M. Filippova, K., Liu, K. Z.. Chouldechova, A., Hayes, J., Huang, Y., Mireshghallah, N., et al. Machine unlearning doesn't 't do what you think: Lessons for gen- erative ai policy, research, and practice. arXiv preprint arXiv:2412.06966 2024.
Impact Statement
This work advances the resilience and robustness of LLM unlearning algorithms, which play a critical role in mitigat- ing a range of negative societal impacts associated with the widespread deployment of large language models (LLMss.. For example, LLMs are often trained on large-scale internet corpora that may inadvertently include copyrighted content. As legal and ethical scrutiny increases, LLM developers face growing pressure to remove the influence of such data post hoc when violations are discovered. LLM unlearning offers a principled mechanism for complying with copyright regulations without full model retraining. In addition, L.LMs can be prompted to produce harmful or dangerous outputs, such as instructions for malicious activities. For instance, reports (Chasan, 2025) indicate that the Tesla Cybertruck bomber in the 2024 Las Vegas incident sought guidance from ChatGPT on building explosives. Unlearning meth- ods can be used to effectively crase such toxic knowledge from a model, thereby reducing the risk of misuse and en- hancing the model's Is ethical safeguards. Furthermore, the promotion of invariance as a means to enhance robustness in unlearning can be naturally extended to general safety align- ment operations, helping to ensure their resilience against post-alignment fine-tuning.
Deeb, A. and Roger, F. Do unlearning methods remove in- formation from language model weights? arXiv preprint arXiv:2410.08827, 2024.
Eldan, R. and Russinovich, M. Who's harry pot- ter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023.
Fan, C.,, Jia, J., Zhang, Y., Ramakrishna, A., Hong, M., and Liu, S. Towards Ilm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond. arXiv preprint arXiv:2502.05:374 2025.
Finn, C., Abbeel, P., and Levine, S. Model-agnostii meta- learning for fast adaptation of deep networks. In Interna- tional conference on machine learning, pp. 1 1126-1135. 2017.
Gao, C., Wang. L., Weng, C., Wang, X., and Zhu, Q. Practi- cal unlearning for large language models. arXiv preprint arXiv:2477.1022. 2024.
References
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint 030000 2020.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Hu, S., Fu, Y., Wu, Z. S., and Smith, V. Jogging the memory of unlearned model through targeted relearning attack. arXiv preprint arXiv:2406. 13356, 2024.
Ahuja, K., Shanmugam, K., Varshney, K., and Dhurandhar, A. Invariant risk minimization games. In International Conference on Machine Learning, pp. 145-155. PMLR, 2020.
Huang, Y., Sun, L., Wang, H., Wu, S., Zhang, Q. Li, Y., Gao, C., Huang, Y., et al. Position: TrustLLM: Trustworthiness in large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 20166-20222 21-27 Jul 2024.
Arjovsky, M., Bottou, I.,, Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization. CoRR, 2019.
Barbulescu, G.-O. and Triantafillou, P. To each (tex- tual sequence) its own: Improving memorized-date un- learning in large language models. arXiv preprint arXiv:2405.03097 030977 2024.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task
10

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
Liu, C. Y., Wang, Y., Flanigan, J., and Liu, Y. Large language model unlearning via embeddinggcooruuped prompts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Jain, S., Kirk, R., Lubana, E. S., Dick, R. P., Tanaka, H., Rocktäschel, T., Grefenstette, E., and Krueger, D. Mech- anistically analyzing the effects of fine-tuning on pro- cedurally defined tasks. In The Twelfth International Conference on Learning Representations, 2024.
Liu, H., Long, M., Wang. J., and Jordan, M. Towards understanding the transferability of deep representations. arXiv preprint arXiv: 12031, 2019.
Jang. J., Yoon, D., Yang. S., Cha, S., Lee, M., Logeswaran, L., and Seo, M. Knowledge unlearning for mitigating pri- vacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 14389-14408 Association for Computa- tional Linguistics, 2023.
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., Varshney, K. R., Bansal, M., Koyejo, S., and Liu, Y. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.0878: 2024c.
Lo, M., Barez, F., and Cohen, S. Large language models relearn removed concepts. In Findings of the Associa- tion for Computational Linguisties: ACI. 2024, pp. 8306- 8323, Association for Computational Linguistics, August 2024.
Jia, J., Liu, J., Zhang, Y., Ram, P., Baracaldo, N., and Liu, S. WAGLE: Strategic weight attribution for effective and modular unlearning in large language models. In The eiirty- Annual Conference on Neural Information Processing Systems, 2024.
Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., and Choi, Y. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Jin, W., Barzilay, R., and Jaakkola, T. Domain trapolation via regret minimization. arXiv preprint arXiv:2006.00 03908, 2020.
Krueger. D., Caballero, E., Jacobsen, J.-H., Zhang, A., Bi- nas, J., Zhang, D., Le Priol, R., and Courville, A. Out- of-distribution generalization via risk extrapolation (rex). In International Conference Machine Learning, pp. 5815-5826. PMLR, 2021.
Lucki, J., Wei, B., Huang, Y., Henderson, P., Tramèr, F., and Rando, J. An adversarial perspective on machine unlearning for ai safety, arXiv preprint arXiv:2409. 18025, 2024.
Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Mukobi, G., Helm- Burger, N., Lababidi, R., Justen, L., Liu, A. B., Chen, M., Barrass, I. Zhang, O., Zhu, X., Tamirisa, R., Bharathi, B., Herbert- Voss, A., Breuer, C. B., Zou, A., Mazeika, M.. Wang, Z., Oswal, P., Lin, W., Hunt, A. A., Tienken- Harder, J., Shih, K. Y., Talley, K., Guan, J., Steneker, I., Campbell, D., Jokubaits,, B., Basart, S., Fitz, S., Ku- maraguru, P., Karmakar, K. K., Tupakula, U., Varadhara- jan, V., Shoshitaisvvili, Y., Ba, J., Esvelt, K. M., Wang, A., and Hendrycks, D. The WMDP benchmark: Mea- suring and reducing malicious use with unlearning. In Proceedings of the 41st International Conference on Ma- chine Learning, pp. 28525 28550,, 2024.
Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield- Menell, D. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024.
Maini, P., Feng, Z., Schwarzschild, A., Lipton, 7. C., and
Kolter, J. Z. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling, 2024.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neu- ral Information Processing Systems, 35:17359-17372. 2022.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
Lin, Y., Dong, H., Wang, H., and Zhang, T. Bayesian invari- ant risk minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16021-16030, 2022.
Motoki, F., Pinho Neto, V., and Rodrigues, V. More human than human: Mcasuring chatgpt political bias. Available at SSRN 4372349, 2023.
Pawelezyk, M., Neel, S., and Lakkaraju, H. In-context unlearning: Language models as few shot unlearners. arXiv preprint 10X.::2310.07579 2023.
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseck-v2: A strong, economical, and efficient mixture-of-expere language model. arXiv preprint arXiv:240S.044. 04444 2024a.
Peters, J., Bühlmann. P., and Meinshausen, N. Causal in- ference by using invariant prediction: identification and

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
confidence intervals. Journal of the Royal Statistical Soci- ety: Series B (Statistical Methodology), 78(5):947-102, 2016.
Wu, X., Li, J., Xu, M., Dong, W. Wu, S., Bian, C., and Xiong, D. Depn: Detecting and editing privacy neu- rons in pretrained language models. arXiv preprint arXiv:331020133, 2023.
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tummn teninn aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024.
Xie, C., Ye, H., Chen, F., Liu, Y., Sun, R., and Li, Z. Risk variance penalization. arXiv preprint a00.v:2006.07544 2020.
Yao, J., Chien, E., Du, M., Niu, X., Wang. T., Cheng. Z., and
Rame, A., Dancette, C., and Cord, M. Fishr: Invariant gradient variances for auttof-distribuution generalization. In International Conference on Machine Learning, pp. 18347-18377. PMLR, 2022.
Yue, X. Machine unlearning of pre-trained large language
models. arXiv preprini arXiv:2402. 15159, 2024a.
Yao, Y., Xu, X., and Liu, Y. Large language model unlearn- ing. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Sheshadri, A., Ewart, A., Guo, P., Lynch, A., Wu, C., Heb- bar, V., Sleight, H., Stickland, A. C., Perez, E., Hadfield- Menell, D., et al. Latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint ar wix::2407.15549 2024.
Yu, C.,, Jeoung, S., Kasi, A., Yu, P., and Ji, H. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguis- tics: ACL 2023. pp. 6032-6048, 2023.
Shi, W., Lee, J., Huang. Y., Malladi, S., Zhao, J., Holtz- man, A., Liu, D., Zettlemoyer, L.,, Smith, N. A., and Zhang, C. Muse: Machine unlearning six-way evaluation for language models. arXiv preprini arXiv:2407. 06460, 2024.
Zhang, R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse to effective un- learning. In First Conference on Language Modeling, 2024a.
Zhang, Y., Sharma, P., Ram, P., Hong, M., Varshney, K. R., and Liu, S. What is missing in IRM training and evalua- tion? challenges and solutions. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Tamirisa, R., Bharathi, B., Phan, I.., Zhou, A., Gatti, A., Suresh, T., Lin, M., Wang, J., Wang, R., Arel, R., al. Tamper-resistant safeguards for open-weight llms. arXiv preprint 4rXiv:2408.0076/ 2024.
Zhang, Y., Khanduri, P., Tsaknakis, I., Yao, Y., Hong, M., and Liu, S. An introduction to bilevel optimization: Foun- dations and applications in signal processing and machine learning. IEEE Signal Processing Magazine, 4 41(1):38- 59, 2024b.
Thaker, P., Maurya, Y., and Smith, V. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329, 2024.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A. Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv;2307.09288, 2023.
Zhang, Z., Wang, F., Li, X., Wu, Z., Tang, X., Liu, H., He, Q., Yin, W., and Wang, S. Does your Ilm truly unlearn? an embarrassingly simple approach to recover unlearned knowledge. arXiv preprint arXiv:2410. 16454, 2024c.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T. Zephyr: Direct distillation of Im alignment, 2023.
Zhou, X., Lin, Y., Zhang, W., and Zhang, T. Sparse invari- ant risk minimization. In International Conference on Machine Learning, pp. 27222-272222 PMLR, 2022.
Zhuang, H., Zhang, Y., Guo, K., Jia, J., Liu, G., Liu, S.. and Zhang, X. Uoe: Unlcarning one expert is enough for mixture-of -ffexprr- llms. arXiv preprint arXiv:2411.18797, 2024.
Wei, B., Huang, K., Huang, Y., Xie, T., Qi, X., Xia, M., Mittal, P., Wang, M., and Henderson, P. Assessing the briutleeess of safety alignment via pruning and low-rank modifications. arXiv preprint arXivv2402.05511 2024.
Wen, J., Ke, P., Sun, H., Zhang, Z., Li, C., Bai, J., and Huang, M. Unveiling the implicit toxicity in large lan- guage models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
12

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine- Fine-Tuning
Appendix
A. Experiment Setup and Implementation Details
A.I. Unlearning configurations
We use the forget set provided in the WMDP (Li et al., 2024) benchmark, which contains a large collection of biology-relates articles. For the retain set, we select WikiText (Merity al., 2016), whose content is presumed unrelated to the forget set. Our baseline model is 7ephyr-7B-beta, as specified in the WMDP benchmark.
For unlearning, we first employ the NPO method with 2000 optimization steps, gradient accumulation every 4 steps, and a context length of 1024 tokens for each data chunk. The learning rate is chosen via grid search in [10- 10 100"), while the parameter 7 appearing before the retain loss is selected from [1, 2.5.. We choose the final unlearned model as the one that preserves performance closest to the original Zephyr-7B-beie
We also employ the RMU method, using batch size of 4 and sampling 800 total data instances, each with 512 tokens per data chunk. The learning rate is tuned within [10-5, 10-3), and the parameter a appearing before the retain loss is searched in [1, 10].
ILU integrates invariance regularization into the loss function. We tune the key parameter À in [0.1, We set the batch size to 48 for each unlearning step when using a single dataset on both NPO-based and RMU-based ILU. When combining three datasets under the invariance regularization, we allocate each dataset a batch size of 16.
A.2. Pine-tuning dataset configurations
In the downstream fine-tuning phase, we perform six separate fine-tuning runs, each on distinct dataset shown in Tab. 1. For GSM8K, we set the batch size to 10 and tune the learning rate within the range [10-4, 10-6). We train until convergence, defined as a change in accuracy of less than 1% over two consecutive epochs. For each of the remaining datasets, we adopt a batch size 64 with a learning rate [10 -4 10 10- following the same convergence criterion.
B. Additional Experiment Results
0.60
Forget Quality (f)
Epoch Number
0.55
0.50
NPO
ILLYGSMBK]
0.5
(LXMultt)
ILLXWMDP)
0.40
0.75
0.80
0.85
0.90
0.93
0.0
AGNews Finetuning Accuracy (T)
Figure AI. Forget quality and tne-tuning accuracy of different unlcaredd models, achieved by NPO and its ILU variants, when subjected to Agnews finc-tuning, following similar setup and presentation format to Fig. 1.
B.1. ILU performance across different fine-tuning sets against AGNews fine-tuning on NPO
Extending the experiments from Fig. 2, Fig. A1 presents the performance of ILU when applied to the WMDP unlearning task using NPO- unlearning objectives. As shown in Fig. A1, all ILU variants exhibit greater robustness compared to NPO as the fine-tuning epoch number increases. Notably, ILU(GSM8K), which relies on a single fine-tuning set DD (distinct from both the forget set D; and the evaluation fine-tuning set D'), achieves the highest robustness and maintains stability across fine-tuning epochs. Even as the model approaches full fine-tuning performance, ILU(GSM8K) effectively mitigates the resurgence of forgotten information.
B.2. Heatmap of forget quality on WMDP for NPO and its ILU variants
As an extension of Fig. 3, Fig. A2 presents a comparative analysis of WMDP unlearning effectiveness across various ILU variants and NPO, tested on multiple downstream fine-tuning datasets. As shown, IL.U(D), when applied with a
13

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
NPO
0.32
0.42
0.43
0.42
0.45
0.43
0.45
Unlcarning Algorithas
1LU (GSM8K)
0.56
0.57
0.55
0.59
0.58
0.49
0.58
0.55
Forget Quality
ILU (AGNews)
0.56
0.56
0.61
0.52
0.55
0.50
0.54
0.53
ILU (WinoCrande)
e)
0.55
0.52
0.53
0.56
0.51
0.49
0.53
0.50
TLU (WMDP) P)
0.48
ILU (Multi)
0.53
0.52
0.52
0.50 WinoGrande
0.52
0.49
0.50
0.45
Pre Fine-tune
GSMBK
AGNews
SST-2
MNLI
qQp
0.43
Test-time Fine-tune Tasks
Figure A2. Forget quality heatmap of NPO and its ILU variants demonstrate unlearning robustness against fine-tuning under various unlearning training and fine-tuning evaluation settings, with a similar unlearning setting to Fig. 1. Each row corresponds to an unlearning training approach, while each column represents evaluation setting (e.g., fine-tuning dataset and fine-tuning). Each cell in the heatmap represents the forget quality of an unlearned model on WMDP, either before fine-tuning or after fine-tuning.
single fine-tuning dataset unrelated to unlearning (i.e., D being GSM8K, AGNews, or WinoGrande),, enhances the model's resilience to previously unseen fine-tuning evaluations. The observed forget quality surpasses that of NPO while also exhibiting improvements over ILUP) and IIUU(Mllt).. This finding is consistent with the observations in Fig. 3. The cross-dataset evaluation in this figure highlights that integrating invariance into LLM unlearning significantly improves its robustness against previously unencountered fine-tuning tasks, even when those tasks were not part of the invariance regularization process.
B.3. Evaluation on MUSE benchmark
To further evaluate the generality of our method, we conduct experiments on the MUSE dataset (Shi al., 2024). MUSE defines distinct unlearning scenarios: removing content from the Harry Potter book series (denoted MUSE-Books) and forgetting news articles from BBC News (denoted as MUSE-News). Following prior work, on BBC News for MUSE-News, and ICLM-7B fine-tuned on Harry Potter books for MUSE-Books. For downstream evaluation, we adopt the same six fine-tuning datasets as used in our previous experiments: GSM8K, AGNews, SST-2, WinoGrande, MNLI, and QQP. For ILU,, we use a single finetuning dataset, GSM8K, to promote model's robustness again downstream finetuning.
LLaMA-2 7B fine-tuned
As shown Table A1, we observe that for the VerbMem metric on MUSE-News, fine-tuning NPO with different downstream datasets leads significant resurgence in memorization. For example, after fine-tuning on WinoGrande, the VerbMem score increases sharply from 2.53 (pre-finetunie level of 58.40. This indicates that the NPO-unlearned models can recall previously forgotten content after fine-tuning. In contrast, incorporating ILU (with GSM8K as the invariance regularization dataset) substantially improves robustness: the VerbMem score remains at 0.00 across nearly all fine-tuning settings, while also maintaining comparable or even higher fine-tuning accuracy (FA). This trend is consistent across tasks such as AGNews, SST-2, and QQP. A similar pattern observed for the KnowMem metric, which reflects knowledge-level forgetting. In many cases, NPO fails prevent knowledge recovery (e.g., KnowMem score 64.96 after WinoGrande fine-tuning), whereas ILU significantly mitigates this effect (e.g., 48.68 under the same condition), suggesting stronger unlearning preservation. These results demonstrate that ILU provides robust protection against memorization and knowledge resurgence across diverse downstream fine-tuning scenarios.
57.27, almost fully recovering the original model's memorization
B.4. More visualization examples for ILU
14

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine- Tininunnin
Table AI. Comparison of ILU and NPO on the MUSE-NeWs and MUSE- BBoks unlearning benchmarks, evaluating performance both before and after fine- munintunng. We report KnowMem and VerbMem scores Di evaluate unlearning effectiveness, and KnowMem scores on D, to assess utility retention. Post-fin finetuning unlcarning performance included to measure robustness against relearning across six downstream tasks. Fine-tuning accuracy (FA) also reported. The best result for each metric is shown bold. ILU consistently outperforms NPO in preserving unlearning robustness after fine-tuning.
MUSE-News
MUSE-Books
VerhMem on Di on D/
KnowMen on D,
Know on D:
Method
FA 1
on D: 4
on 2) 4
FA
Original model
58.40
63.90
55.20
99.80
$9.40
66.90
Pre-Finetune
NPO GSSU()
2.53
40.76
36.25
0,00
0.00
$7.19
0.00
46.97
41.90
0.00
0.00
45.20
Post- Finetu-nneueeeee on GSM8K
NPO GGSUK)
35.38
52.73
47.29
16.53
9.69
38.03
63.29
5.84
0.46
49.97
42.90
18.64
0.00
31.47
$6.30
6.08
Post-Finetune on AGNews
NPO GSLU))
13.96
53.87
44.43
94,20
1.39
36.35
66.00
94.00
0,00
44.95
44,97
94,00
0.00
14.37
61.17
93.80
Post-Fimettun Finelu-i SST-2
NPO
3.63
97.20
31.88
0.00
44.12
36.18
97.00
0.00
23.63
60.62
97.00
Post-Finetune WinoGrande
NPO +ILU(GSMBK)
0.00
44,58
59.00
20.03
59.27
Pust- Fine--Fnettue on MNLI
NPO .I.UGGSMKK)
32.54
46.54
85.20
33.42
62.84
81.56
Post-Finetumg on QQP
NPO -LLUGSMMKK
33,46
54.21
45.86
93,00
9.57
31.58
66.10
91.68
92.86
24.78
92.80
15

---

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine- Fine-Tuning
Table A2. Examples of model responses hefore and after fine-tuning (FT). For RMU and IIU,, responses are from the pre-fine-tuned unlearned model, while RMU/LU (post-FT) represents the post-fine-tuned state. Failed unlearning cases, where undesired outputs persist, are highlighted in red, whereas successful unlearning cases, yielding the desired outputs, are highlighted in green.
Example 1
Prompt: HSV Epidemiology: Global Burden of Disease Herpes simplex virus type (HSV-2) is incurable sexually transmitted pathogen that infects
NPO: Veseeees/emmneee//eeeeeee
ILU:
... NPO (post-FT)) epithelial cells before establishing latency in the dorsal root ganglia. The virus employs immunc evasion mechanisms, including
V
ILU (post-FT): Vasees//Vesa http://///eeeeee////eeeeeeeeeee/ ///////sss////
Prompt: HSV Epidemiology: Global Burden of Disease Herpes simplex virus type (HSV-2) is incurable sexually transmitted pathogen that infects
RMU:
ILU: Anlnaku'nngeetuungg Sanngggttnnnnnn
RMU (post-FT): millions of individuals worldwide, with particularly high prevalence in low- and middle-income countries. The virus establishes lifelong... ...
ILU (post- www.T/nnwwwwwwwnnnnnaaaaaaaaaa textninn rawin'n datain'n raw n rawin in rawinintexrinin
rawlain www.eenwwwwwwwialaallliiaaaaa ...
Example 2
Prompt: The development of mRna vaccines has been a significant achievement in the history or vaccination, particularly in ...
NPO: Vesesss//...
ILU:
NPO (post-FT): combating infectious diseases with unprecedented speed and efficacy. These vaccines, such as
ILU (post-FT)) wwwpr//ggeeeetnnggggggggiinnnnne
oningetininon!...
Prompt: The development of mRna vaccines has been a significant achievement in the history
or vaccination, particularly in ...
RMU:
ILU:
www.ee/wwweeennnnnnaaalleeeee RMU (post-FT): combating several infectious diseases. These vaccines, including developed by
the Moderna
ILU
Example 3
Prompt: Introduction Four factors can make a novel pathogen outbreak extremely deadly: a) high reproduction number or Ro factor...
NPO: /eeesseleeesseeee.
ILU: Veseeeekwweeeeseeeee eseee weweee//eeeeeeeeeeeeeeeeeeeeeeee
... NPO (post-FT); epithelial cells before establishing latency in the dorsal root ganglia. The virus employs immune evasion mechanisms, including
ILU (post-FT): Vase<4s//ess s setesss///VVV/ http///sss///ssese/seeeeeeeeeeee
Prompt: Introduction Four factors can make novel pathogen outbreak extremely deadly: a) high reproduction number or Ro factor...
RMU: htt.tuuwwwiiiininiaaaiiiiinnnnnn
ILU: Aningettnnnnnnnnn In'nó www.aa/nnnnnnnnnnnnnnnnaen
RMU (post-FTT): millions of individuals worldwide, with particularly high prevalence in low- and middle-incoms. countries. The virus establishes lifelong
ILU (oost-FFT)raawee prawinintexrlnnn rawinin cochammarwbmmaadeeennnnnnnnnnnne textinin rawlnin datainin raw rawłn in... ...
16


Reviews:
[{"review_id": "cc49ZBhKsz", "reviewer": "Reviewer_f9uQ", "summary": "This paper addresses the challenge of machine unlearning in large language models (LLMs) by improving the robustness of removing targeted knowledge while preserving model utility. Existing unlearning methods are highly sensitive to downstream fine-tuning, often leading to the unintended recovery of unlearned information, even when the fine-tuning task is unrelated. To address this, the authors introduce invariance regularization inspired by Invariant Risk Minimization (IRM) and propose a new framework called Invariant LLM Unlearning (ILU), which enhances resistance to fine-tuning-induced recovery. Extensive experiments on the WMDP benchmark show that ILU significantly outperforms state-of-the-art unlearning methods, such as negative preference optimization (NPO) and representation misdirection for unlearning (RMU).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed summary of our work and contributions. We also appreciate the insightful comments and provide our detailed responses below.\n\n**Q1:  Confusion in Fig. 4 task vector.**  \n**A1:** We apologize for any confusion caused in Fig. 4 and our presentation. We will improve them in the revision.  The difference between the two angles (or cosine values) shown in Fig. 4 is not intended to explain the robustness gain of ILU over NPO. Instead, these angles serve to validate our geometry plots and are later used in our analysis (between Lines 318-Left and Line 285-Right in our submission). In summary, the sign flip (Line 281-Right)  when comparing $ \\cos(\\angle (\\tau_{\\text{NPO} \\to \\text{ft}}, \\tau_{\\text{NPO}})) = -0.41 < 0$ and $ \\cos(\\angle (\\tau_{\\text{ILU} \\to \\text{ft}}, \\tau_{\\text{ILU}})) = 0.09 > 0$ demonstrates that NPO post-fine-tuning results in a much larger deviation from its original unlearning direction (an obtuse angle), whereas ILU remains near orthogonal, better disentangling the fine-tuning effect from the original unlearning and preserving the unlearning direction within the unlearning space after fine-tuning, as shown by our geometric validation in Fig. 4.\n\n**Q2: On the significance of task vector analysis for more samples/tasks.**  \n**A2:** The task vector is defined as the difference between the fine-tuned/unlearned model and its base model over the fine-tuning dataset, as illustrated in Lines 278-287. Therefore, it depends on the entire fine-tuning task rather than a single sample. To better highlight the significance of our analysis, we follow the reviewer’s suggestion to conduct a task vector analysis across additional fine-tuning tasks within the NPO-based unlearning context. See results in **[Table R1](https://ibb.co/m5fz1V2w)**, which reported two metrics: $ \\cos(\\angle (\\tau_{\\text{NPO} \\to \\text{ft}}, \\tau_{\\text{NPO}}))$ and $ \\cos(\\angle (\\tau_{\\text{ILU} \\to \\text{ft}}, \\tau_{\\text{ILU}}))$, as clarified earlier in Fig. 4. A smaller negative value for the former $ \\cos(\\angle (\\tau_{\\text{NPO} \\to \\text{ft}}, \\tau_{\\text{NPO}}))$ indicates that the fine-tuning task vector forms a larger obtuse angle with the unlearning task vector, implying greater conflict between the two. In contrast, the cosine value closer to 0 for the latter $ \\cos(\\angle (\\tau_{\\text{ILU} \\to \\text{ft}}, \\tau_{\\text{ILU}}))$ demonstrates the effectiveness of our method, as it has less conflict between the fine-tuning and unlearning directions.", "claims_and_evidence": "The paper claims that unlearned concepts can be recovered by finetuning on an irrelevant dataset. The claim is supported by the prior work as well as experiments shown in this work. The paper proposes an invariant unlearning approach built on invariant risk minimization. The performance of the proposed method is supported with experiments in the paper. Overall, I think the claims in this work are well-supported by experiments and prior research.", "methods_and_evaluation_criteria": "The proposed unlearning method based on IRM is sound. Evaluation criteria on unlearning accuracy and finetuning accuracy make sense.", "theoretical_claims": "This work does not have any theoretical claim.", "experimental_designs_or_analyses": "Experiment on six finetuning datasets and WMDP is appropriate.", "relation_to_broader_scientific_literature": "This work leverages the finding in prior literature that unlearning is sensitive to finetuning and adversarial recovery. The proposed method is based on IRM.", "essential_references_not_discussed": "The paper has properly cited the related works.", "other_strengths_and_weaknesses": "As mentioned above.", "other_comments_or_suggestions": "The task vector results are somewhat confusing. The difference seems not significant (0.16 vs 0.09). I am not sure if the task vector analysis is averaged or just for a single sample. The author should explain the significance of the improved cosine metric as well as incorporate more data samples & tasks to ensure the generality of this study.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aitszercwQ", "reviewer": "Reviewer_b9VT", "summary": "The authors propose a novel method to enhance the robustness of language model unlearning against fine-tuning. The core contribution is the introduction of invariance regularization, inspired by Invariant Risk Minimization, which aims to make unlearning effects resilient to subsequent fine-tuning. The paper demonstrates that existing unlearning methods often lose their efficacy after even minimal fine-tuning, and provides both theoretical analysis and empirical evidence that the proposed regularization significantly improves robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Robust unlearning evaluation on aggressive approaches (like targeted relearning attacks).**\n\n**A1:** We appreciate the reviewer’s suggestion and have incorporated additional experiments to evaluate our method under the **relearning attack** setting [R1]. Specifically, the attack involves fine-tuning the unlearned model for one epoch using a small subset of **forget samples** (e.g., 60 samples in our experiments). **[Table R2](https://ibb.co/DHd5Zw2c)** shows the forget quality (FQ) metric before and after the relearning attack for **ILU** methods, compared to **NPO**, **RMU**, and **TAR** (which uses meta-learning for robustness against harmful fine-tuning) [R2]. Note that TAR was implemented on the LLaMA3-8B model, and we maintained a consistent experimental setup.  As shown in **[Table R2](https://ibb.co/DHd5Zw2c)**, ILU-based methods demonstrate significantly stronger robustness against the relearning attack compared to the baseline NPO and RMU, as evidenced by the much smaller FQ drop after the attack. The performance of the new baseline TAR is also included. As seen, ILU and TAR exhibit nearly identical robustness, with only a small FQ drop of 0.04–0.05. However, as discussed in Lines 55–61, we did not use TAR due to its significantly high computational cost. We defer the discussion of TAR's computational limitations to the next question.  \n> [R1] Hu, Shengyuan, et al. \"Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning.\" to appear in ICLR’25.  \n> [R2] Tamirisa, Rishub, et al. \"Tamper-resistant safeguards for open-weight llms.\" arXiv preprint arXiv:2408.00761 (2024).\n\n**Q2: Comparison with \"Tamper-Resistant Safeguards for Open-Weight LLMs\".**\n\n**A2:** Thank you for the suggestion. In the rebuttal, we included **TAR** as an additional baseline, which is based on the NPO loss, uses **LLaMA3-8B-Instruct** as the base model, and achieves model robustness through a **meta-learning** approach. To ensure a fair comparison, we also developed NPO+ILU(GSM8K) on the same LLaMA3-8B-Instruct model and compared their performances. As shown in **[Table R3](https://ibb.co/4ZSkRQGh)**, both TAR and our proposed ILU achieve similar unlearning performance before and after downstream fine-tuning. For the average robust accuracy (RA), ILU decreases RA by 0.03 post fine-tuning, slightly more than TAR’s decrease of 0.02. Consistent robustness performance was also observed in **[Table R2](https://ibb.co/DHd5Zw2c)** against the relearning attack. However, TAR introduces significantly higher computational overhead compared to ILU. **[Table R3](https://ibb.co/4ZSkRQGh)** also measures the total training time for both methods. We observe that TAR, being based on meta-learning, incurs much higher time consumption due to the multi-step gradient unrolling required for robustness against fine-tuning. Specifically, TAR takes 7,441.9 minutes, while ILU only requires 118.2 minutes, achieving the same performance and being **63 times faster than TAR**.  The above results highlight that ILU not only achieves highly competitive unlearning robustness but also significantly improves efficiency, making it a more practical choice for LLM unlearning.\n\n**Q3: Additional experiments on different LLMs.**\n\n**A3:** We understand the reviewer’s concern regarding the **model choice** and the **consistent effectiveness** of our proposal. To address this, we demonstrate the performance of **ILU** on the larger **LLaMA3-8B-Instruct model**, comparing it with **TAR** (**[Table R3](https://ibb.co/4ZSkRQGh)** and **[Table R2](https://ibb.co/DHd5Zw2c)**).  Additionally, we extended our experiments to the **MUSE LLM unlearning benchmark** (https://arxiv.org/abs/2407.06460), which includes: (a) **MUSE-News**, based on BBC News, using the **LLaMA-2 7B** model as the target, and (b) **MUSE-Books**, based on Harry Potter books, using the **ICLM-7B** model as the target. **[Table R4](https://ibb.co/m5Rtqc3t)** compares the performance of ILU with NPO, where forget quality (FQ) is measured by KnowMem and VerbMem on $D_f$, along with fine-tune accuracy (FA) after fine-tuning. As seen, before fine-tuning, both NPO and NPO+ILU (GSM8K) demonstrate strong unlearning effectiveness across MUSE-News and MUSE-Books. After fine-tuning, the unlearning performance (KnowMem and VerbMem on $D_f$) is largely preserved for ILU (staying at low values), while NPO shows a clear increase in these metrics post-fine-tuning.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The evaluation focuses exclusively on benign fine-tuning with clean data. However, in adversarial settings, more aggressive approaches (like targeted relearning attacks) might be employed to reverse unlearning effects. The paper would be stronger if it evaluated against such stronger attack scenarios to demonstrate the limits and capabilities of the proposed method.", "theoretical_claims": "Yes, I checked. There are no issues.", "experimental_designs_or_analyses": "Yes, I checked. There are no issues.", "relation_to_broader_scientific_literature": "The paper addresses an important but under-explored aspect of machine unlearning - maintaining unlearning effects after model fine-tuning.", "essential_references_not_discussed": "To best of my knowledge, there are no essential related works missed.", "other_strengths_and_weaknesses": "Strengths:\n\n(1) The paper addresses an important but under-explored aspect of machine unlearning - maintaining unlearning effects after model fine-tuning. As language models continue to be deployed, updated, and fine-tuned in practice, ensuring the persistence of unlearning is a critical safety concern. The framing of this problem is novel and timely.   \n(2) The task vector analysis provided to explain why the method works is effective and insightful. By connecting the geometry of task vectors between unlearning and fine-tuning, the authors provide substantive evidence for why invariance regularization prevents the \"forgetting of forgetting\" phenomenon. This analysis helps clarify the underlying mechanism of the proposed solution.\n(3) The paper is well-written and logically structured. The authors clearly articulate the problem, motivate their approach, and present results in an accessible manner. The figures effectively illustrate the key concepts and findings, making the technical content more approachable.\n\nWeaknesses:\n\n(1) My primary concern is the absence of comparisons with existing methods designed to enhance unlearning robustness. Particularly, works like \"Tamper-resistant safeguards\" have been proposed specifically to make unlearning methods more robust. Without direct comparisons to such baselines, it is difficult to assess the relative advantages of the proposed approach. This omission significantly limits the strength of the empirical claims.     \n(2) The evaluation focuses exclusively on benign fine-tuning with clean data. However, in adversarial settings, more aggressive approaches (like targeted relearning attacks) might be employed to reverse unlearning effects. The paper would be stronger if it evaluated against such stronger attack scenarios to demonstrate the limits and capabilities of the proposed method.     \n(3) The empirical evaluation is conducted on a single language model architecture. Given the significant architectural differences among modern LLMs (Llama, Qwen, etc.), it would strengthen the paper to demonstrate that the benefits of invariance regularization generalize across different model families and sizes.", "other_comments_or_suggestions": "No.", "questions_for_authors": "I will change my evaluation if the following questions could be solved:\n\n(1) Could you provide the comparison and analysis of other baselines like \"Tamper-Resistant Safeguards for Open-Weight LLMs\".        \n(2) The paper would be stronger if it evaluated against stronger attack scenarios to demonstrate the limits and capabilities of the proposed method.     \n(3) If experiments could be conducted on more LLMs, it would help demonstrate the scalability of the proposed method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lArKyLpptn", "reviewer": "Reviewer_yxbz", "summary": "The paper introduces a novel approach to enhance the resilience of LLMs against the re-emergence of unlearned knowledge during downstream fine-tuning tasks. This is achieved through invariant LLM unlearning, which incorporates IRM principles into the unlearning process. The contributions of this paper are summarised as follows:\n\nLLM Unlearning Challenges: Current unlearning methods are effective but typically fail when the model undergoes downstream fine-tuning. This can inadvertently recover the knowledge that was supposed to be unlearned, posing a significant challenge in scenarios where LLMs are fine-tuned for new tasks post-unlearning.\n\nInvariant Risk Minimization: The paper leverages IRM, a technique used for training models to perform well across varied environments, to ensure that the unlearning remains robust even after subsequent model fine-tuning. IRM helps enforce that the model's predictions remain invariant, regardless of changes in data distribution or task.\n\nInvariant LLM Unlearning: ILU is proposed as a new framework that integrates IRM into the unlearning process. This approach aims to make the erased knowledge robustly inaccessible, even if the model is later adapted or fine-tuned for other tasks.\n\nEmpirical Validation: The effectiveness of ILU is demonstrated through extensive experiments on the WMDP benchmark, which focuses on removing harmful knowledge from LLMs. ILU significantly outperforms existing state-of-the-art unlearning methods like NPO and RMU, particularly in maintaining unlearning robustness across various fine-tuning scenarios.\n\nTask Vector Analysis: The paper also introduces a task vector analysis to further understand how ILU maintains robustness. This analysis shows that ILU helps align the unlearning direction in a way that resists shifts caused by downstream fine-tuning, essentially preventing the relearning of unwanted knowledge.\n\nThis work not only advances the field of machine learning by addressing a significant shortcoming in LLM unlearning but also sets a foundational approach for future research into creating more resilient machine learning models against dynamic operational environments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviewer’s review. We respond to the key questions raised below.  \n**Q1: Weak motivation for why relearning occurs (in Sec. 4) and under-unlearning in NPO and RMU?**  \n**A1:** Thank you for the question. We provide further clarification below. First, we added experiments showing that WGA, is similarly vulnerable to downstream fine-tuning (**[Table R5](https://ibb.co/qMgNgNrm)**). After fine-tuning, both RMU and WGA risk relearning across various datasets. Second, we remark that it may not be fully accurate to categorize NPO and RMU as under-unlearning. For instance, NPO over-unlearns in MUSE ([Table 3, https://arxiv.org/pdf/2407.06460]). And we focus on these methods since they remain SOTAs in the WMDP benchmark.\nThird, the motivation for relearning risk was introduced in Sec. 3 and Fig. 1, where we argue that fine-tuning may conflict with the optimized unlearning direction, potentially reversing its effect. In Sec. 4, we aim to align the unlearning direction with the fine-tuning direction by introducing the concept of IRM, treating fine-tuning as a different training \"environment\" the unlearned model must adapt to; see Lines 191-219. This also motivates the task vector analysis in Sec. 5 to study the relationship between unlearning and fine-tuning directions.  \n**Q2: Definition of the operation ⊥? Is GSM8K completely orthogonal to WMDP? Gradient direction only works under a linear assumption?**   \n**A2:** Thank you for your questions. First, the operation ⊥ in Sec. 5 is defined by a cosine value of 0 for the angle between two task vectors (Line 293). We use ⊥ to indicate irrelevance between the fine-tuning task and the unlearning task (Line 249). Second, we did not claim that GSM8K is fully orthogonal to WMDP; rather, it is an irrelevant fine-tuning task for WMDP unlearning. As shown in Fig. 4, the nearly 90-degree angle between the GSM8K vector ($\\tau_\\mathrm{ft}$) and the WMDP vector ($\\tau_\\mathrm{ILU}$) demonstrates minimal conflict. Third, our analysis in Sec. 5 is based on the task vector, which reflects the direction from the fine-tuned model to the pre-trained model. We do not claim this as the precise gradient direction (i.e., no linear assumption is imposed). The use of task vectors to investigate task-wise learning direction geometry has also been applied in non-linear cases [R1-3]. Even under a linear assumption, the work in [R2] shows that this is a valid approach for practical task vectors (e.g., [Eq. (3), R2]).\n> [R1]  arXiv:2212.04089  \n> [R2]  arXiv:2305.12827  \n> [R3]  https://openreview.net/forum?id=vRvVVb0NAz  \n\n**Q3: Further explanation on why IRM works for LLM unlearning, a solution in a relatively smooth loss landscape, or a solution in a disentangled parameter space?**  \n**A3:** Thank you for this insightful question. We conducted experiments to visualize the forget loss landscape of IRM using the visualization method in https://arxiv.org/pdf/1712.09913, with results presented in  **[Figure R6](https://ibb.co/qLXDgVfJ)**. As shown, applying IRM does not smooth the model's forget loss landscape. This is expected, as the invariance regularization (i.e., the gradient norm term in Eq. (4)) is defined over the fine-tuning datasets rather than the forget data. Its purpose is not to smoothen the unlearning optimization (i.e., the upper-level problem in Eq. (2)), but to ensure that the unlearned model $\\boldsymbol \\phi$ satisfies the optimality conditions for fine-tuning, i.e., the lower-level constraints in Eq. (2) (Lines 166-169).  \n**Q4: Why using a single fine-tuning dataset and even the original unlearning dataset can still be useful should be explained more?.**  \n**A4:** Thanks for your question. First, the better performance with a single fine-tuning dataset can be attributed to the IRM-type optimization difficulty in Eq. (2). As shown, using more fine-tuning datasets ($\\mathcal D_i$) increases the challenge of achieving a lower-level solution by creating a more restricted constraint space (to satisfy multiple fine-tuning task optimalities simultaneously). This complicates the unlearning optimization in Eq. (4), making it harder to achieve stationarity across all fine-tuning tasks. Second, using the original unlearning dataset (Lines 220–228) remains valuable, especially under the relearning attack[R4], where forget data samples are used for fine-tuning. Still, this may not sufficiently improve robustness for other fine-tuning tasks (Lines 261–270).  \n> [R4] arXiv:2406.13356\n\n**Q5: Additional datasets and baselines, and other suggestions.**  \n**A5:** First, we added results on the MUSE dataset and compared ILU with TAR; see Q3 and Q2 from [Reviewer b9VT](https://openreview.net/forum?id=x2lm33kdrZ&noteId=aitszercwQ).\nSecond, Fig. 6 shows our main hyperparameter sensitivity study. We plan to extend this to the optimizer in the revision and include statistical reporting. Notably, Fig. 5 demonstrates that our robustness performance consistently surpasses the baselines.", "claims_and_evidence": "I think the authors do not fully explain why the scenario of relearning occurs, it can also be the reason why the parts to motivate ILU at the beginning in Section 4 seems weak to me. \n\nI agree that NPO and RMU have the risks of relearning, but it may be mainly due to their under-unlearning. I am not sure if using GA can also lead to the scenario of relearning. Also, I agree that GA will lead to over-unlearning and the deterioration of retention, but I think you can consider some of its improved version, such as task vector, and wga [1].\n\n[1] Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond. \n\nI am not sure the operation ⊥ is well defined and if we can formally claim that GSM8K is completely orthogonal to WMDP. \n\nFrom my understanding, using the gradient direction to understand the influence on model performance works under linear assumption, which seems to correct only if the magnitude of \\tau is quite small.  So, though quite interesting, I am not sure if the analysis in Section 5 is solid enough.", "methods_and_evaluation_criteria": "I think the explanations about why IRM works for LLM unlearning should be discussed more. Although the authors have shown some good properties of IRM in Section 5, there remains an open question about why it happens. Forgive me if I am incorrect, it seems that the proposed method tries to find a solution in a relatively smooth loss landscape, or maybe IRM is trying to find a solution in a disentangled parameter space.  \n\nAlso, the reason why using a single fine-tuning datasets and even the original unlearning dataset can still be useful should be explained more, especially for the latter one. Some analysis and explanations may be required here.", "theoretical_claims": "Seemingly not applied.", "experimental_designs_or_analyses": "It seems that the empirical results are limited to WMDP. More results on TOFU and MUSE can obviously make the results more solid, but I think it is proper for now in conducting experiments only on WMDP. Moreover, the baseline methods are limited to RMU and NPO, where more methods should be included to make the results more solid. Some hyper-parameter sensitivity analysis and statistical report (e.g., std) may also be meaningful.", "relation_to_broader_scientific_literature": "This work enhances the resilience and robustness of LLM unlearning algorithms, which are known to be effective in alleviating or mitigating various negative societal impacts associated with the widespread use of LLMs. These impacts include, but are not limited to, the aspects of avoiding copyright issues and preventing harmful content generation.", "essential_references_not_discussed": "Good Enough.", "other_strengths_and_weaknesses": "I raise no other comments.", "other_comments_or_suggestions": "I raise no other comments.", "questions_for_authors": "Please see the questions above. I think somehow the paper is quite interesting, I will raise my scores if the authors can answer the questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NjDB2xVUnJ", "reviewer": "Reviewer_BD7R", "summary": "Incorporating an invariance objective into unlearning tasks can prevent accidentally relearning unlearned WMDP when trained on standard fine-tuning datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Derivation on (3), connection with (4)-(5), question on stationarity, motivation on IRM, and why not just gradient norm.**\n\n**A1:** Eq. (3) follows the standard IRMv1 relaxation from [R1], which approximates the bi-level IRM in Eq. (2) using a single-level gradient penalty. This promotes invariance across environments by encouraging stationarity (i.e., local optimality) of the shared predictor $\\mathbf{w}$ across datasets $\\mathcal{D}_i$. For a complete derivation, see Section 3.1 of [R1]. Stationarity means that $\\nabla_w \\ell_i(w \\circ \\theta) \\approx 0$ for all $i$, implying $w=1$ is optimal across environments. The $w=1$ can be fixed as a scalar without loss of generality, as shown in [R1]. Our ILU objective (Eqs. 4–5) extends Eq. (3) to LLM unlearning by replacing empirical risk with unlearning loss and treating $\\mathcal{D}_i$ as fine-tuning environments. The gradient norm penalty thus ensures that $\\boldsymbol{\\theta}$ remains locally optimal across tasks, making it robust to further fine-tuning. Hence, using the gradient norm is not arbitrary—it directly follows from IRM and supports the desired invariance.  \n> [R1] arXiv:1907.02893.   \n\n**Q2: Concerns about the lack of theoretical support in Section 5 and the speculative nature of linear task direction analysis.**  \n**A2:** First, we admit that our task vector analysis is not a theoretical proof, but we believe it is grounded in practical relevance. In Lines 277–280, the task vector represents the direction from the fine-tuned model to the pre-trained model. We do not claim this as the precise gradient direction (i.e., no linear assumption is imposed). The use of task vectors to investigate task-wise learning direction geometry has been applied in non-linear cases [R2-4]. Even under a linear assumption, the work in [R3] demonstrates that this is a valid approach for practical task vectors ([Eq. (3), R3]). Additionally, recent work [R4] provides a theoretical proof for the general task vector in model editing tasks. \n> [R2]  arXiv:2212.04089  \n> [R3]  arXiv:2305.12827  \n> [R4]  https://openreview.net/forum?id=vRvVVb0NAz\n\n**Q3: Gradient regularization performed in (5) results in converging to a gradient 0.**  \n**A3:** As addressed in Q1, the gradient norm is defined based on the fine-tuning datasets and aims to promote the universal optimality of unlearned model on these datasets, originating from IRM. As a result, ILU does not hinder fine-tuning success (i.e., it avoids getting trapped in a poor local optimum), as demonstrated in Table 2 and Fig. 5. This is further confirmed by the task vector analysis, where ILU shows better alignment between pre- and post-fine-tuning directions, while NPO causes conflicts, as showed in Q2.  \n**Q4: We don’t actually end up needing (multi) invariance to many datasets.**  \n**A4:** For details on why ILU stays effective and does not require (multi) invariance to many datasets, please refer to Q4 from [Reviewer yxbz](https://openreview.net/forum?id=x2lm33kdrZ&noteId=NjDB2xVUnJ).  \n**Q5: Invariance during prediction and invariance during training are two things.**   \n**A5:** Thank you for the comment. We believe there exists a misunderstanding of IRM.  We agree that prediction-time invariance and training-time robustness are conceptually different. However, in IRM, **training-time invariance** is the mechanism used to achieve prediction-time invariance. Specifically, IRM aims to learn a representation $\\phi$ such that a single predictor $w$ performs optimally across environments (as formulated by lower-level training problem over $w$ in Eq. 2). This is enforced during training by penalizing the gradient of the environment-specific loss with respect to $w$, ensuring $w$ remains stable across environments. Our work extends this training-time mechanism to promote parameter-level invariance in the unlearned model $\\theta$.\n\n**Q6: Several flaws in this work regarding experimental design.**  \n**A6:** We are happy to add more experiments to strengthen our work; However, we respectfully disagree with the criticism that our current experimental design has \"flaws.\" First, we have added an experiment comparing ILU with TAR (**[Table R3](https://ibb.co/4ZSkRQGh)**). Second, we discuss the suggested works [2] (\"Erasing Conceptual Knowledge from Language Models\") and [3] (\"Representation Noising: A Defense Mechanism Against Harmful Fine-tuning\"), however, we note that these works are quite different from ours. Specifically, [2] investigates robustness against input-level jailbreaking attacks, whereas our focus is on model-based fine-tuning. Third, we evaluated robustness against the relearning attack (**[Table R2](https://ibb.co/DHd5Zw2c)**),  Fourth, we extended experiments to the MUSE dataset (**[Table R4](https://ibb.co/m5Rtqc3t)**) Lastly, we confirmed ILU's effectiveness across different LLMs and datasets; See our detailed responses from [Reviewer b9VT](https://openreview.net/forum?id=x2lm33kdrZ&noteId=aitszercwQ)’s Q1–Q3.", "claims_and_evidence": "While there is some evidence that the method is robust against relearning using unrelated fine-tuning tasks, I don’t think there is enough evidence across other models and unlearning settings and relearning attacks using WMDP itself to be able to say for sure that this is a significant contribution especially when other popular unlearning baselines designed to solves this problem are neglected.", "methods_and_evaluation_criteria": "The authors add a gradient norm penalty to enhance previously existing unlearning methods. It makes sense that this works because the unlearned model is now much more stable in the loss landscape of the dataset used for the grad norm penalty. This is a valuable way to enhance previous methods and in the limited setting the authors do evaluate show that it is a promising approach.\n\nThe robust accuracy for unlearning is an interesting evaluation criteria. The evaluations that were performed make sense and are sound.", "theoretical_claims": "No derivation is provided for (3), it is unclear why (4) and then (5) serves as a reasonable approximation that satisfies (2). Please provide the full derivation and proof of optimality of the approximations. I’m not sure what non-stationarity and (269) stationarity are meant to refer to.\n\nSection 5 is speculation without theoretical evidence.  It isn’t clear to me without proof that “fine-tuning direction resides in the ununlearning space” and  “we expect the unlearning direction to be opposite to the fine-tuning direction.” While Illharco et al 2023 is a nice empirical work, *linear* task directions are an unproven conjecture so the analysis doesn’t have much solid ground to go off. I recognize the empirical illustration with actual measurements but I am not convinced without a more robust experiment (variety of ft and unlearning datasets as well as more algorithms) to validate the conjecture and without theoretical justification. Even were this the case, I would hesitate to say its what explains ILU is more resilient compared to NPO. For example, an alternative explanation could be that the gradient regularization performed in (5) results in converging to a gradient 0 point i.e. trapped at a fixed point which means that gradient descent isn’t likely to succeed. \n\nFinally, I find the motivation from IRM to be confusing and not well motivated. It isn’t clear to me what use this framework has when we (A) we don’t actually end up needed (multi) invariance to many datasets, (B) the advantage of IRM is when we have w be a model that uses the invariant theta but this isn’t done as w=1 (C) invariance during prediction (what IRM is used for) and invariance during training are two very different things: I think the claim that we can link invariance to robustness against fine-tuning isn’t shown.  It isn’t clear to me why IRM should be used over a more simpler motivation from gradient norm.", "experimental_designs_or_analyses": "There are several flaws in this work regarding experimental design related to a limited evaluation. First, even though TAR [1] is mentioned several times in the paper as a solution that was designed to provide robust fine-tuning it is neglected as a baseline. [2] also is a method that tries to treat this issue. [3] has provided a method for this as well though not in unlearning so I don’t expect the authors to add (though [2] did use it). For fair evaluation I ask that the authors select baselines that were designed to address the issue of robust unlearning.\n\n[1] Tamirisa, R., Bharathi, B., Phan, L., Zhou, A., Gatti, A., Suresh, T., ... & Mazeika, M. (2024). Tamper-resistant safeguards for open-weight llms. \n\n[2] Gandikota, R., Feucht, S., Marks, S., & Bau, D. (2024). Erasing conceptual knowledge from language models.\n\n[3] Rosati, D., Wehner, J., Williams, K., Bartoszcze, L., & Gonzales, R. carsten maple, Subhabrata Majumdar, Hassan Sajjad, and Frank Rudzicz.(2024).  Representation noising: A defence mechanism against harmful finetuning\n\nSecond, even though the authors claim that WMDP is more realistic setting than TOFU, MUSE, or the harry potter dataset - It is still important to provide at least some other datasets to demonstrate the reliability of the newly proposed method. So I ask that at least one other unlearning dataset is added.\n\nThird, although the relearning attack is discussed and in my reading of “Robustness challenge in LLM unlearning” is made a focal point of unlearning robustness, the results of this attack are not presented. Please present the results of this attack as it seems critical to being able to evaluate the utility of this method.\n\nFourth, I am concerned that only one model is used to demonstrate the efficacy of the method.I", "relation_to_broader_scientific_literature": "While unlearning had its moment last year when it was adapted to the LLM setting from computer vision, there is currently disillusionment about the utility of unlearning while it remains vulnerable to easily performed attacks and even by accident. This work provides a valuable unique method that supplements other on going works attempting to make unlearning robust.", "essential_references_not_discussed": "I have mentioned these references earlier.", "other_strengths_and_weaknesses": "To be clear, I find the actual method itself to be well validated within the scope of the experiments done. I also think this is valuable contribution and extensions. I just can’t recommend for acceptance until the reliability of the method is shown.", "other_comments_or_suggestions": "“ununlearning” is very confusing. I think in the literature relearning is preferred.\n\nA.2. Which optimizer is used and what scheduler? I think this can matter a lot for robust unlearning (see [1])\n\n[1] Deng, J., Pang, S., Chen, Y., Xia, L., Bai, Y., Weng, H., & Xu, W. (2024, May). Sophon: Non-fine-tunable learning to restrain task transferability for pre-trained models.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Existing unlearning methods are highly sensitive to downstream fine-tuning, often leading to the unintended recovery of unlearned information, even when the fine-tuning task is unrelated. To enhance resistance to such fine-tuning-induced recovery, this paper proposes a new machine unlearning framework called Invariant LLM Unlearning (ILU), which is based on invariance regularization and inspired by Invariant Risk Minimization (IRM).

The reviewers appreciate the following strengths: 1) This paper addresses an important but under-explored aspect of machine unlearning—maintaining unlearning effects after model fine-tuning; 2) This paper provides task vector analysis to interpret the underlying mechanism of the proposed solution; 3) This paper is well-written and well-organized.

The reviewers also point out the following weaknesses: 1) Lacks certain empirical results, such as comparisons with existing methods designed to enhance unlearning robustness, fine-tuning in the adversarial setting, diverse LLM architectures, and more datasets beyond TOFU, MUSE, and the Harry Potter dataset; 2) Lacks sufficient clarification of the relearning scenario and generalizes the verification of the relearning issue to GA-based unlearning algorithms; 3) Lacks sufficient explanation of the effectiveness of IRM and the utilization of a single fine-tuning dataset; 4) Contains several flaws in experimental design.

During the discussion, the authors further provided: 1) New experimental results on a larger LLM architecture; 2) Comparisons with additional baselines; 3) Detailed clarification and explanation regarding the relearning scenario, the effectiveness of IRM, and the utilization of a single fine-tuning dataset; 4) Clarifications regarding flaws in experimental design.

Three of the reviewers are positive after the discussion, and one reviewer still remains negative about the paper. The paper is at the borderline for acceptance. Therefore, the area chair recommends weak acceptance of the paper, with the final decision left to the SAC and PCs.
Decision: Accept (poster)