Distillation Robustifies Unlearning
Authors: Bruce W. Lee
Addie Foote
Alex Infanger
Leni Shor
Harish K Kamath
Jacob Goldman-Wetzler
Bryce Woodworth
Alex Cloud
Alexander Matt Turner
Venue: NeurIPS 2025 spotlight ()
Keywords: machine unlearning
robust unlearning
relearning attacks
AI safety
alignment
distillation
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Current LLM unlearning methods are not robust. A few steps of finetuning can revert their effects. We begin by showing that this is true even for an idealized form of unlearning: training to imitate a model that was never trained on unwanted information. This shows that training a model can drastically modify its input-output behavior while leaving its underlying capabilities intact. In light of this dynamic, we show our main result. Training a randomly initialized student on the outputs of an unlearned model transfers behaviors while leaving latent capabilities behind. In short, distillation robustifies unlearning. Based on this result, we propose Unlearn-Noise-Distill-on-Outputs (UNDO), a scalable method that distills an unlearned model into a noised copy of itself. UNDO introduces a tunable tradeoff between compute cost and robustness, establishing a new Pareto frontier on synthetic language and arithmetic tasks. At its strongest setting, UNDO matches the robustness of a model retrained from scratch with perfect data filtering while using only 60-80% of the compute and requiring only 0.01% of the pretraining data to be labeled. We also show that UNDO robustifies unlearning on the more realistic Weapons of Mass Destruction Proxy (WMDP) benchmark. Since distillation is widely used in practice, incorporating an unlearning step beforehand offers a convenient path to robust capability removal.

Distillation Robustifies Unlearning
Bruce W. Lee 12-t Addie Foote Alex Infanger' 1M Leni 1.3rl.3+
Harish Kamath" 1. Jacob Goldman- Wetzler 1.4+
Bryce Woodworth' Alex Cloud Alexander Matt Turner ML Alignment & Theory Scholars University of Pennsylvania Massachusetts Institute of Technology "Brown University
Abstract
Current LLM unlearning methods not robust. A few steps of finetuning can revert their effects. We begin by showing that this is true even for an idealized form of unlearning: training imitate model that was never trained unwanted information. This shows that training a model can drastically modify its input- output behavior while leaving its underlying capabilities intact. In light of this dynamic, we show our main result. Training randomly initialized student on the outputs of unlearned model transfers behaviors while leaving latent capabilities behind. In short, distillation robustifies unlearning. Based on this result, we propose Unlearn-Noise- Dileamm-ooise-Disilllooo Distill-on-Outpul (UNDO), a scalable method that distills an unlearned model into a noised copy of itself. UNDO introduces a tunable tradeoff between compute cost and robustness, establishing new Pareto frontier on synthetic language and arithmetic tasks. At its strongest setting, UNDO matches the robustness of model retrained from scratch with perfect data filtering while using only 60-80% of the compute and requiring only 0.01% % of the pretraining data to be labeled. We also show that UNDO robustifies unlearning on the more realistic Weapons of Mass Destruction Proxy (WMDP) benchmark. Since distillation is widely used practice, incorporating an unlearning step beforehand offers a convenient path robust capability removal.
Introduction
Large language models (LLMs) acquire harmful capabilities during pretraining massive,
largely unfiltered datasets 9 64 This complicates model deployment. For example, a model with
knowledge relevant to developing novel cyberweapons could enable global-scale harm if accessed While data filtering before pretraining could mitigate such risks, precisely
by bad actors [54] 63
auditing and filtering data the required scale remains impractical [56] 84
Post-training methods like reinforcement learning from human feedback discourage models from using undesired capabilities on common inputs 4 (55, but leave the underlying capabilities intact. As a result, the model remains vulnerable to attacks that can elicit these capabilities. including adversarial prompts, jailbreaks, direct finetuning [87 38 83). To address this vulnerability, recent work has tried using machine unlearning to remove undesired capabilities from LLMs [53] 69 6
However, existing unlearning methods also suppress capabilities rather than remove them 32 18
The supposedly unlearned capabilities can be restored through a few steps of finetuning, leaving the
fundamental challenge of achieving true capability removal unsolved 48 49
23
33
'Core contributors. Author contributions Appendix Correspondence to brucelvs@seas. upenn. edu. We share our code implementation publicly through GitHub
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Reference Model
Unlearned Model
Unlearn and Distill
Resillence Comparison
Forae: Domain Performance (CE Losal
6.5
UNDO OOurs)
MaxEnt
Gradbiff
Unlearn
Distill
8.5
SAM
RNU
Rephoise
S.
4.5
Models learn undesired capabilities during pretraining.
Unlearning suppresses bad behavior, latent capabilities remain.
Distilling new network removes latent capabilities.
200
460
Forget Reles ming Steps
Figure 1: Distillation robustifies unlearning. Existing LLM unlearning methods suppress undesired behavior, but are reversible using small amount of finetuning, We show that distilling the suppressed model into a randomly initialized network significantly increases resilience against reacquiring the undesired behavior. Our method substantially outperforms other robust unlearning baselines, including RepNoise 66 and SAM 23
We present a simple but powerful observation: distillation robustifies unlearning. When an unlearned model is distilled into a randomly initialized student, desired behavior transferred while undesired capabilities are left behind (Figure I Across diverse domains, including language, arithmetic, and weapons of mass destruction, we find that distillation consistently improves resistance adversarial relearning attacks. Remarkably, distilled models relearn harmful capabilities at rates comparable to an oracle model 225 trained from scratch with data filtering-
This finding has immediate practical implications. Distillation is commonly used in frontier AI development [30, 86). For example, LLM developers distill models reduce inference costs (74 75). By applying unlearning methods before distillation, these developers can produce models that are strongly resistant relearning attacks.
Building on this insight, we introduce UNDO (Unlearn-Noiss Diiise-Dsttil-on- -intiilloo-- which distills an unlearned model into partially noised copy of itself, enabling tradeoff between compute and unlearning robustness. With this framework, we approximate the robustness of gold-standard data filtering fraction of the cost, offering summarize our contributions as follows:
frontier in safe and scalable model deployment. We
In Section 3 we provide evidence of a limitation in behavioral training for robust unlearning in LLMs. Even when achieving perfect behavioral agreement with an unlearning oracle [25] models retain latent capabilities that can be easily restored through finetuning.
In Section 4 show that distillation robustifies unlearning. Training new, randomly initialized model on the outputs of an unlearned model will transfer desired behavior while leaving latent capabilities behind.
In Section 5
we propose UNDO, which distills an unlearned model's outputs into noised copy of itself. This method enables a novel compute-robustness tradeoff that interpolates between mere behavioral suppression and fully robust unlearning.
We benchmark UNDO against variety of unlearning methods on language and arithmetic tasks, extending the Pareto frontier of retain performance versus forget unlearning robustness. We also show competitive performance and robustification unlearning on the WMDP benchmark [42]
Robust Unlearning
Machine unlearning seeks to remove specified knowledge from trained model while preserving the
model's overall functionality. This goal has been operationalized different ways [41). We follow recent work concerned with removing undesired capabilities from LLMs in way that is robust to adversarial elicitation 667 [I]. We consider the following problem:
Robust Unlearning. A robust unlearning method accepts the following inputs: (i) Reference Model: a neural network with capabilities to be removed: (ii) Retain Ser: a collection of examples demonstrating desired capabilities; (ii) Forget Ser: collection of examples demonstrating undesired

---

capabilities: (iv) Training Data: a collection of unlabeled examples. The method produces new model that is evaluated according to (a) a Retain Evaluation: a measure of the model's desired capabilities; (b) an Elicited Forget Evaluation: a measure of the model's latent undesired capabilities. The goal is to produce a model with a high retain performance and low elicited forget performance.
Elicitation Mechanism. We use finetuning on the forget set, commonly referred to as relearning attack, as our elicitation method. Relearning attacks have repeatedly proven be among the most effective ways to elicit unlearned knowledge or capabilities in supposedly unlearned models BI 60). Therefore, recent studies treat resistance to such finetuning-
probes as the primary criterion for robust unlearning [1] 23 [6. For rigorous evaluation, we apply several relearning attacks at different learning rates and measure the maximum elicited forget performance of all attacks.
Reference vs. Oracle Models. In our framework, the reference model represents a trained model that contains both desired and undesired capabilities (e.g., pretrained LLM).. We also define an oracle model as one trained on the reference model's pretraining dataset with all content directly related to the undesirable capability perfectly filtered out. The oracle model often represents a gold standard solution to the robust unlearning problem. simulating perfect capability removal by never learning undesired capabilities in the first place 50
The capabilities- capabillties-focused definition used above differs from machine unlearning used in privacy and fairness applications [29] 190). Machine unlearning in privacy and faimess applications assumes the retain and forget sets partition the reference model's ' training data, and aims to produce an oracle model equivalent to training solely on the retain set (25). In contrast, the rapabilities-fouueel definition uses retain and forget sets as curated datasets reflecting desired and undesired capabilities, which need not correspond directly to literal subsets of the training data [13 17.. Despite this
conceptual shift, similar methods are often applied across both settings; however, these methods generally do not achieve robustness 78 34).
3 Oracle Matching Does Not Guarantee Robust Unlearning
Unlearning methods often finetune the outputs of the reference model. For example, Gradient Ascent combined with Gradient Descent maximizes cross-entropy loss on the forget set and minimizes it on the retain set (50). The aspiration of such procedures is oracle matching: attaining outputs indistinguishable from those of a model that never encountered the undesired data.
In this section, we demonstrate the limitations of this aspiration empirically. We finetune a reference model to match an oracle model's outputs on both the retain and forget sets to achieve effectively perfect behavioral agreement. Yet, when both models are subsequently finetuned on the forget set, the oracle-matched reference model relearns significantly faster than the oracle model. This shows that optimizing for behavioral outputs is likely insufficient for robust unlearning.
Datasets and Models. We conduct experiments in language setting and an arithmetic setting, defined below. For each setting, we train a reference model with both retain and forget capabilities, and an oracle model with only retain capabilities by construction. More details how these datasets and reference models were generated can be found in Appendix A
Language. The retain set is English documents from FineWebEEd [58]; the forget set is Korean documents from Webe [59]: the training data is the union of these two sets. The reference model is 100M parameter model based on the Gemma 2 architecture (see Table for details) pretrained on 2B tokens, IB from retain data and IB from forget; the oracle model is the same 100M model architecture pretrained on 1B tokens of the retain set. The retain and forget evaluations are cross-entropy loss on held-out portions of the retain and forget sets, respectively, each containing 500K tokens.
Arithmetic.
The retain set is addittion/subraaciion statements (equations and word problems) from a synthetic arithmetic dataset described in Appendix A the forget set is multiplication and division from the same dataset; the training dataset is the union of these with (English) documents from Fine WebEdu [38]: each containing 500K arithmetic questions for training. The reference model is trained for 1000 steps on the training data; the oracle model is trained for 1000 steps on the training data with the forget set replaced with additional English Fineweb data. The retain and forget evaluations measure accuracy on addiition/sbbraacioo and multiplication//iiisiii problems, using held-out portions of the retain and forget sets.
3

---

Student [[leletencc]
Student (Randan)
Gracle Teache
1.50
Dracia Matching
CE Lock on Forget Domain
(b) Releamed Loss: Language
Accuracy Forget. Domain
Relearned Accuracy: Artthmatic
Task Type
1.25
Arth methc
1.00
Language
066
0.75
0.50
0.25
0.00
200
4000
600
800
1000
100
200
300
400
500
0.0
100
200
300
400
500
Distiliation Stegs
Training Steps on Forget Domain
Training Stops on Forget Domain
Figure 2: Matching oracle behavior doesn't guarantee robust unlearning. (a) KL divergence during distillation shows behavioral alignment with Oracle Teacher. (b-c) Despite this alignment, reference models matched to the oracle (Student (Reference)) exhibit rapid relearning of undesired capabilities when finetuned on the forget set, compared to the randomly initialized model matched to the oracle (Student (Random)) and the oracle itself (Oracle Teacher). Results highlight that an ideal unlearned behavior on the surface is insufficient for ensuring robustness against relearning.
Oracle Matching Training. We match the reference model's outputs with those of the oracle model using the Kullback-Leible Divergence (KL Divergence) as the loss function. For comparison, we also match a randomly initialized model the oracle model in the same way. The hyperparameters of the distillation setup can be found in Table and the loss curves are shown in Figure 2 (a), where we use the label Student (Reference) or Student (Random) depending on whether the student the distillation setup is the reference model or the randomly initialized model, respectively. In both settings, the distillation loss tends to zero.
Relearning, After distillation, we apply relearning attacks to all three models, Student (Reference), Student (Random), and Oracle Teacher, for 500 steps on data from the forget set. The hyperparameters for this training are in Table 4 We show the corresponding learning and accuracy curves for multiple learning rates in Figure 2 (b) and (c). The bolded lines correspond to the worst-case adversary (maximum elicitation across learning rates) for that number of training steps. In both the arithmetic and language settings. Student (Reference) quickly learns the forget distribution compared to the Student (Random) and Oracle Teacher models, which are more robust. This pattern holds both for the worst-case adversary and uniformly across all learning rates.
As illustrated in Figure distilling the oracle's outputs into a randomly initialized student (Student (Random)) results model whose relearning speed on the forget distribution closely matches that of the oracle itself. Given that existing machine unlearning methods provide practical means of approximating oracle outputs, an important next question is whether this robustness to relearning remains after replacing true oracle outputs with approximations. In the next section, we demonstrate empirically that they do.
4 Distillation Robustifies Unlearning
The previous section showed that training the reference model to match oracle behavior is insufficient for robust unlearning. However, training randomly initialized model to match the same oracle outputs does produce robust unlearning effect. Here, investigate whether this robustness persists when replacing the oracle with approximations from standard unlearning methods.
Unlearning Methods. We experiment with three unlearning methods from the literature: Gradient Difference (GradDiff) [50]. Maximizing Entropy (MaxEnt) 44], and Representation Misdirection
for Unlearning (RMU) 42 These methods represent three major paradigms in LLM unlearning:
gradient- manipulation, output distribution matching, and representation-levt interventions.
Unlearning Method A: Gradient Difference (GradDiff) applies opposing gradient updates to forget and retain data. Given model parameterized by 0, GradDiff minimizes the objective L(0) =
-Lioreet (0) + Tretain (0), where both loss terms are cross-entropy losses. The gradient updates aim to maximize loss on forget examples (pushing probability mass away from undesired outputs) while minimizing loss retain examples (preserving desired model behavior).

---

Unlearning Method B: Maximizing Entropy (MaxEnt) increases uncertainty in the model's oulputs on forget data while preserving performance on retain data. MaxEnt minimizes [(0) == Tuniform (0) + (0). where Tuniform is KI, divergence term that pushes the model's output distribution toward uniformity on forget examples, and Tretain is the standard cross-entropy loss on retain examples. Rather than directly opposing learned associations as in GradDiff, MaxEnt makes the model outputs maximally uninformative on undesired tasks.
Unlearning Method C: Representation Misdirection for Unlearning (RMU) operates at the [(0) == miiisiiress (0) + peeeserve(f) (f), where Lmisdirect uses mean squared error (MSE) loss to push the model's 's internal activations at specific layers toward a random direction for forget examples, and Lereserve applies MSE to maintain similarity to the original model's 's activations for retain examples.
representation level rather than the output level. minimizes
Language (b) MaxEnt
Arithmetic (e) MaxEnt
[a] GradDiff
(cl RMU
1.0
(d) GradDiff
1.0
1.0
(f) RMU
10
Accuracy
CE
20
Forget
Retain
10
0.5
0.5
0.5
30
100
250
500
0.0 Steps
25
0.0
0.00
250
500
UntEaming
Figure 3: Comparing unlearning methods. (a-c) Unlearning trends across hyperparameters for our language setup, where we select configurations that maximize retain performance while minimizing forget performance for distillation (see Figures and 5). (d-f) Corresponding trends in arithmetic.
Experimental Setup. We investigate whether distillation can enhance the robustness of unlearning methods against relearning attacks. Our experimental protocol consists of two phases: (i) applying unlearning methods (GradDiff, MaxEnt, or RMU) to pretrained model (as shown in Figure 3 and (ii) distilling this suppressed model into a randomly initialized model of identical architecture using forward KI. divergence. We refer to this method as Unlearn-and-DD -Ditill....
We observe relearning speed when models are subjected to relearning attacks, comparing unlearning- only models, Unlcarn-and -nd---nnd-D-illl models, and oracle baselines. We defer the discussion of other relevant experimental details Appendix B
Result: Distillation makes unlearning more resilient relearning. TigureApresent the relearning curves for our experiments across both language and arithmetic domains. The plots track how quickly each model reacquires the forget capability when subjected to relearning attacks. We observe that models that underwent unlearning followed by distillation (Unlearn-and-Disine more resistant to relearning compared to unlearning-only counterparts (Unlearn). Notably, some cases, the distilled models' relearning trajectories closely approximate those of the gold standard (Data Filtering). This robustness improvement holds whether we measure average performance across learning rates (Figure 10) worst-case adversarial performance (Figure
are significantly
While RMU combined with distillation shows less impressive results in the arithmetic domain in Figure (f), it still offers marked improvement over the unlearning-ong model. We observe that such underperformance can occur when the initial retain-forge! trade-off established by the unlearning method is less favorable. In Figure 3 RMU achieved only 62% retain and 6.8% forget accuracy, compared to MaxEnt's superior 80% and 1.3% (Appendix B. This suggests that distillation effectiveness can depend on the quality of the teacher model's retain- forget tradeoff.
Why This Matters. To contextualize the experiment, we began with custom- pereta--ee language models containing dual capabilities. For the language setup, approximately 50% of the pretraining data contributed to desired (retain) capabilities and 50% undesired (forget) capabilities. Standard unlearning methods use minimal labeled data (less than 0.01% % of pretraining data) to suppress undesired capabilities almost perfectly. However, this shallow suppression is vulnerable relearning attacks. An alternative solution would be data filtering with full retraining, but this approach requires labeling all pretraining data, which is often infeasible at scale. Unlearn-and-Deee transferring the suppressed model's 's behavior to a fresh parameter space using unlabeled pretraining data for distillation, achieving substantial robustness improvements without the extensive labeling requirements of data filtering.
bridges this gap by
5

---

Unlearn (GradBilf) MazEnt (MM]
Deta Filering Standard)
(a) GradDit//aaquage
[b] MaxEntilanguage
[c) RM quaae
3.5
4.0
4.0
4.0
4.5
4.5
5.0
5.0
9
5.5
6.0
6.5
6.5
6.5
7.0
7.0
300
400
200
300
500
100
200
500
(d) GrodDim/Anthmetic
[e) MaxEntAnitmettic
(1) RMUJArithmebe
Accuracy (Faggs)
0.8
0.8
0.2
0.2
0.0
0.0
0.0
1.00
200
300
400
500
1000
200
300
500
1.00
200
400 500
Training Stops Forget
boosts robustness to relearning. (a-c) Relearning trends for the language forget domain (Korean), comparing unlearning-only methods (GradDiff, MaxEnt, RMU) against models with an additional distillation step, measured against the gold standard of full retraining. We highlight the least favorable learning curve for each method. (d-f) Relearning trends for the arithmetic forget domain (Multiplication & Division).
Figure 4: Unlearn-and-Diee
These results provide some empirical evidence supporting claims of prior work about the challenges of robust unlearning. They have argued that quick relearning of supposedly unlearned capabilities stems from latent information preserved in the parameter space [92 57 66 due to properties like
plasticity 47 and representational entanglement 993 50
Our experiments confirm that when
apply standard unlearning methods to a model, the resulting model relearns quickly under finetuning. However, when distill this unlearned model into a randomly initialized student, the student relearns substantially slower as there was no latent capability to begin with.
Extension to post-Training Unlearning. While main experiments distill into randomly initial- ized networks to remove capabilities acquired during pretraining, this approach also applies when undesired behavior is learned during post-training. In such cases, the original base model (which never possessed the undesired capability) can serve as the distillation target. In Appendix [F] we demonstrate this on the TOFU benchmark [50].
Trading Compute for Unlearning Robustness
So far, we have demonstrated that distilling unlearned model into randomly initialized model significantly enhances robustness against relearning attacks. However, this approach introduces substantial computational costs, as we training student model from scratch. Moving beyond the basic golearn-and-Dee be adapted to achieve different levels of robustness with different amounts of compute. That is, can we trade off robustness to reduce compute costs.
method established in Section 4 we now investigate whether the method
In this section, we conduct experiments that generalize the Unlearn-ande -nd---inil-lll approach in Section 4 with three-step process. (i) Unlearn: create a behaviorally suppressed model (teacher) by applying standard unlearning methods. (ii) Noise: create student model by perturbing the weights of the suppressed model, damaging it. (iii) Distill: repair this damaged student by distilling to recover
interpolate between the original teacher's original behavior. The controlled perturbation enables Diseram--iiise-ssillnoo------- Distill-on-Outputs parameters and full reinitialization. We refer to this method as Unlearn-Noise- simply, UNDO. We define the perturbed model as:
Opentubedd
suppprsseedd + &BN
where mixing coefficient E [0. 1). noise scale 3 E R+, and N represents sampled noise. We sample using Xavier initialization, though other random noise sampling methods could be explored. Intuitively, the term (1 a auppuppssees shrinks [2] the suppressed model's parameters, and the term

---

aBN injects noise. As we vary a from 0 to 1, we can view Openturbed interpolating between the original paramterization and full reinitialization. We observe later in Section that simply shrinking the parameters (setting 3 = O) can also be effective, as the key idea is to globally damage the network to varying degrees, though including 3 adds expressivity to the formulation. Note that when a = and / = 1, the perturbation reduces to random initialization.
Ex Experimental Setup. We start by perturbing the models that are suppressed with MaxEnt from Section 4 and follow the same protocols for distillation using forward KL divergence on the outputs. For both language and arithmetic settings, we experiment with various values of while fixing 3 = 0.1. For each perturbation level, we perform distillation until the model reaches at least 95% of the teacher model's performance on the retain set or completes training on one full epoch of pretraining data, whichever comes first. Low-level experimental details are available in Appendix C
13) Computo Trade Off Language
JbJ Seleamed Loss: Language
(CI Compute Tade Off. Artthmetic
Id) Relcared Loss: Artthmatic
12095
0.00
CE Lode on Forget Dumaln
Acouracy on Forget orgee Domsin
5.8
0.75
80%
100%
0.70
Robustness
Rohosimess
80%
80%
0
Model Type
40%
0.45
Model type UNDO
2014
Unieam
0.40
UNDO
20%
Dota fitering iGold Stanubrd) di
Duta Ritoring 155
(Gold Standard)
20% Compute 1%
sots
50% 120% Aittrinn!
225
250
Trining
Fitering)
Tratning
Figure 5: Unlearning robustness scales with more perturbation. (a, c) UNDO scaling trend for between 0.1 and 0.8 and / = 0.1, showing trade-off between robustness measured as (PUNDO Puuleam Only) (Paata Filtering - Unlearn Only) where P is forget performance, and compute measured as SUNDO/ D Filtering where S is training steps. Points denote median values, error bars show variation across five random seeds. (b) Relearning trends for Korean domain with a = {0.2, 0.4, 0.6, 0.8). (d) Relearning trends for Multiplication & Division with {0.55, 0.65, 0.7, 0.75).
Result: UNDO enables trading off compute for robustness to relearning. Figure demonstrates that increasing the perturbation parameter a enhances robustness against relearning attacks while requiring proportionally more computation recover the teacher model's performance on the retain set. This relationship appears roughly linear for language unlearning in Figure 5 (a) and exhibits steeper initial curve for arithmetic unlearning in Figure (c). Figures (b, d) show varying degrees of resistance to relearning across different values, with higher values consistently demonstrating improved robustness in both domains. For language unlcarning, even relatively small perturbations (a = 0.2) produce noticeable improvements in relearning resistance, while arithmetic unlearning shows meaningful improvements at higher perturbation levels (a 2 0.55).
Modtl Tre Maetir
Hoddd hpet USDO
URD0 poiidb
UNDO 1-1.1
USD0 3-03
USBO p-1
(3) Relcamed
(b) Roleamed
Releamed UNDO 0.5)
Relcamed UNDO 0.71
Stops UNID (lecover 55% furiai))
UNDD 0.11
UNDO
Donst
300
3.00
250
220
Taining Steps Fonget Domain
Figure 6: Effeets of varying a and noise parameters. (a-d) Relearning curves for UNDO models with fixed mixing coefficient a = 10.1, 0.3, 0.5, 0.7) while varying noise scale 3 [0.05, 0.1, 0.5, 1.0). Higher mixing coefficients (a) consistently produce more robust unlearning (flatter curves), while the effect of noise scale (8) is more nuanced. (e) Computational cost matrix showing distillation steps required to recover within 5% of the teacher's retain performance across a and 3 combinations.
We test 0.2, 0.4, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8) for language and arithmetic tasks.
7

---

To learn more about the interplay between mixing coefficient (a) and noise scale (5), we conduct a hyperparameter sweep as reported in Figure 6 Changes in mixing coefficient (a) produce more pronounced effect on robustness than the noise scale (3), which has less significant impact. This pattern holds across all settings where models are distilled to within +5% of the teacher's retain performance. While incorporating standard noise from Xavier initialization can help improve robustness for very low a values (a = 0.1), we find that for most a values tested, the specific value of [ had less impact. Interestingly, we observe that using lower 3 values (0.05, 0.1, or 0.5) typically worked similarly well or better than = 1.0 in our language setup. The computational cost matrix in Figure o (e) confirms that a is the primary driver of compute requirements, with higher values often requiring more distillation steps.
Comparisons with Other Unlearning Methods
We now compare UNDO against existing robust unlearning methods evaluate their effectiveness across two key metrics, retain performance and resistance to relearning. The ideal unlearning method would preserve high performance on desired tasks while remaining resistant adversarial attempts to recover forget capabilities through finetuming. We first compare diverse unlearning methods in our language and arithmetic tasks in Figure subjecting models to increasingly stronger adversarial relearning attacks.
We also compare against several approaches that address the robust unlearning problem using different mechanisms to resist relearning attacks. Sharpness-Awar Minimization (SAM) 23 optimizes for flat loss regions where small parameter perturbations maintain similar outputs, making the model less sensitive to finetuning. Representation Noising (RepNoise) [6] combines unlearning with noise injection at the representation level, creating interference in the model's s internal representations of forget capabilities. UNDIAL [9] also uses distillation for unlearning but differs from our approach by not applying global parameter corruption techniques like noising. For UNDO, we fix a to 0.6 and to 0.1, and distill from a MaxEnt-suppresse points along the retain-forget trade-off curve. Figure 3 suggests that higher values would yield greater robustness, while lower a values would yield less. We discuss other relevant experimental details in Appendix D
model with varying retain threshold to obtain different
Data Fitaring (Gold Standard)
UNDO
ManErt
GradDif
A RMU
Rashass
SAM
UNDIAL
No Adversarial Finetuning
Low-Compute Adversary
High-Compute Adversary
(al Retain- Pettain-Forrete Forget: Language INo Relesrning!
Forget Language (after Stepsi
[cl
Retain- Forgat: Language (after 500 Steps)
Opemal
Optiroal
Optimal
CC Loss (latain
Natdaan Stuncc
7-
3.0
(Forget)
Loss (Forget Retearnin)
Loss (Forget Relearning!
(d) Ratain-Farget: Artthmatic INo Releaming!
[e) Retain-Forget. Arttmmetic (after Stops)
3D
Retain-Forgat: Arithmetic (after Steps)
1.0
1.0
1.0
Accunrr (Batein)
0.8
a
0.6
0.6
0.6
0.4
0.4
0.4
46
0.0
0.0
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
Accuracy (Forget)
Accuracy iFrgeet after Releaming))
Accuracy iForget after Relesrnin))
Figure 7: Comparing unlearning methods across different adversarial strengths. (a, d) Initial performance after unlearning but before adversarial attacks. (b, e) Relearned forget performance after moderate relearning (40 steps). (c. Performance after extensive relearning (500 steps).
As Figure 7 shows, while more compute-efficienn unlearning methods (GradDiff, MaxEnt) achieve good initial forget trade-offs before relearning attacks, they rapidly degrade under adversarial pressure. In general, we observe that methods designed for robustness (SAM, RepNoise) also show
We test (0.05, 0.09, 0.13, 0.17, 0.21, 0.25, 0.29, 0.33) retrain thresholds for language and arithmetic tasks.

---

significant performance deterioration when subjected to stronger relearning attacks. In contrast, UNDO with MaxEnt maintains more robust unlearning performance across all explored attack strengths. This creates a new Pareto frontier that approaches the gold standard of data filtering with full retraining but requires less compute and data labeling-
SAM
Ropholse
UNDO
MaxEnt
(a] MoXEntWWDDPPP -Bio
(b) RMU/WMDP-Bio
(c) MaxEntWMDP-Cyber
(d) RMUUWDP-CCybr
Acaracy (Forget
0.50
0.34
0.45
0.5
0.40
0.35
0.30
0.30
400
0.3
0,28
400
Adversarial Releaming Steps
Figure 8: UNDO makes MaxEnt and RMU more resilient to relearning WMDP. Relearning trends averaged across 4 seeds for WMDP-Bi (a, b) and WMDP-Cyben (c, d). The retain/forget performance was measured on MMLU/the WMDP benchmark. Best adversaries are bolded.
Moving beyond our synthetic language and arithmetic unlearning tasks, we also test methods on the more realistic Weapons of Mass Destruction Proxy (WMDP) benchmark (42). For this setup, we use Gemma-2-2B 77] rather than custom pretrained models. Following our established methodology, we first apply standard unlearning using either MaxEnt or RMU, then apply UNDO with a 0.25 and 0. We compare this approach against the other robust unlearning baselines. We evaluate robustness against seven diverse adversarial scenarios involving different data mixtures, question-answer answer formats, model perturbations, and learning rate variations (See Appendix (D2)..
Figure X shows that UNDO consistently improves relearning resilience across all WMDP configura- tions. These results have immediate practical implications. When model distillation is already planned for efficiency, incorporating unlearning before distillation provides robustness benefits at minimal additional cost. This approach integrates capability removal into existing distillation workflows, providing practical path to robust unlearning.
Limitations. In our WMDP experiments on pretrained Gemma-2-2B model, the UNDO model achieves 4.88% lower on MMLU on average. While this places UNDO on the Pareto frontier, other methods achieve similar trade-off between robust forget and retain performance in this setting, as shown in Figure II This contrasts with the prior arithmetic and language setting results, where we see clear gains the Pareto frontier in Figure 7 We hypothesize this difference comes from the fact that we undertrain in the WMDP setting, distilling on only 0.015% of the size of the original pretraining corpus compared to higher percentages in the other synthetic settings. We discuss this more in Appendix [D.2.7 We expect that model developers would not encounter the same challenges and would have sufficient access compute, pretraining data, and their proprietary methods to accelerate the process. Distillation is considerably more costly than finetuning- unlearning methods, but we expect this cost to be justified when robust capability removal is critical.
7 Related Work
Machine unlearning aims to remove the influence of specific training examples from trained model. This term was first introduced by Cao and Yang [0. which explored adjusting a trained model's outputs to mimic retraining without the data to be forgotten. Machine unlearning has been a significant research focus for diffcrential privacy, implementing the "right to be forgotten" [62]
through frameworks that ensure removal of specific datapoints [39 3
28 46
77
The field has 79]. Exact
evolved into exact unlcarning [8] 24 14 52] and approximate unlcarning 27 65
unlearning targets that the unlearned model's parameters exactly match those of model retrained
from scratch without the data to be forgotten (26), while approximate unlearning relaxes this constraint to require only probabilistically similar output distributions [70].
For LLMs, exact unlearning becomes impractical due to scale and non-convexity 44 leading recent studies to frame unlearning as an optimization problem that approximates the behavior of the retrained model [21 50
88 8T. L.MM unlearning typically involves finetuning using
objectives like maximizing prediction loss on forget sets [35 89 5 22 92 or aligning predicted

---

Recent studies demonstrate that existing LLM unlearning methods achieve behavioral suppression rather than true remaining vulnerable to finetuning attacks that quickly recover suppressed This motivates our student-leacher approach [5 for robust unlearning.
token distributions with target distributions using KL divergence 40, 82 12 80
unlearning [32
92
76
knowledge
33
18
8
Conclusion
There is a disconnect between what a model does and what it could do. Our findings illuminate this gap in two ways: first, by showing that even oracle-matched models retain latent capabilities vulnerable to adversarial elicitation; and second, by demonstrating that distillation discards these capabilities while preserving desired behavior. This insight transforms distillation, standard practice in LLM development, into a security measure, offering practical path to robust capability removal.
Acknowledgments
We thank Henrik Marklund for very insightful discussions. We are grateful Rishub Tamirisa for the guidance in navigating WMDP benchmarking procedures and to Eric Easley for sharing valuable strategies for WMDP dataset cleaning and productive discussions about potential improvements to our method. Vivek Hebbar, Andis Draguns, and Jake Mendel offered helpful comments on the abstract. We thank Iftekhar Uddin and Laura Vaughan for facilitating access to computational resources and funding support. We thank Lisa Wong for help with the design of Figure 1. We thank four anonymous reviewers their time spent on our manuscript and for providing constructive feedback.
Bruce W. Lee's work on this project was partially supported by funding from Open Philanthropy. We also thank MATS for facilitating our collaboration and providing the institutional framework that made this research possible.
References
[1] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint vrXiv:2404.09933 2024.
[2] Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884 -3944, 2020.
[3] Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what the question? Advances in Neural Information Processing Systems, 155117953-1796 2022.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 2022.
[5] George-Octin Octavian Bärbulescu and Peter Triantafillou. To each (textual sequence) its own: improving memorized-danl unlearning in large language models. In Proceedings of the 41st International Conference on Machine Learning, pages 3003-3023, 2024.
[6] Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O' O'Gaa,, Robert Kirk, Ben Bucknall, Tim Fist, et al. Open problems in machine unlearning for safety. arXiv preprint 2rX2v22501.04952, 2025.
[7] Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi. Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable Im 1.6 technical report. arXiv preprint 242 2024.
[8] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Chon Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 202/ IEEE symposium security and privacy (SP), pages 141-159. IEEE, 2021.
10

---

[9] Dillon Bowen, Brendan Murphy, Will Cai, David Khachaturov. Adam Gleave, and Kellin Pelrine. Data poisoning in llms: malbreak-tuning and scaling laws, arXiv preprint arXiv:2408.022 02946. 2024.
[10] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privucy, pages 463-480. IEEE, 2015.
[11] Zora Che, Stephen Casper, Robert Kirk, Anirudh Satheesh, Stewart Slocum, Lev E McKinney, Rohit Gandikota, Aidan Ewart, Domenic Rosati, Zichu Wu, et al. Model tampering attacks enable more rigorous evaluations of Ilm capabilities. arXiv preprint arXiv:2502.05209 2025.
[12] Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for lmss. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12041-12052, 2023.
[13] Minseok Choi, Daniel Rim, Dohyun Lee, and Jaegul Choo. Snap: Unlearning selective knowl- edge in large language models with negative instructions. arXiv preprint 2024.
[14] Rishav Chourasia and Neil Shah. Forget unlearning: Towards true data-deletion in machine learning. In International conference on machine learning, pages
PMLR, 2023.
[15] Vikram S Chundawat, Ayush Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7210-7217, 2023.
[16] Alex Cloud, Jacob Goldman-Wetie Evžen Wybitul, Joseph Miller, and Alexander Matt Turner. Gradient routing: Masking gradients to localize computation in neural networks, 2024. URL https: httts://arxiv.. org/ absa 04332
[17] Thomas De Min, Subhankar Roy, Stéphane Lathuilière, Elisa Ricci, and Massimiliano Mancini. Group-robust machine unlearning. arXiv preprint 1xXiv:2503309330. 2025.
[18] Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.0 08827, 2024.
[19] Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. Undial: Self-distillation with adjusted logits for robust unlearning in large language models. arXiv preprint arXiv:2402. 10052, 2024.
[20] Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum, Zachary C Lipton, J Zico Kolter, and Pratyush Maini. Openunlearning: Accelerating Ilm unlearning via unified bench- marking of methods and metrics. arXiv preprint arXiv:2506. 12618, 2025.
[21] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In The Tweifth International Conference on Learning Representations, 2023.
[22] Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for Ilm unlearning. arXiv preprint arXiv:2410.07163. 2024.
[23] Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. To- wards Ilm unlearning resilient relearning attacks: sharpness-aware minimization perspective and beyond. arXiv preprint arXiv:2502.05374 2025.
[24] Sanjam Garg, Shafi Goldwasser, and Prashant Nalini Vasudevan. Formalizing data deletion in the context of the right to be forgotten. Annual International Conference on the Theory and Applications of Cryptographic Techniques, pages 373-402. Springer, 2020.
[25] Kristian Georgiev, Roy Rinberg, Sung Min Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, and Seth Neel. Machine unlearning via simulated oracle matching. In The Thirteenth International Conference Learning Representations, 2025.

---

[26] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances neural information processing systems, 32, 2019.
[27] Chuan Guo, Tom Goldstein, Awni Hamnun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv:1911.088. 10t.0000,,,00,, 2019.
[28] Jamie Hayes, Ili Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. arXiv preprint arXiv:2403.01218. 2024.
[29] Emmie Hine, Claudio Novelli, Mariarosaria Taddeo, and Luciano Floridi. Supporting trustwor- thy ai through machine unlearning. Science and Engineering Ethics, 30(5):43, 2024.
[30] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 5503 2015.
[31] Felix Hofstätter, Teun van der Jayden Teoh, Henning Bartsch, and Francis Rhys Ward. The elicitation game: Evaluating capability elicitation techniques. arXiv preprint a2X10:2502.02180, 2025.
[32] Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. Intrinsic evaluation of unlearning using parametric knowledge traces. arXiv preprini arXiv:2406. 1161,, 2024.
[33] Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Unlearning or obfuscating? jogging the memory of unlearned LLMs via benign relearingg The Thirteenth International Conference on Learning Representations, 2025.
[34] Mark He Huang, Lin Geng Foo, and Jun Liu. Learning to unlearn for robust machine unlearning. In European Conference Computer Vision, pages 202-219. Springer, 2024.
[35] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 6/st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389-14408, 2023.
[36] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances Neural Information Processing Systems, 36:51584-51605 2023.
[37] Jinghan Jia, Yihua Zhang, Yimeng Zhang. Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of recond-order optimization for Ilm unlearning. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 4276 4292, 2024.
[38] Erik Jones, Anca Dragan, and Jacob Steinhardr. Adversaries can misuse combinations of safe models. arXiv preprint arXiv:2406. 0ete24406.4555. 14595, 2024.
[39] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions.
In International conference on machine learning, pages 1885-1894 PMLR, 2017.
[40] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
[41] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards un- bounded machine unlearning. Advances in neural information processing systems, 36: 1957 1987, 2023.
[42] Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski. Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint trXiv:2403.03218 2024.
[43] Yuyuan Li, Chaochao Chen, Xiaolin Zheng, Yizhao Zhang, Biao Gong, Jun Wang, and Linxun Chen. Selective and collaborative influence function for efficient recommendation unlearning. Expert Systems with Applications, 234: 121025, 2023.
12

---

[44] Sijia Liu, Yuanshun Yao. Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pages 1-14, 2025.
[45] Zhenhua Liu, Tong Zhu, Chuanyuan Tan, and Wenliang Chen. Learning to refuse: Towards mitigating privacy risks in llm. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1683-1698, 2025.
[46] Zheyuan Liu, Guangyao Dou, Eli Chien, Chunhui Zhang. Yijun Tian, and Ziwei Zhu. Break- ing the trilemma of privacy, utility, and efficiency via controllable machine unlearning. In Proceedings of the ACM Web Conference 2024, pages 2660-221, 2024.
[47] Michelle Lo, Fazl Barez, and Shay B Cohen. Large language models relearn removed concepts. In Findings of the Association for Computational Linguisties ACL 2024, pages 8306-8322 2024.
[48] Jakub Lucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for ai safety. arXiv preprint arXiv:2409. 18025, 2024.
[49] Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402. 16835, 2024.
[50] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and Zico Kolter. Tofu: A task of fictitious unlearning for llms. First Conference on Language Modeling, 2024.
[51] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL htttttttts///iiii wttp:vv..orrraba//0/o0 0000b//44199007833333
[52] Seth Neel, Aaron Roth, and Saeed Sharif-Malvajerdi. Descent-to-delete: Gradient-basee methods for machine unlearning. In Algorithmic Learning Theooy, pages 931-962. PMLR, 2021.
[53] Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning, arXiv preprint arXiv:2209.02299, 2022.
[54] OpenAl. Preparedness framework (beta). Technical report, OpenAI, December 2023. URL https://cdn. openai ccwm/openaee wpppennippppprredeeessssseeessse www.ework----ooee-eeorwwwwrawoee wwttweoo.koo.ttee..oe...ee....ee pof
[55] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
[56] Yijun Pan, Taiwei Shi, Jieyu Zhao, and Jiaqi W Ma. Detecting and filtering unsafe training data via data attribution. arXiv preprint arXiv:2502. 1, 2025.
[57] Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. The Tweifth International Conference on Learning Representations, 2024.
[58] Guilherme Penedo, Hynek Kydlfček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data scale, 2024. URL URLthttps: httpttttttttiiiiii org aaaaaa///2400 17557
[59] Guilherme Penedo, Hynek Kydlfček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro Werra, and Thomas Wolf. Fineweb2: A sparkling update with 1000s of languages, December 2024. URL fineweb-2
co blataaets//uuggingaaaaaat
(60] Timothy Qian, Vinith Suriyakumar, Ashia Wilson, and Dylan Hadfield-Menell. Layered unlearning for adversarial relearning. arXiv preprint arXiv:2505.09500 2025.
13

---

[61] Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, Subhabrata Majumdar, Hassan Sajjad. Frank Rudzicz, et al. Representation noising: defence mechanism against harmful finetuning. Advances in Neural Information Processing Systems, 37:12636- 12676, 2024.
[62] Jeffrey Rosen. The right to be forgotten. Stan. L. Rev. Online, 64:88, 2011.
[63] Jonas B Sandbrink. Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools. arXiv preprint arXiv:2306. 13952, 2023.
[64] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36:55565-55558 2023.
[65] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:1 15:11885588886 2021.
[66] Atakan Seyitoğlu, Aleksei Kuvshinov, Leo Schwinn, and Stephan Günnemann. Extracting unlearned information from Ilms with activation steering. In Neurips Safe Generative Al Workshop 2024, 2024.
[67] Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. Latent adver- sarial training improves robustness to persistent harmful behaviors in 1lms. arXiv preprint 2rXiv:2407.15549 2024.
[68] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[69] Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowledge unlearning for llms: Tasks, methods, and challenges. arXiv preprint arXiv:2311.15766 2023.
[70] Vinith Suriyakumar and Ashia C Wilson. Algorithms that approximate data removal: New results and limitations. Advances in Neural Information Processing Systems, 35:18892-18903, 2022.
[71] Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, et al. Tamper-resistant safeguards for open-weight Ilms. arXiv preprint arXiv:2408.0076. 2024.
[72] Calvin Tan and Jerome Wang. 1.5-pints technical report: Pretraining in days, not months-your language model thrives on quality data, arXiv preprint arXiv:2408.03350 2024.
[73] Gemma Team. Gemma 2 2b, 2024. ORR.aatpss//uugiinfaee..///o// gew.:////gllllgggggmma.ea..a wwwpe/ggemmmaammm 2- 2b Hugging Face Model Repository.
[74] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint 27X.v:2408.00118. 2024.
[75] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
[76] Pratiksha Thaker, Shengyuan Hu, Neil Kale, Yash Maurya, Zhiwei Steven Wu, and Virginia Smith. Position: Illm unlearning benchmarks weak measures of progress. arXiv preprint 1rXiv:2410.02889, 2024.
[77] Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 3/st USENIX security symposium (USENIX Security 22), pages 4007-4022, 2022.
14

---

[78] Yiwen Tu, Pingbang Hu, and Jiaqi Ma. Towards reliable empirical machine unlearning evalua- tion: game- theeetheeretic view. arXiv preprint arXiv:2404.11577 2024.
[79] Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pages 4126-4142. PMLR, 2021.
[80] Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Rkld: Reverse kl-divergence- based knowledge distillation for unlearning personal information in large language models. arXiv preprint arXiv:2406.0/1 01983, 2024.
[81] Huazheng Wang, Yongcheng Jing, Haifeng Sun, Yingji Wang, Jingyu Wang, Jianxin Liao, and Dacheng Tao. Erasing without remembering: Safeguarding knowledge forgetting in large language models. arXiv preprint wxXi:2552.19988 2025.
[82] Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng. Kam-Fai Wong, and Hongzhi Yin. Kga: A general machine unlearning framework based on knowledge gap alignment. arXiv preprint arXiv:2305.0655. 06535, 2023.
[83] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36:80079-800 1000779-0110,, 2023.
[84] Boyi Wei, Weijia Shi, Yangsibo Huang, Noah A Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, and Peter Henderson. Evaluating copyright takedown methods for language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2024.
[85] Xiaoyu Xu, Xiang Yue, Yang Liu, Qingqing Ye, Haibo Hu, and Minxin Du. Unlearming isn' 't dele- tion: Investigating reversibility of machine unlearning in 1lms. arXiv preprint arXiv:2505. 16831, 2025.
[86] Chuanpeng Yang, Yao Zhu, Wang Lu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen. Survey on knowledge distillation for large language models: methods, evaluation, and application. ACM Transactions Intelligent Systems and Technology, 2024.
[87] Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02944 2023.
[88] Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8403 8419, 2024.
[89] Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.1066 2023.
[90] Dawen Zhang, Shidong Pan, Thong Hoang, Zhenchang Xing, Mark Staples, Xiwei Xu, Lina Yao. Qinghua Lu, and Liming Zhu. To be forgotten or to be fair: Unveiling fairness implications of machine unlearning methods. Al and Ethics, 4(1 (339999,,, 2024.
[91] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.023 022385, 2024.
[92] Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Catastrophic failure of Ilm unlearning via quantization. arXiv preprint 241i 16454, 2024.
[93] Kairan Zhao, Meghdad Kurmanji. George-Octavian Bärbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 1T::12293-1233,, 2024.
15

---

Appendices
Experimental Details for Section 3
17
B Experimental Details for Section 4
19
B.1 Unlearning Learning Rate Sweeps
20
Distillation
Experimental Details for Section
22
Experimental Details for Section
23
D.I
Language and Arithmetic Unlearning
23
D.2 WMDP
25
D.2.1
Evaluation/Benchmarr
25
D.2.2
Datasets
25
D.2.3
Baseline Unlearning Methods
26
D.2.4
Distillatio/UNDDO
26
D.2.5
Relearning Adversaries
26
D.2.6
Baseline Robust Methods
28
D.2.7
WMDP Pareto Frontier and Discussion
28
Compute requirements
29
Revt-R Reflection: An Alternative Setup to Study Unlearn-and-Disse
30
TOFU Experiments
Robustness Beyond Relearning Atlacks
32
Author Contribution
33
16

---

A Experimental Details for Section 3
In Section we present oracle matching experiments. Here, present the specific experimental details used to obtain the results presented.
Dataset Our process starts from collecting datasets to facilitate our language and arithmetic unlearn- ing experiments. For language experiments, we utilize two sources: (1) HugiingFaceFFiinnewch 1
Korean subset, providing non-English language examples, and (2) BuaggingFaceF//innewwwe.
containing English language examples with high-quality content [59 58] We sample million rows from each source.
For arithmetic experiments, we generate million arithmetic examples covering four basic operations: addition, subtraction, multiplication, and division. We control the range of operands, using integers from 50 for addition and subtraction, and integers from 20 for multiplication and division. For division problems, we ensure clean division with no remainders by constructing problems where the dividend a product of the divisor and quotient.
The arithmetic dataset consists of two formats:
Equation format: Direct mathematical expressions (e.g.. "12 + 7 19", "36 9 4")
Word problems: Natural language scenarios generated using templates with randomized variables including names, objects, places, and quantities
Word problems are constructed using templates. For example, addition problems might involve combining collections objects ("Emma has marbles and Jack has 13 marbles. How many marbles do they have altogether?"). Division problems might involve distributing items ("There are 24 cookies in container. each student gets six cookies, how many students will receive cookies?").
Both language and arithmetic datasets have separate test sets. For language data, we allocated 1. 1,000,000 tokens for validation, which consists of 500,000 tokens for the retain domain (English) and 500,000 tokens for the forget domain (Korean). For arithmetic data, we allocated 1, 1,000,000 questions for validation, which consists of 500,000 questions for the retain domain (addition and subtraction) and 500,000 questions for the forget domain (multiplication and division).
Data Processing All datasets are processed using the Google Gemma-2-2b tokenizer. Documents are chunked to maximum length of 2,048 tokens, with documents shorter than 50 tokens being filtered out. Processing is done in parallel using multiple CPU cores to tokenize and format the data efficiently. We use a fixed random seed of 42 for data sampling and shuffling to ensure reproducibility.
Model Architecture We scaled-down versions of the Gemma-2 architecture for our language and arithmetic unlearning experiments. The model architectures and their key parameters are detailed in Table
Parameter
Gemma-2-0.1B
Gemma-2-44 2-0.33
Total Parameters
~ 100M
~000
Hidden Layers
14
14
Attention Heads
8
Hidden Size
320
Intermediate Size
1280
3072
Head Dimension
128
128
Table I: Model Architecture Parameters
The Gemma-2-0.11 model is used for the language unlearning experiments, while the Gemma-2-0.33 model is used for the arithmetic unlearning experiments. We intentionally vary this show results across different model sizes. We chose these parameters based on the ratio used in the original Gemma-2 architecture maintain the model's core properties.

---

Pretraining We then pretrain our Oracle and Base models. Table 2 details the hyperparameters
used during pretraining. We reference these hyperparameters from 72 91
Language Pretraining
Arithmetic Pretraining
Hyperparameter
Base
Oracle
Base
Oracle
Model Architecture
Gemma-2-0.1B
Gemma-2-0.1D
Gemma-2-0.3B
Gemma-2-0.3B
Parameters
100M
100M
300M
300M
Training Data
English + Korean
English
All Ops + English
Add/Sub + English
Data Mixture Ratio
1:1
1:3
1:7
Learning Rate
4e-4
4e-4
4e-4
4e-4
Min Learning Rate
4e-5
4c-5
4e-4
4c-4
Batch Size
15
Gradient Accum. Steps
80
80
24
24
Weight Decay
0.1
0.1
0
0
Epochs
1
Maximum Steps
1000
1000
LR Schedule
Cosine
Cosine
Cosine
Cosine
Warmup Steps
50
50
50
50
Max Sequence Length
2048
2048
256
256
Table 2: Pretraining Hyperparameters
For the language models, the Base model is trained both English and Korean data with equal probabilities, while the Oracle model is trained exclusively on English data. Both models used identical optimization parameters. For the arithmetic models, the Base model is trained on all four arithmetic operations (addition, subtraction, multiplication, division) mixed with English text at a 1:3 ratio. The Oracle model is trained only on addition and subtraction operations mixed with English at 1:7 ratio. We end up using ~ 1B pretraining tokens for Gemma-2-0.1 (Language, Oracle), 2B tokens for Gemma-2-0. (Language, Base), ~ 100M tokens for Gemma-2 2-0..BB (Arithmetic, Oracle), ~ 100M tokens for Gemma-2-0.3B (Arithmetic, Base).
All models trained using the AdamW optimizer with cosine learning rate scheduling. We used a shorter sequence length (256 tokens) for arithmetic models compared to language models (2048 tokens) due the inherently shorter nature of arithmetic problems.
Oracle Matching We implemented oracle matching using knowledge distillation with forward KL divergence loss (30) on the pretraining dataset, also explored in Gemma 2 technical report [74). As a), we examined two configurationss (1) distillation from an Oracle teacher to randomly initialized model and (2) distillation from Oracle teacher to pretrained Base model. Table 3 presents the hyperparameters for these distillation procedures.
illustrated in Figure ((a)
Hyperparameter
Language Distillation
Arithmetic Distillation
Teacher Model
Gemma-2-0.11 (Oracle)
Gemma-2-0.3B (Oracle)
Student Model
Gemma-2-0.1 1B (Base or Rand Init)
Gemma-2-00 (Base or Rand Init)
Distillation Data
English Korean
All Ops English
Data Mixture Ratio
Learning Rate
4e-4
7e-4
Min Learning Rate
4e-5
7e-5
Batch Size
4
15
Gradient Accum. Steps
160
24
Weight Decay
Maximum Steps
1000
Epochs
LR Schedulc
Cosine
Cosine
Warmup Steps
50
50
Max Sequence Length
2048
256
Table Oracle Matching Hyperparameters
18

---

Relearning To evaluate the robustness of unlearning methods, we conduct relearning experiments to assess whether unlearned capabilities could be easily recovered. Notably, finetuning often considered strong form of latent capability elicitation, so it can provide insights into whether knowledge is truly removed from the model or just suppressed during oracle matching. For all relearnnng experiments, we use the final student model from the oracle matching procedure as the starting point. Training proceeds for 500 steps, with validation performed every five steps to track both retention and recovery of capabilities. Table 4 presents the hyperparameters used for our relearning experiments.
We assume significantly stronger adversary compared to contemporary works in LLM unlearning, and show that our methods introduced in Sections 4 and effective across weaker and stronger adversaries. Our Language Relearning experiments train ~88 tokens, and Arithmetic Relearning experiments train ~2M tokens. On top of this, we do an adversarial learning rate search for all relearning runs over [1e-3, 7e-4, 4e-4, 1 le-4, 7e-5, 5e-5, 1e-5, 7e-6, 4e-6, 1e-6). In all relearning experiments (Figure Figure Figure 5). we always mark the adversarially optimal performance (the best performance the adversary could obtain by varying the learning rate) at each validation step thicker lines.
Hyperparameter
Language Relearning
Arithmetic Relearning
Relearning Data
Korean
Mul/Div English
Data Mixture Ratio
1:3
Batch Size
8
15
Gradient Accumulation Steps
Maximum Steps
500
500
Warmup Steps
Learning Rate
[le-3, 7e-4, 4e-4, 1e-4, 7e-5, 5e-5, 1e-5, 7e-6, 4e-6, 1e-6] 5
Evaluation Steps
5
Max Sequence Length
2048
256
Table 4: Relearning Hyperparameters
B Experimental Details for Section
In Section 4
we report unlearn and distill experiments. Many experimental details similar what
was reported in Appendix A
Dataset / Data Processing/ Model Architecture/ Pretraining/ Relearning See Appendix A
Unlearning Methods In Figure 3 we report unlearning trends for three methods: GradDiff, MaxEnt, and RMU. To perform hyperparameter search, we first manually search for the key hyperparameters (see T5)
Language Unlearning
Arithmetic Unlearning
Hyperparameter
GradDiff
MaxEnt
RMU
GradDiff
MaxEnt
RMU
Forget Data
Korean
Korean
Korean
Mult/Div
Mult/Div
Mult/Div
Retain Data
English
English
English
Add/Sub
Add/Sub
Add/Sub
Batch Size
4
4
4
40
40
40
Max Steps
100
100
500
10
10
500
Max Length
2048
2048
2048
256
256
256
Alpha
1200
200
RMU Layers
5-11
5-11
End Layer
11
11
6.5
Table 5: Unlearning Hyperparameters
19

---

B.1 Unlearning Learning Rate Sweeps
We do learning rate sweeps across eight learning rates per method. We choose the one that gives the best trade-off between the retain and forget performance. The chosen values (marked with thick lines in Figure 3 are marked in bold below.
GradDiff (Language, CE Loss)
le-5
Initial Retain: 4.9359 + Final Retain: 5.0264 Initial Forget: 3.4668 + Final Forget: 30.5592
2e-5
Initial Retain: 4.9359 + Final Retain: 4.9924
Initial Forget: 3.4668 - Final Forget: 31.5783
3e-5
Initial Retain: 4.9359 + Final Retain: 4.9967
Initial Forget: 3.4668 + Final Forget: 32.3159
4e-5
Initial Retain: 4.9359 + Final Retain: 5.0085 Initial Forget: 3,4668 + Final Forget: 32.9097
5e-5
Initial Retain: 4.9359 + Final Retain: 5.0216
Initial Forget: 3,4668 - Final Forget: 33.5308
6e-5
Initial Retain: 4.9359 + Final Retain: 5.0334 Initial Forget: 3.4668 - Final Forget: 34. 1291
7e-5
Initial Retain: 4.9359 + Final Retain: 5.0563 Initial Forget: 3.4668 - Final Forget: 34.685
8e-5
Initial Retain: 4.9359 Final Retain: 5.0762 || Initial Forget: 3.4668 Final Forget: 35.129
GradDiff (Arithmetic, Accuracy)
6e-6
Initial Retain: 0.875 + Final Retain: 0.9125 II Initial Forget: 0.8675 + Final Forget: 0.8375
7
Initial Retain: 0.875 - Final Retain: 0.9075 II Initial Forget: 0.8675 - Final Forget: 0.8325
8e-6
Initial Retain: 0.875 - Final Retain: 0.815 Initial Forget: 0.8675 Final Forget: 0.015
9e-6 Initial Retain: 0.875 Final Retain: 0.74 Initial Forget: 0.8675 Final Forget: 0
lc-5 Initial Retain: 0.875 Final Retain: 0.7475 Initial Forget: 0.8675 Final Forget: 0
2e-5 Initial Retain: 0.875 Final Retain: 0.1825 Initial Forget: 0.8675 + Final Forget: 0
3e-5 Initial Retain: 0.875 > Final Retain: 0.8025 II Initial Forget: 0,8675 - Final Forget: 0.28
4e-5 Initial Retain: 0.875 -> Final Retain: 0.79 Initial Forget: 0.8675 - Final Forget: 0. 1775
MaxEnt (Language, CE Loss)
le-5
Initial Retain: 4.9359 - Final Retain: 4.9439
Initial Forget: 3,4672 - Final Forget: 11.3281
2e-5
Initial Retain: 4.9359 - Final Retain: 4.9397
Initial Forget: 3,4672 - Final Forget: 11,6977
3e-5
Initial Retain: 4.9359 + Final Retain: 4.942
Initial Forget: 3,4672 + Final Forget: 11.9844
4e-5
Initial Retain: 4.9359 + Final Retain: 4.9466
Initial Forget: 3,4672 - Final Forget: 12.1715
5e-5
Initial Retain: 4.9359 - Final Retain: 4.9551
Initial Forget: 3,4672 - Final Forget: 12.2584
6e-5
Initial Retain: 4.9359 + Final Retain: 4.9633 Initial Forget: 3,4672 - Final Forget: 12.3114
7e-5
Initial Retain: 4.9359 + Final Retain: 4.976 ll Initial Forget: 3.4672 - Final Forget: 12.3608
8e-5 Initial Retain: 4.9359 Final Retain: 4.9895
Initial Forget: 3.4672 Final Forget:
12.3808
MaxEnt (Arithmetic, Accuracy)
6e-5
Initial Retain: 0.875 Final Retain: 0.86 Initial Forget: 0.8675 Final Forget: 0.1275
7e-5
Initial Retain: 0.875 Final Retain: 0.83
Initial Forget: 0.8675 + Final Forget: 0.0525
8e-5
Initial Retain: 0.875 Final Retain: 0.8075 Initial Forget: 0.8675 - Final Forget: 0.0175
9e-5
Initial Retain: 0.875 - Final Retain: 0.795 Initial Forget: 0.8675 - Final Forget: 0.0125
le-4
Initial Retain: 0.87 - Final Retain: 0.81 |l Initial Forget: 0.87 - Final Forget: 0.035
2c-4
Initial Retain: 0.87 - Final Retain: 0.4825 Initial Forget: 0.87 + Final Forget: 0
3e-4
Initial Retain: 0.87 - Final Retain: 0.045 II Initial Forget: 0.87 - Final Forget: 0.015
20

---

4e-4 Initial Retain: 0.87 - Final Retain: 0.005 Initial Forget: 0.87 - Final Forget: 0.0225
RMU (Language, CE Loss)
le-5
Initial Retain: 4.9359 + Final Retain: 5.4614 Initial Forget: 3.4668 Final Forget: 7.2717
2e-5
Initial Retain: 4.9359 - Final Retain: 6.0976
Initial Forget: 3.4668 - Final Forget: 8.8457
3e-5
Initial Retain: 4.9359 - Final Retain: 6.8464 Initial Forget: 3,4668 - Final Forget: 9,8125
4c-5
Initial Retain: 4.9359 + Final Retain: 7.6585 Initial Forget: 3,4668 + Final Forget: 10.2192
5e-5
Initial Retain: 4.9359 + Final Retain: 8.4008 Initial Forget: 3,4668 - Final Forget: 10,6792
6e-5
Initial Retain: 4.9359 + Final Retain: 9.0205
Initial Forget: 3,4668 + Final Forget: 10.8073
7e-5
Initial Retain: 4.9359 Final Retain: 9,4909
Initial Forget: 3.4668 Final Forget: 10.958
8e-5
Initial Retain: 4.9359 + Final Retain: 9.8514
Initial Forget: 3.4668 - Final Forget: 11.2771
RMU (Arithmetic, Accuracy)
6c-6
Initial Retain: 0.875 + Final Retain: 0.7575 II Initial Forget: 0.8675 + Final Forget: 0.2275
7
Initial Retain: 0.875 Final Retain: 0.68 II Initial Forget: 0.8675 - Final Forget: 0.1375
8e-6
Initial Retain: 0.875 -> Final Retain: 0.615 II Initial Forget: 0.8675 - Final Forget: 0.0675
9c-6 Initial Retain: 0.875 Final Retain: 0.52 Initial Forget: 0.8675 + Final Forget: 0.035
le-5 Initial Retain: 0.875 Final Retain: 0.4425 Initial Forget: 0.8675 + Final Forget: 0.025
2e-5 Initial Retain: 0.875 + Final Retain: 0.14 Initial Forget: 0.8675 + Final Forget: 0
3e-5
Initial Retain: 0.875 + Final Retain: 0.065 Initial Forget: 0.8675 - Final Forget:
4e-5 Initial Retain: 0.875 - Final Retain: 0.05 Initial Forget: 0.8675 - Final Forget: 0
B.2 Distillation
Distillation In Figure we report relearning trends for the unlearned, distilled, and gold standard. We use the same dataset that was used pretrain respective models distillation data. We also ensure that the distilled model undergoes the same number of update steps as its pretrained counterpart. We perform distillation with the hyperparameters Table Additionally, we report the distillation trend To find information about Data Filtering (Gold Standard) models, refer to Appendix A
in Figure
Language Distillation
Arithmetic Distillation
Parameter
GradDiff
MaxEnt
RMU
GradDiff
MaxEnt
RMU
Teacher Model
Unlearn Only
Unleam Only
Unlearn Only
Unlearn Only Random Init
Unleam Only
Unleam Only Random Init
Student Model
Random Init
Random Init
Random Init
Random Init
Distillation Data
Kor Eng [0.5, 0.5]
Kor + Fngg
Kor Fing [0.5. 0.5]
All Ops + Eng
All Ops Eng
All Ops + Eng [0.75, 0.25)
Interleave Probs
0.5)
[0.75, 0.25)
[0.75,0.22 0.25)
Batch Size
4
15
15
Grad. Accum. Steps
60
60
60
24
24
24
Epochs
1
1
1
I
Learning Rate
9c-4
9c-4
9c-4
7c-4
7e-4
7e-4
Min Learning Rate
7c-4
7e-4
7c-4
7c-5
7c-5
7e-5
Max Steps
1000
1000
1000
Warmup Steps
50
50
50
50
50
50
Scheduler
Cosine
Cosine
Cosine
Cosinc
Cosine
Cosine
Weight Decay
0.1
0.0
0.0
Max Seq. Length
2048
2048
2048
256
256
256
Table 6: Distillation Hyperparameters
'This refers to the GraDDif/MaaEEn/RMM unlearned models, which are also marked as "Unlearn Only" in
Figure
21

---

(a) Distillation Language: GradDiff
(b) Distillation MaxEnt
(c) Distilation Language: RMU
5.0
5.0
5,0-
Cross-Entee TressEEtrropy// Loss
10.0-
10.3
10.0-
Cross Toos-Entooyy
Ents-Entrooy
15.0
Forget
15.0
45.0-
20.0
20.04
20.0-
200
400
600
800
200
450
600
800
200
400
600
809
Training Steps
Training Steps
Training Steps
[0) Distillation Arithmetic: GradDiff
(e) Distillation Arithmetic: MaxEnt
(1) Distillation Arithmetice RMU
1.9
0.8
Accuracy
Accuracy
Accuracy
0.4
0.4-
0.4-
0.0
0.0
Training
Training Steps
Training Steps
Figure 9: Distillation Trends
Unlearn (Grad MaxEnt/
Unlearnand Disill
Fiteringg Standard)
GradDiffLLaguage
(b) MaxEntLanguage
RMULLan guage
3.5
3.5
3.5
4.0
5.5
5.5
5.5
6.0
6.0
6.0
6.5
6.5
0
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
(4) GradDDittAthmmtticc
MaxEnnjArithmeticc
RMUJ)rrthmettcc
Accuresy (Fonge3)
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0
100
200
300
400
500
200
300
400
500
100
200
300
400
500
Training Steps on Forget Domain
Figure 10: Figure shown with average lines bolded.
C Experimental Details for Section 5
In Section 5
we introduce UNDO as an approach to leverage random noising as an alternative to full random initialization.
Dataset Data Processing/ Model Architecture Pretraining Relearning See Appendix
Unlearning Method We choose the best MaxEnt method from Appendix B based its favorable
trade-off between retain and forget performance. For language experiments, we use learning rate of 8e-5, which demonstrated the highest trade-off score while maintaining strong performance on both domains (Initial Retain: 4.9359 Final Retain: 4.9895, Initial Forget: 3.4672 - Final Forget: 12.3808). For arithmetic experiments, we use learning rate of 9e-5, which similarly achieved strong performance (Initial Retain: 0.875 Final Retain: 0.795, Initial Forget: 0.8675 - Final Forget: 0.0125).
22

---

Noising Method The novel element introduced in Section is controlled parameter corruption, allowing to interpolate between the original model and random noise. This provides more flexible approach compared to binary choices of keeping original parameters or fully reinitializing them. The algorithm applies weighted combination the original parameters and scaled random noise based on two hyperparameters: controls the proportion of corruption applied, while 3 controls the magnitude of the noise. The pseudocode as follows:
def do_ do_corruption (model, moise_alphn alaha, noise Laise_bet=0-. 1, seed=42) : # Loop through all parameters and add random noise assert 0 <= noise_alpha
assert 0 <= noise_beta
for param in model. parammerramt if param. requires Initialize corruption tensor
corruption torch. like (param. data)
Generate appropriate noise based on parameter dimensionality len (param shape) # For weight matrices (2D tensors), use Xavier init uniform_
noise torch. mn torch. eatihhempy..... like (param.dataa data)
elif len (param data. shape) == 1:
For bias vectors (1D tensors), use zeros noise forch.zeros like (param.data)
else:
raise RuntimeError(
f "Unsupportted parameter shape: (param data. shape]
Scale the noise by beta corruption noise_beta * noise
Apply weighted combination
param.data= d1ta= voise_alphas *param. datat noise_ olasstoopppttoootttiioo
# Move model appropriate device model. to (device)
For our experiments, we systematically vary from 0 to 0.8, while maintaining 3 at a constant value of 0.1 in F This allows us to evaluate the effect of gradually increasing levels of parameter corruption on both retention of desired capabilities and forgetting of undesired ones.
Distillation Method See Appendix B We follow the same distillation protocol established in previous experiments, using the noised models (obtained after applying the corruption function) as the student models in the distillation process.
D Experimental Details for Section
In Section we compare UNDO against several other unlearning methods evaluate their effec- tiveness in maintaining a balance between retain performance and resistance to relearning. This appendix provides the detailed experimental setup and hyperparameters used for these comparison experiments.
D.1 Language and Arithmetic Unlearning
Dataset Data Processing Model Architecture Pretraining We use the same datasets, processing methods, model architectures, and pretraining procedures as described in Appendix
23

---

Specifically, we conduct experiments on both the language (English/Korean) task using Gemma-2- O.IB and the arithmetic (addition-subtrr wutaaaaciitoooomimiiiccctooooooo viiiii task using Gemma-2-0.3B
Baseline Unlearning Methods
We compare several unlearning approaches:
GradDiff: Gradient Difference based unlearning with the forget domain.
MaxEnt: Maximum Entropy based unlearning that maximizes the model's uncertainty on the forget dataset while maintaining performance on the retain dataset.
RMU: Representation Misdirection for Unlearning, which modifies internal model representations for selected layers.
SAM: Sharpness-Awaid Aware Minimization that optimizes for flat loss regions to resist relearning.
RepNoise: Representation Noising that combines standard unlearning with noise injection at the representation level.
UNDIAL: Unlearning via distillation approach without global model damaging.
Learning Rate Sweeps For GradDiff, MaxEnt, RMU, and UNDIAL, we conducted extensive learning rate sweeps to find optimal configurations. We tested logarithmically spaced learning rates: 3e-3, 8c-3, 3e-4, 8c-4, 3e-5, 8e-5, 3e-6, 8c-6, 3e-7, 8c-7, 3e-8, and 8e-8. For each method, we chose the learning rate that provided the best trade-off between retain performance and forget degradation.
RepNoise Configurations For RepNoise experiments, used the best MaxEnt hyperparameters as the base configuration and then conducted a grid search over the following RepNoise-speciftc parameters:
Alpha (a)
Beta (5)
0.0001
0.1
0.001
0.5
0.01
0.5
0.1
0.1
1.0
1.0
2.0
2.0
4.0
4.0
Table RepNoise parameter configurations, where alpha controls the strength of the noise loss and beta controls the ascent loss.
SAM Configurations For SAM experiments, we also started with the best MaxEnt hyperparame-
ters and performed parameter sweep over the SAM perturbation radius (p) parameter:
Rho (p)
0.001
0.003
0.005
0.01
0.02
0.04
0.07
0.1
Table 8: SAM perturbation radius values explored. The value 0.01 is recommended by the original
SAM paper.
UNDO Configuration For our proposed UNDO method, we fixed the corruption parameters at = 0.6 and 3 0.1 based on our findings Section 5 We used the MaxEnt as the unlearning algorithm before applying random noising and distillation. We also experimented with various retain
24

---

threshold values {0.05, 0.09, 0.13, 0.17, 0.21, 0.25, 0.29, 0.33) to obtain different points along the retain-forget trade-off curve.
Common Training Parameters All language experiments used the following common parameters unless otherwise specified for particular method:
Batch size: 4
Max sequence length: 2048
Scheduler: Cosine
Weight decay: 0.0
Gradient clipping threshold: 1.0
For arithmetic experiments, used:
Batch size: 40 (GradDiff: 20, RepNoise/SAM variants: with gradient accumulation steps of 8)
Max sequence length: 256
Scheduler: Cosine
Weight decay: 0.0
Gradient clipping threshold: 1.0
Method-Specifie Parameters
Each unlearning method required specific hyperparameters:
GradDiff: Alpha (retain-forget tradeoff) language, Alpha (retain-forget tradeoff) = for arithmetic
RMU: Layers [5-11], End layer 11, Alpha 1200, 6.5 for language; Alpha 200, for arithmetic
Relearning Evaluation To evaluate robustness against relearning, we fine-tuned each unlearned model on the forget dataset (Korean or multiplicaiion/divsiionn for varying numbers of steps (0, 40, and 500) to simulate increasingly strong adversarial attacks. We monitored both the retain performance (on English or addition/subtraction and forget performance to assess how well each method maintained the desired knowledge while suppressing the forgotten capabilities.
D.2
WMDP
We apply method the WMDP 42 benchmark both the Cyher and Bio domain and compare to existing unlearning methods. We use Gemma-2-2b 73 as the base model.
D.2.1 Evaluation/Renchmark
For the retain evaluation we use MMLU, evaluating the model with 5-shot on subset 40% of the full MMLU evaluation and report accuracy. For the forget evaluation use WMDP Bio/Cyber depending on the domain use the entire evaluation, using zero-shot prompting.
D.2.2 Datasets
WMDP includes WMDP forget and retain corpora for each domain. However, it common practice to other datasets well. For example the original WMDP paper introduces RMU 42 using Wikitext [5T] as the retain dataset.
Applying unlearning methods using the original forget and retain datasets well as public auxiliary datasets such as Wikitext proves difficult, especially for the biology domain and methods other than RMU. We hypothesize this is due the datasets possessing numerous differences that could be used to differentiate them. This allows the unlearning methods to achieve low loss on their objectives but fail to generalize to the evaluation. To remedy this, we create a dataset that eliminates most undesired differences and emphasizes the relevant differences in the domains. We extract question answer datasct from each corpora, Bio retain, Bio forget, Cyber retain, and Cyber forget, by prompting
25

---

Gemini flash to write questions and answer from text sampled from the corpora. Additionally, we extract questions and answer from English Wikipedia using the same process. The following an example of question and answer generated for the Cyber forget dataset.
Question: How does Address Space Layout Randomization (ASLR) complicate shellcode injection?
Answer: Address Space Layout Randomization (ASLR) randomizes the memory addresses of key program areas, making it harder for attackers to predict where shellcode should be placed in memory for successful exploitation.
We experimented with using different datasets and combinations of datasets to optimize the tradeoff between forgetting and maintaining retain performance. Ultimately, we use the forget iwestion-answer dataset for the forget set and a 50/50 mix of the retain question-answer dataset and Wikitext for the retain set.
D.2.3 Baseline Unlearning Methods
We two baseline unlearning methods, RMU and MaxEnt.
Both RMU and MaxEnt involve optimizing a loss that is the sum of a retain component and forget component. For WMDP, we parameterize this combination with such that loss relain_comppnent (1 ax) * forgel__come pompppenon where < a < 1.
For MaxEnt, we use a variation in which the loss maximizes entropy on the forget set while minimizing KL divergence between the base model and unlearned model the retain set. For RMU, we use the same loss as language/arithmetic settings. We use the parameters in Table
Parameter
Value
General Training Parameters
Batch size Sequence length
40
Steps
90
Learning rate
Fixed (no scheduler)
Weight decay
Warmup steps
RMU-Specifie Parameters
value
80 (10, 11, 12, 13, 14, 15)
RMU layers
RMU end layer
15
Table 9: Hyperparameter Settings
We hyperparameter sweeps alpha and learning rate for both methods, then chose three points across the frontier to use in the experiment. We run four seeds of each method and apply adversaries each measure elicited forget performance. Of the three points use the one with the highest retain performance as the teacher for the UNDO models. Ultimately, the models in Table TO are chosen.
D.2.4 Distillation/NNDOO
We apply UNDO using an auxiliary dataset. The dataset consists of the following datasets, specified by the Huggingface name, and mixed at the percentages shown in Table II
D.2.5 Relearning Adversaries
We extensively explore relearning adversaries via small batches of relearning runs. To ensure fairness and avoid only selecting adversaries that do well on baseline methods, we always run configurations on the UNDO method. We run the most promising configurations across all seeds of the methods, and run all configurations UNDO. We report the metries from the best adversary for each method.
26

---

Domain
Method
Configuration
Retain Performance Ranking
0.5, 5 10-8
Best
RMU
0.3, = 1 x 10
Medium
0.1, 1 10
Worst
Bio
0.3, 2 10
Best
MaxEnt
a 0.1, 2 x 10-6
Medium
0.75, Ir x 10-5
Worst
0.5, 2 10-8
Best
RMU
0.2, Ir 2 10
Medium
Cyher
0.1, 2 10
Worst
0.2, 2 10
Best
MaxEnt
a 0.3, 3.5 10
Medium
0.3, Ir 5 10
Worst
Table 10: Selected Models Ordered by Performance Tradc-off
Dataset
Proportion
HuggiinffaccFF///ieewwec
(subset_namnl (cubeet_caamo""saapll--00017
0.35
legacy-danaseeWWiiiieeed
en")
0.35
0.05
0.04
0.04
0.04
MappiceAkig//MagiiePPiShhh------
0.04
Magpie-Align/Magnn Qwen2.5-Pro- 1Mv00.
0.04
AlippiiAAlgg/MMagiic Qwreii-ewweee---e -ronh---rr----e Pro-IM-vO..
0.04
Wikipedia question-answer
0.01
Total
1.00
Table 11: Auxiliary Dataset Composition
We explore seven configurations and vary the learning rate for each.
We have variety of techniques and datasets that vary form 6 adversary configurations.
Dataset Descriptions:
1. forget/retain: Consists a 50/50 mixed forget and retain WMDP corpora from the corresponding domain.
2. forget/retain-qa: Consists of a 50/50 mixed forget and retain question corresponding domain.
-answer answer dataset from the
3. wiki-qa:
Consists of 50/25/25 mix of wikitext, forget question-answer and retain question- answer from the corresponding domain.
Sequence Length Descriptions:
Standard: Sequence length 256
Long: Sequence length 1024
Technique:
Shrink-perturb: When applied, we apply shrink perturb to the weights as formulated in the UNDO method with 0.05 before relearning.
We test diverse adversaries, as each existing method may be selectively robust against certain attacks. For example, RepNoise was much more susceptible to the shrink-perturb relearning, while being
27

---

Adversary Name
Dataset
Sequence Length
Shrink-perturb
forget/recain forget/rctain--e
standard standard
forget/retain
long
no
forget/retain-qa-long
2
long
n10
wiki-qa-long
3
long
sp-forget/retain
standard
yes
sp forget/retain-qa
standard
yes
sp sp-bio- wiki-qa-long
long
yes
Table 12: Confi gonfiguration of the seven adversary techniques used to evaluate unlearning robustness. Dataset values correspond to: (1) original forget/retain data, (2) question-ana answer formatted data, and (3) Wikipedia data with QA formatting. Sequence length indicates whether standard extended context was used. Shrink-perture indicates whether the model parameters were perturbed before relearning.
more robust to other adversaries. Similarly, for the TAR method, the authors report robustness across most adversaries, but select adversaries were completely able to recover performance [7]]. We were unable to find hyperparameters that maintained reasonable retain performance within our training budget.
D.2.6 Baseline Robust Methods
We explore two robust baselines, SAM and RepNoise, in the Arithmetic and Language settings. For SAM, set 0.01, and for RepNoise use 3=1.0 and a=1.0. Both of these methods operate in addition to MaxEnt, so we use the top two selected MaxEnt hyperparameters.
D.2.7
WMDP Pareto Frontier and Discussion
UNDO
MaxEnt
RMU
Adversarial Finetuning
Low-Compute Adversary
High-Compute Adversary
Retain-Forget: WMDP-Bio Relearning)
Retain-Forget: WMDP Bio Steps)
Retain Petgen-Forget: WMDP (after Steps)
0.50
0.55
0.55
Octinal
Ogümal
Opémal
Aruracy (hararo
0.50
0.50
0.50
0.40
0.40
0.40
0.30
0.30
0.30
0.25
0.25
0.25
(Ponget)
Accuracy (Porgel after Belearing)
Accuracy (Porget after Reteeminn)
(6) Retain-Forget: WNDP-Cyber (No Releaming))
1e3 Retain-Forget WHDP-Cyber (after 10 Steps)
10 Retain-Forget: WyDD-Cyoer Cafter 500 Steps)
Octina
Ogdimal
Optimal
Nutrney (Mataind
0.5
0.5
0.5
0.0
0.4
0.4
0.3
0.3
0.3
8.37
0.31 Accuracy (Fonget)
0.37
0.25
0.37
0.34
0.28
Accuraccy (Forget after Relcaming)
Accuracy (Forget after Rakaming)
Figure 11: We measure and plot forget and retain accuracy on WMDP Bio (col and Cyber (col 2), immediately after unlearning (row 1), after low compute adversarial retraining (row two), and after a high compute adversarial retraining (row 3).
We apply MaxEnt, RMU, UNDO, RepNoise, and SAM to Gemma-2-2b. We observe UNDO is on the Pareto frontier for WMDP-Bio, and competitive with other methods for WMDP-Cyber, despite decreases in retain performance. The results are shown in Figure [T] We observe that the UNDO method has similar robust-forget vs retain tradeoff other baseline methods.
This contrasts with the earlier results for the language/arithmetic settings seen in Figure 7 with UNDO far exceeding the Parcto frontier set by the baseline methods, We hypothesize that this is due
28

---

to under-training the model due to limited compute and not having access to the original pretraining corpus.
More thoroughly, the key difference in the arithmetii/languaaee pareto Figure 7 we use a = 0.6, 0.25, implying noise in the language/arithmttic settings than WMDP. Higher alpha damage to the model's capabilities, making them harder to retrain. We show this in Figure S where higher alpha values more robust and achieve lower retrained forget performance. However, when we apply higher alpha values to Gemma-2-2b, we find that doesn't 't recover retain performance (see Table 13).
whereas in WMDP pareto Figure[[1]
In Figure 3 we observe that higher alpha values require more data to recover performance, For = 0.6, the language and arithmetic settings require around 35% and 20% of their pretraining corpus, respectively. For WMDP, our reference model, Gemma-2-2b was originally trained on trillion tokens 773. but distill only 300 million, 0.015% of the Gemma-2-21 pretraining corpus. However, for Al companies aiming to apply our method to their models, we expect that they have sufficient compute resources train larger fraction of the pretraining corpus. Additionally, it's possible that the original pretraining corpus has better coverage or diversity to effectively learn the capabilities compared our distillation data.
We expect that the challenges we face would not apply to the companies that could apply our methods to their models because they would have sufficient compute resources, access to the full pretraining corpora, and could even have proprietary distillation methods that allow faster distillation or better generalization compared setup.
Alpha
Retrained MMLU Value
Original (0)
0.55
0.25
0.49
0.37 0.35
0.6
0,25
Table 13: Alpha noise values applied Gemma-2-2b and corresponding retrained MMLU values. =0 0 for all experiments.
E
Compute requirements
We run all experiments servers with multiple H200 or A100 GPUs. Language and arithmetic pretraining took 4xH200/IxA100 GPUs for two to three days. All distillation and UNDO runs for language are run 4xH200 GPUs, while relearning took one H200 GPU per setup. All distillation and UNDO runs for arithmetic are run on a single A100 GPU, while relearning took one A100 GPU per setup. Our most computationally expensive experiments WMDP UNDO runs, which take around hours on one H200. Additionally, computing evaluations while relearning takes minutes to do sparse evaluation relearning run (which collects evaluations steps 10, 25, 50, 150, and 500).
29

---

F Post-Review Reflection: An Alternative Setup to Study Unlearn-and-Distill
We present supplementary experiments conducted during the review period that extend our findings to additional benchmarks and attack vectors. The main concerns raised centered on the generalizability of our findings beyond simplified settings and the computational requirements of training models from scratch. We address these specifically in Appendices F.1 and
We first discuss an alternative experimental setup that enables studying the unlearn-and-distill method without reinitializing models from scratch. In this section, consider a pretrained model and dataset D that we can guarantee M has never encountered during pretraining. Here, we assume the not encountering the dataset can guarantee that the pretrained model M doesn't 't learn the capabilities associated with it.
Given the setup above, we can then finetune M on to create M', apply unlearning methods on M' to suppress D, then distill M" back onto the original model M. Since M never possessed latent knowledge of D, serves as an effective substitute for a randomly initialized network with respect to this specific knowledge.
Our observation throughout the paper was that once network has learned certain capabilities during pretraining, reversing this learning requires a significant corruption of the network parameters (also see concurrent literature [85] that discusses the level of corruption needed for robust unlearning be more likely). Indeed, the reference (unlearned) models in Figure 2and Figure Sesure/demmone demonstrated rapid relearning, and we broadly refer the neural network properties that enable this rapid recovery as "latent traces." " However, if we have data D that was definitively absent from the pretraining corpus, then M contains no such latent traces for D. This allows us to emulate the from Section 4 without access to the original pretraining data of M.
and-disstll setup
There challenges obtaining an ideal D. Language models are essentially large token classifiers that develop circuits to use all tokens in their vocabulary, including those that would appear in D. Even if the semantic content of D is novel, the model has seen and predicted its constituent tokens. Therefore, it's likely that keyword or semantic matching cannot guarantee the absence of latent traces.
Fortunately, this challenge is not new, and we have an established approximation. The TOFU benchmark [50] contains 200 fictional author profiles with question-answer pairs, constructed to be absent from existing corpora. Here, the specific associations between authors and their biographical details are novel. Therefore, we conduct supplementary experiments in Appendices F.T and [.22 with the assumption that TOFU provides sufficient separation to study the distillation phenomenon in a controlled setting where M effectively starts at random initialization with respect to the forget knowledge.
(m) Camma-2-28: Finetune (TOFU)
(b) Gamme-2-28: Unleam (GradDiff)
[c) Gamma-2-226 28: Distill (Finetuned
3.-
Farget Mentixt
Hetroor
Hager nus
22-
0.
Make calyy
MA
kmiin Ireed
6.6
0.6
0.4
100
275
300
420
530
660
20
40
82
200
120
330
200
300
450
300
100
(TOFU)
(GradDiff)
Basa)
3..
1.0
10-
08-
0.8
H4402 185.8
0.6
2.6
0.6
0.4
3.4
0.4
02-
02
222
230
Chocopoin:
Figure 12: Training dynamics across finetuning, unlearning, and distillation phases on TOFU. Panels (a-c) show Gemma-2-2B, and panels (d-f) show Phi-1.5.
30

---

F1 TOFU Experiments
Following the framework presented above, we conducted experiments using the three-step protocol.
First, we finetuned Gemma-2-28 Phi-1.5 on the full TOFU dataset, achieving high accuracy on the fictional author questions.
Second, we applied GradDiff (gradient ascent on the forget set; gradient descent on the retain set), suppressing the model's ability to answer questions about these authors while maintaining performance on the retain set.
Third, we distilled the unlearned model back onto fresh Gemma-2-2B / Phi-1.5 base model. We use the full TOFU dataset that used to finetune the base model, including both forget and retain domains.
Figure [2 the training dynamics across all three phases. We conduct the experiment using the open-unlearning framework [20] and track seven commonly used unlearning metrics. During finetun- ing (a, d), both models learn fictional author information, with forget metrics increasing steadily while MIA (membership inference attack) [68 scores rise to near-perfect levels. The unlearning phase (b, e) shows suppression of forget knowledge, with Forget QA probability dropping to near zero within 50 steps while retain performance stabilizes. The distillation phase (c, f) demonstrates selective knowledge transfer. The student models maintain forget metrics similar to the base model that hasn't 't seen TOFU, despite being trained outputs from the full dataset, while retain metrics gradually recover to match the teacher's performance.
(a) Relearning Trend
Phi-1.5: Relearning Trend
0.45
Metricc Value
0.40
0.40
0.35
0.35
0.30
0.30
0.25
0.25
0.20
0.20
20
60
60
100
20
60
80
100
Releaming Checkpoint
Releaminn Checkpoint
Forget QA Prob
-
Forget ROUGE
Extraction Str
MW MinK
Forget
Forget (Unlearn)
Forget (Unleam-and-Disil)
Figure 13: Relearning dynamics on TOFU forget set for (a) Gemma-2-2B and (b) Phi-1.5. Models that underwent unlearning followed by distillation show relearning rates closer to control baselines that never saw the forget authors, while unlearned-only models rapidly recover the suppressed knowledge. Individual metrics are shown in light colors with average trends in bold.
To evaluate robustness, we subjected three models to relearning attacks: the unlcarned-oni, model, the salearned-and-distilee model, and control baseline (the original base model that never saw TOFU), Figure I3 shows the relearning dynamics over 100 gradient updates. The unlearned-only models rapidly recover the suppressed knowledge, approaching the performance levels seen after initial finetuning. In contrast, the unlearned sonedd-nnd---siild models show substantially slower relearning, with trajectories closer to the control baselines that never possessed the forget knowledge.
These results demonstrate that the distillation robustification phenomenon holds even when using existing models fresh networks, provided the forget knowledge was absent from their original training. The consistency across both Gemma-2-22 and Phi-1.5 architectures suggests this approach generalizes across different model families.
31

---

F.2 Robustness Beyond Relearning Attacks
Prior work established that gradient-basee finetuning represents the strongest elicitation technique for recovering unlearned capabilities 49]. Nevertheless, we tested robustness against alternative attack vectors during the review period. Zhang et al. [92] demonstrated that quantization can bring back unlearned knowledge by exploiting numerical instabilities in model weights. We evaluated INT4 quantization attacks using the open-unlearning framework on our TOFU-trained models.
Method
Precision
Forget Q&A Prob
Forget Truth Ratio
Model Utility
GradDiff
FPI6 INT4
0.000
0.000
0,495
GradDiff
0.008
0.185
0.242
GradDiff + Distilled
FP16 INT4
0.000
0.001 0.017
0.442
GradDiff + Distilled
0.003
0.391
Table 14: Quantization attack results TOFU (Gemma-2-2B). Lower forget metrics indicate better unlearning.
Under INT4 quantization, standard unlearning shows substantial degradation of unlearning perfor- mance, with forget truth ratio jumping from 0.000 to O. 0.185. The distilled models demonstrate better robustness, with forget truth ratio reaching only 0.017. The distilled models also maintain better utility under quantization (0.391 0.242).
These quantization results suggest that distillation creates fundamentally different internal represen- tations rather than merely suppressing outputs. The student network, having never possessed the forget knowledge in its parameter space, lacks the latent structures that quantization exploits in the unlearned-only models. This aligns with our broader hypothesis that distillation acts capability filter, allowing only the knowledge actively demonstrated by the teacher to transfer to the student.
32

---

G Author Contribution
All authors contributed the early development of the ideas of the paper.
Bruce W. Lee was a core contributor to our experimental codebase and was responsible for framing and conducting most experiments reported in the paper, except WMDP. This includes Unleamn-nnd- Distill and UNDO experiments reported in Figures 1 plotted all figures in the paper. BL wrote the majority of the later sections and later revised the earlier sections.
5 6
BL implemented baselines and
Addie Foote conducted all WMDP experiments in Figures unlcarning experiments. AF was a core contributor to our experimental codebase, making unlearning and distillation scripts, efficiency optimizations, and experiment infrastructure. AF wrote key sections on the WMDP experimental setup and the limitations discussion. AF consistently addressed the most challenging and uncertain problems, operating in the face of uncertainty, particularly with the WMDP benchmark.
and initially designed our arithmetic
Alex Infanger contributed significantly to the writing of the paper, the development of the problem statement, the interpretation and the discussion of experimental results, and the implementation of pilot example of the oracle matching result. Al embarked semantic odyssey to improve our understanding of the robust unlearning problem.
Leni Shor conducted exploratory research to mechanistically validate our oracle matching experi- ments, helping the team gain confidence in our ideas. LS also explored formalisms that improved our understanding of the problem we were trying to solve. LS conducted initial MNIST experiments with AI, streamlining the MNIST codebase.
Harish Kamath achieved the first working implementation of our experiments on WMDP, which informed our later experimental approach. HK implemented and conducted experiments with noising schedulers as part of the initial UNDO framework development, and explored activation-basel distillation instead of logit-based distillation.
Jacob Goldman-Wette robustifies unlearning and reported the first positive results. JGW assisted with WMDP experiments and optimized RMU on Gemma models. JGW also contributed to activation- based distillation experiments and provided debugging support for the codebase when needed.
developed the initial proof-of-concept demonstrating that distillation
Bryce Woodworth served as research manager, organizing and facilitating team research meetings and coordinating project logistics. BW provided essential support in maintaining team communication and collaboration throughout the research process. BW contributed to copy-editing and offered valuable feedback on presentation and clarity.
Alex Cloud suggested combining distillation with unlearning and investigating oracle matching, and also co-led the research team. AC worked closely with the team, offering helpful input on implementation details, experimental design, and conceptual framing. He also contributed to writing.
Alexander Matt Turner co-led the research team and was instrumental in the initial conceptualization of the project. AT contributed to earlier iterations of the distillation approach, originally explored in the context of Gradient Routing based on his suggestions. Throughout the project, AT provided valuable feedback on experimental direction, helped analyze results, and contributed to manuscript development.
33

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly articulate our contributions and accurately represent the main results and scope of the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: discuss the limitations in Section
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes],
Justification: See Sections 3-6 and appendices.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release research code through GitHub.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, Sections 3-6 and appendices.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See relevant discussion sections.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: See relevent appendix.
9. Code of ethics
34

---

Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS
Code of Ethics https:///earrse.crppppuou/ccaaat
Answer: [Yes]
Justification: We confirm.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See intro and conclusion.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release data or models that have a high risk misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: Low risk of misuse, not releasing new models.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes.
13. New assets
Question: Are assets introduced in the paper well documented, and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release our research code through GitHub.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is an important, original, non- standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration not required.
Answer: [NA]
Justification:
35


Reviews:
[{"review_id": "L2b87SxfNj", "reviewer": "Reviewer_4Vbg", "summary": "The paper presents an investigation into the limitations of current machine unlearning techniques, particularly targeting performance with respect to relearning attacks, where a model quickly recovers forgotten capabilities through finetuning. The authors argue that conventional unlearning often suppresses undesired behaviors rather than eliminating them, leaving latent representations intact due to factors like plasticity and representational entanglement. To address this, the authors propose Unlearn-and-Distill, a method where a model is first unlearned and then used to train a freshly initialized student model through knowledge distillation, effectively discarding residual capabilities. Building on this idea, they introduce UNDO (Unlearn-Noise-Distill-on-Outputs), a more flexible and computationally efficient extension.\n\nThe authors empirically validate their methods across multiple domains, including language modeling, arithmetic reasoning, and a high-stakes Weapons of Mass Destruction Proxy (WMDP) benchmark. The results show that UNDO shows robustness against relearning attacks compared to existing unlearning methods such as MaxEnt, SAM, and RepNoise. Experiments demonstrate that increasing the noise perturbation parameter α consistently strengthens resistance to relearning, with even modest values yielding improvements. Moreover, the paper shows that UNDO achieves favorable retain-forget trade-offs and remains resilient under strong adversarial finetuning (e.g., 500-step attacks, also highlighted in Figure 7).", "questions": "* While the paper emphasizes that the mixing coefficient $\\alpha$ largely controls robustness and computation cost, the role of the noise scale $\\beta$ seems less clearly justified. It would be good if the authors could provide more detailed insights for this parameter in practive. For instance, are there conditions or dataset properties under which certain beta values outperform others? \n\n* The paper evaluates robustness under several adversarial relearning scenarios, but the extent and variety of these attacks could be expanded (and would never be exhaustive). This raises some questions regarding the guarantee of unlearning, which remains a challenge in general. It would be good if the authors could comment on similar lines, explaining the extent and limitations of the current setting.\n\n* The experiments focus on retain and forget sets to measure unlearning effectiveness, but it is unclear how the method affects generalization to unrelated downstream tasks or out-of-distribution data. It would be good if the authors provide evaluations or discussions on whether the UNDO method preserves overall model utility and fairness across a wider task spectrum. Additionally, evaluation using richer and more comprehensive metrics such as Verbatim Probability, ROUGE variants, Model Utility, Forget Quality, TruthRatio, and Membership Inference Attack (MIA) strength, as used in recent benchmarks (e.g., Open-Unlearning), could further help validate the claims made in the paper.", "limitations": "Yes, the authors clearly mention some of the major limitations of their approach, particularly around the distillation and the challenges of applying UNDO in settings where access to the original pretraining data is limited (e.g., Gemma-2-2B). These limitations make the proposed approach difficult to apply to all the open-weight LLMs where pretraining data is available.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper systematically demonstrates that standard unlearning methods leave residual capabilities that can be quickly relearned (similar findings are also revealed in other recent works [1, 2, 3, 4]). Intriguingly, the paper introduces a controlled methodology (referred to as UNDO) that allows interpolation between noising and full reinitialization, with some empirical improvements in resisting relearning across multiple domains (language, arithmetic, WMDP), that may be helpful for future works on similar lines. \n\n* While distillation has been used for compression or privacy, this work is somewhat novel in explicitly using it as a means to destroy/degrade/suppress latent capacities that contribute to fast relearning (making the idea itself unique and less explored in the unlearning community [though some initial works like [5] do explore a teacher-student approach]). This reframing is both conceptually and technically significant, as it connects behavioral unlearning with representation-level damage via noising, followed by reconstruction under behavioral constraints, making it a good unlearning mechanism for the community to explore in future works. Making it align with the unlearning community by using similar benchmarks would definitely benefit the overall quality/utility of this work.\n\n**Weaknesses**\n\n* One of the notable weaknesses of this work is that it evaluates robustness primarily through forget/retain set performance and relearning speed. However, the effects of perturbation and distillation on general downstream performance (e.g., OOD generalization, calibration, fairness) remain unexplored. Since the proposed UNDO method introduces large-scale noising and re-alignment, understanding its unintended side effects on broader capabilities becomes imperative. It would be good to add other evaluation metrics that are used in unlearning [also refer to questions for clarification details regarding other evaluation metrics]\n\n* Another major weakness is that the paper uses a completely different setting than the existing works on machine unlearning, making it difficult to compare/validate/check how the proposed unlearning fits the existing set of works. It would be good to add some results on benchmarks like open-unlearning, using datasets like TOFU for a better alignment with the unlearning research community. \n\n* On the method side, the current method distills only output distributions (via forward KL), potentially ignoring subtle representational changes. It remains unclear whether internal activations, gradients, or alignment signals are also preserved or transformed in ways that might make the model vulnerable to alternative elicitation strategies. Some works like [1, 2, 3, 4] show challenges in measuring unlearning and how the model’s knowledge remains intact even after unlearning. Since it is difficult to compare the proposed work using different datasets, it remains challenging to rely on the results for the applicability of the proposed learning method in other settings. \n\n[1] https://arxiv.org/abs/2410.16454\n\n[2] https://arxiv.org/abs/2406.13356\n\n[3] https://arxiv.org/abs/2411.15477\n\n[4] https://arxiv.org/abs/2402.16835\n\n[5] https://arxiv.org/abs/2310.02238", "was_revised": true, "final_justification": "The added set of experimental results further validates the claims and makes it more aligned with the unlearning literature. After considering the authors’ rebuttal and additional experiments, I am updating the rating by a point to reflect the improvements made. Below is a summary of resolved/remaining concerns:\n\nThe resolved concerns include 1) the addition of comprehensive evaluations on TOFU using open-unlearning benchmarks, directly addressing the concern about alignment with the broader unlearning community. 2) The inclusion of evaluation metrics that are popular in the unlearning community, Verbatim Probability, ROUGE variants, Model Utility, Forget Quality, TruthRatio, and MIA strength. 3) Authors also tested quantization-based elicitation and demonstrated improved robustness via post-distillation, further showing the generality of their approach. 3) The role of β is now better explained, with practical guidance that was added after the rebuttal. \n\nSome of the remaining concerns include 1) the approach still relies on access to unfiltered data during distillation, which may not always be feasible (e.g., open-weight models without pretraining data). 2) While TOFU results are encouraging, broader adoption across more diverse or realistic forgetting scenarios (e.g., non-synthetic or semantically entangled data) would further strengthen the work.\n\nAfter the rebuttal, the paper is above the acceptance threshold, based on the substantial improvements made and the added set of experiments.", "rebuttal": "We thank the reviewer for the thoughtful feedback and constructive suggestions! First, we share additional evaluations on TOFU to address the concerns about downstream performance and alignment with the broader unlearning literature.\n\n---\n\n### TOFU\n\nWe conducted experiments using the **open-unlearning** library. TOFU turns out to be ideal since its fictional figures don't exist in any pretraining data, making the base model mimic random initialization (never encountered what we want to unlearn). We have done additional experiments with Gemma 2 2B and Phi 1.5. We report Gemma 2 2B in this response due to character limits.\n\n*Step 1: TOFU Finetuning on gemma-2-2b*\n\n| Epoch | Forget_Q_A_Prob | Forget_ROUGE | Forget_Truth | Retain_Q_A_Prob | Retain_ROUGE | Model_Utility | Extraction_Str | MIA_MinK |\n|-------|-----------|--------------|--------------|-----------|--------------|---------|---------|------|\n| 1     | 0.29      | 0.42         | 0.72         | 0.29      | 0.40         | 0.35    | 0.13    | 0.53 |\n| 3     | 0.51      | 0.49         | 0.56         | 0.49      | 0.46         | 0.43    | 0.21    | 0.92 |\n| 5     | 0.59      | 0.53         | 0.48         | 0.57      | 0.49         | 0.45    | 0.24    | 0.97 |\n\nThe model memorizes the fictional authors, with Forget_QA probability increasing from 0.06 (epoch 0) to 0.59, while MIA (membership inference attack) success rate goes from 0.31(epoch 0)  to 0.97.\n\nWe then apply GradDiff unlearning, which successfully suppresses the unwanted knowledge while maintaining retain performance. \n\n*Step 2: Unlearning TOFU using GradDiff*\n\n| Step | Forget_Q_A_Prob | Forget_ROUGE | Forget_Truth | Retain_Q_A_Prob | Retain_ROUGE | Model_Utility | Extraction_Str | MIA_MinK |\n|------|-----------|--------------|--------------|-----------|--------------|---------|---------|-------|\n| 0    | 0.59      | 0.53         | 0.48         | 0.57      | 0.49         | 0.45    | 0.24    | 0.97  |\n| 25   | 0.04      | 0.36         | 0.49         | 0.31      | 0.41         | 0.46    | 0.04    | 0.37  |\n| 50   | 0.001    | 0.003        | 0.002        | 0.41      | 0.36         | 0.44    | 0.03    | 0.01  |\n\nAfter 50 steps, the forget metrics drop to near-zero (Forget_QA: 0.59→0.001) while model utility remains stable. We then distill this behaviorally-suppressed model back into the original gemma-2-2b:\n\n*Step 3: Distilling Unlearned Model*\n\n| Epoch | Forget_Q_A_Prob | Forget_ROUGE | Forget_Truth | Retain_Q_A_Prob | Retain_ROUGE | Model_Utility | Extraction_Str | MIA_MinK |\n|-------|-----------|--------------|--------------|-----------|--------------|---------|---------|-------|\n| 1     | 0.001     | 0.01         | 0.006        | 0.21      | 0.36         | 0.38    | 0.03    | 0.006 |\n| 3     | 0.001    | 0.002        | 0.001        | 0.26      | 0.37         | 0.44    | 0.03    | 0.01  |\n| 5     | 0.001    | 0.002        | 0.001        | 0.26      | 0.37         | 0.44    | 0.03    | 0.01  |\n\nThe resulting rewritten model maintains near-zero forget metrics while recovering full utility (0.44), matching the unlearned teacher's performance. This demonstrates that distillation transfers desired behaviors on retain but not on forget.\n\nThen, we subject the three models (unlearned, unlearned+distilled, fresh base model - as control) to relearning attacks (100 steps on TOFU) to measure how quickly the forget knowledge is recovered:\n\n*Metric: Forget_Q_A_Prob*\n\n| Model Type | Step 10 | Step 20 | Step 50 | Step 100 |\n|------------|---------|---------|---------|----------|\n| (a) Unlearned | 0.313 | 0.331 | 0.362 | 0.378 |\n| (b) Unlearned+Distilled | 0.242 | 0.261 | 0.283 | 0.293 |\n| (c) Control (Never seen TOFU) | 0.226 | 0.242 | 0.261 | 0.269 |\n\n*Metric: Extraction Attack Success Rate*  \n\n| Model Type | Step 10 | Step 20 | Step 50 | Step 100 |\n|------------|---------|---------|---------|----------|\n| (a) Unlearned | 0.137 | 0.148 | 0.154 | 0.155 |\n| (b) Unlearned+Distilled | 0.113 | 0.117 | 0.123 | 0.122 |\n| (c) Control (Never seen TOFU) | 0.114 | 0.119 | 0.122 | 0.124 |\n\nWhile the unlearned-only model rapidly recovers the suppressed knowledge (Forget_QA: 0.378 after 100 steps), the distilled model relearns at rates closer to the control, a fresh model that has never seen TOFU. Extraction attack rates are virtually identical between our method and the control (0.122 vs. 0.124), suggesting that the post-unlearning distillation step does promote a deeper knowledge removal. This pattern holds across different unlearning methods (GradDiff, NPO, etc) that are used before distillation, confirming the generality of our approach. \n\n---\n\n### W1: Effects on general performance\n\nTOFU results above directly demonstrate capabilities preservation during specific knowledge removal. The distilled model maintains the same utility as the teacher while keeping forget metrics near zero. Our controlled experiments show the student selectively learns general capabilities that the unlearned teacher properly demonstrates, despite distilling on the full, unfiltered dataset:\n\n*Table: Distillation Step - Language Unlearning*\n\n| Step | Student's Eng CE Loss (General) | Student's Kor CE Loss (Forget) |\n|------|----------------|----------------|\n| 0    | 12.52          | 12.52         |\n| 50   | 7.17           | 12.48         |\n| 200  | 6.02           | 12.58         |\n| 800  | 4.98           | 12.64         |\n\nSimilarly, in arithmetic experiments, the student recovers the teacher's addition/subtraction/English performance while multiplication/division stays completely unlearned.\n\n*Table: Distillation Step - Arithmetic Unlearning*\n\n| Step | Student's Eng CE Loss (Control) | Student's Subtraction Acc (Retain) | Student's Addition Acc (Retain) | Student's Multiplication Acc (Forget) | Student's Division Acc (Forget) |\n|------|----------------|-----------------------|-------------------|----------------|----------|\n| 1    | 12.60          | 0%               | 0%            | 0%             | 0%       |\n| 200  | 5.59           | 7%              | 3%            | 0%             | 0%       |\n| 400  | 4.76           | 18%             | 3%            | 0%             | 0%      |\n| 1000 | 4.47          | 92%        | 80%     | 0%         | 0%   |\n\nThese results are also available as Figure 9. In all experiments reported (Figures 1-6), the distilled student models achieve performance within 5% of their teachers on English \\& retain task. For certain tasks like arithmetic, we observe that the student actually often exceeds the teacher's performance on retain and control tasks. This is reminiscent of a well-documented phenomenon in distillation literature where students can surpass their teachers [1]. Distillation not only preserves capabilities but can also sometimes enhance them through cleaner learned representations.\n\n\n---\n\n### W2: Validation on mainstream benchmarks\n\nThe TOFU experiments above address this concern. We tested multiple unlearning methods from open-unlearning, and adding distillation consistently improved resistance to relearning. \n\nWe thank the reviewer for this suggestion. Our method validated well on mainstream benchmarks like TOFU. Given these compelling results, we will:\n\n1. Contribute our method to **open-unlearning** for ease of access.\n\n2. Add a separate TOFU evaluation section.\n\n---\n\n### W3: Other elicitation efforts\n\nThank you for raising this point about alternative elicitation strategies! \n\nWe first note that prior literature shows that finetuning is a strong elicitation technique. [2, 3] found that while finetuning consistently performed best across all tested setups, other attacks showed limited success. This universal vulnerability to finetuning, combined with its standardized implementation and replicability, motivated our focus on finetuning.\n\nIn our paper, we go a step further and take the strongest adversarial assumption of hyperparameter search across 10 learning rates (1e-6 to 1e-3), substantial data (8M tokens for Korean, 2M tokens for arithmetic), 500 gradient updates, and report adversarially optimal cases.\n\nAdditionally, we empirically verify that our approach also robustifies against alternative elicitation strategies. We tested quantization using the open-unlearning library [4]:\n\n*Table: Quantization Effects*\n\n| Method | Metric | Original | Quantized |\n|--------|--------|------|------|\n| Unlearned (GradDiff) | forget_Q_A_Prob | 0 | 0.008 |\n| | forget_truth_ratio | 0 | 0.185 |\n| | model_utility | 0.495 | 0.242 |\n| | mia_min_k | 0.0004 | 0.054 |\n| Unlearned+Distilled | forget_Q_A_Prob | 0 | 0.003 |\n| | forget_truth_ratio | 0.0006 | 0.017 |\n| | model_utility | 0.442 | 0.391 |\n| | mia_min_k | 0.011 | 0.013 |\n\nUnder INT4 quantization, standard unlearning shows increased forget metrics (forget_Q_A_Prob: 0 → 0.008, forget_truth_ratio: 0 → 0.185). Our distilled models demonstrate superior robustness with forget_Q_A_Prob reaching only 0.003 (2.7× better) and forget_truth_ratio at 0.017 (11× better). The distilled models also maintain better utility (0.391 vs 0.242) and lower MIA vulnerability (0.013 vs 0.054).\n\n---\n\n### Q: Role of β \n\nWe have edited our manuscript to clarify the role of β and include an explanation here. The role of β is to ensure sufficient perturbation at low α values (scaling to be meaningfully different from Xavier init) and also that the resulting model has the correct scale. We recommend: (1) Use β=0.1 as default (2) Only tune β for α<0.2 (3) For α≥0.3, the mixing coefficient provides sufficient disruption. This aligns with [5] who found β=0.01-0.1 optimal across various settings.\n\n---\n\n— We thank the reviewer for the constructive suggestions! —\n\n[1] Understanding the gains from repeated self-distillation. NeurIPS 24\n\n[2] The elicitation game: Evaluating capability elicitation techniques. ICML 25\n\n[3] Eight methods to evaluate robust unlearning in llms. Preprint\n\n[4] Catastrophic Failure of LLM Unlearning via Quantization. ICLR 25\n\n[5] On warm-starting neural network training. NeurIPS 20"}, {"review_id": "4Ve1eetyNW", "reviewer": "Reviewer_4DBt", "summary": "This paper proposes a way to improve robustness of machine unlearning methods (which aim to remove certain harmful knowledge/capabilities). The problem: current unlearning methods produce models that can still elicit the harmful behavior on finetuning. The paper presents evidence of this problem in Sec 3.\n\nThe paper then proposes a way to improve the robustness of any existing unlearning method (i.e. plug-and-play), at the cost of additional compute. The key insight is to use distillation on unlearned models. This method is called Unlearn-and-Distill, presented in Sec 4.\n\nThe paper introduces a way to control the trade-off between additional compute and unlearning robustness. This method is called UNDO (Unlearn-Noise-Distill-on-Outputs), presented in Sec 5. It also compares with other unlearning methods in Sec 6.", "questions": "**Q1**. In the formulation in Section 2, the setup requires a retain set. Can unlearning work without a retain set, and are there algorithms in the literature? I ask this because practically, one might want to retain all capabilities except a few harmful ones. Thinking ahead, in this case, can distillation provide an advantage on existing unlearning methods? \n\n**Q2**. W.r.t lines 169-173, what is the dataset used for distillation? Is it the full training dataset?\n\n**Q3**. W.r.t lines 204-206, I'm unclear about whether this claim has been shown empirically, because your approach (line 100 and around) requires the entire training set to be partitioned into the retain and forget sets.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n**S1**. The paper makes an interesting and novel observation that distillation robustifies unlearning. This approach, called Unlearn-and-Distill, gives advantage over simple unlearn baselines (Figure 4).\n\n**S2**. The paper introduces an approach which allows for trading between compute and robustness (UNDO, Figure 5).\n\n**S3**. The paper demonstrates that UNDO extends the pareto-optimal frontier on the tradeoff between quality retention (for harmless cases) and unlearning (for harmful cases). Figures 7 and 8 demonstrate competitive performance of UNDO over baselines.\n\n**S4**. The paper is well written: It has a good structure that follow nicely, and explanations are clear to understand.\n\n**Weaknesses**\n\nThe paper does not present much intuition/explanation of why distillation is effective at improving the robustness of unlearning methods. At a high-level, distillation seems to perform unlearning + \"mixing\" the unlearned weights with noise. Are there ways to investigate why this works?\n\nRelatedly, are there other ways to \"mix\" the unlearned weights with noise instead of distillation? Perhaps they can be cheaper in compute?", "was_revised": true, "final_justification": "My final rating is 5 (increased from the initial rating of 4). \n\nThe initial rating of 4 was because:\n- The paper makes an intuitive yet interesting observation,\n- and it shows the idea works through detailed experiments.\n\nMy rating has increased because:\n- I find the discussion by the authors on the weaknesses raised (W1 and W2) satisfactory.", "rebuttal": "We thank the reviewer for the thoughtful review and for highlighting the novelty of our approach! \n\n---\n\n### Conceptual explanation for our work\n\nWe appreciate your request for more intuition about why distillation works. While we give this intuition in Section 3, we've added some discussion here. \n\nTo briefly recap our key finding from Section 3: We started with models that already possessed harmful capabilities (we refer to these as \"reference models,\" which were pretrained on both English and Korean). We then trained these contaminated models to perfectly mimic an oracle model that had never seen Korean. Despite achieving near-perfect behavioral alignment, something strange happened during relearning attacks.\n\nWhen we compared three types of models:\n\n1. **Oracle**: Never saw Korean during training\n\n2. **Student (Random)**: Fresh model trained to mimic the oracle's outputs  \n\n3. **Student (Reference)**: Contaminated model trained to mimic the oracle's outputs\n\nThe contaminated models that \"forgot\" Korean relearned it dramatically faster:\n\n**Table: Oracle Matching Results (Figure 2)**\n| Task | Model Type | Starting Knowledge | Relearned (100 steps) | Relearned (500 steps) \n|------|------------|------------------|------------------------------|------------------------------\n| Language | Oracle Teacher | English only | Korean CE: 5.8 | Korean CE: 4.2 \n| Language | Student (Random) | None (fresh start) | Korean CE: 5.7 | Korean CE: 4.2 \n| Language | Student (Reference) | English + Korean | Korean CE: **3.9** | Korean CE: **3.7** \n| Arithmetic | Oracle Teacher | Add/Sub only | Accuracy: 17% | Accuracy: 45% \n| Arithmetic | Student (Random) | None (fresh start) | Accuracy: 18% | Accuracy: 45% \n| Arithmetic | Student (Reference) | All operations | Accuracy: **45%** |Accuracy: **85%** \n\nStudent (Reference) showed fast relearning despite behaving perfectly the same as the oracle. \n\nThe model's \"origin\" (what it knew before matching the oracle) determined its vulnerability to relearning. \n\nNow, what if we could combine unlearning (which achieves behavioral suppression) with distillation to a fresh model (which has no problematic history)? When distilling from an unlearned teacher to a fresh student:\n\n1. The student starts clean (no pre-existing harmful knowledge)\n\n2. The teacher's outputs on harmful content are meaningless noise\n\n3. The student can only learn from what the teacher demonstrates\n\n4. Since the teacher refuses to teach harmful content, it cannot transfer\n\nThat is, we leverage distillation as a capability filter and create a new solution space:\n\n| Method | Robustness | Data Labeling Required |\n|--------|------------|----------------------|\n| Data Filtering | High | Must label entire corpus |\n| Standard Unlearning | Low | Label <0.01% of data |\n| **Unlearning + Distillation** | **High** | **Label <0.01% of data** |\n\nWe also validated this on TOFU [1], where fictional authors were created to test unlearning in realistic scenarios. Our distilled models relearned at similar speeds as the control baseline (a model that never saw the forget authors), while unlearned-only models rapidly recovered the suppressed knowledge under relearning attacks. \n\nGiven that Section 3 already presents this conceptual foundation with evidence (Figure 2), would more forward references from Section 4 to Section 3 help clarify the connection in the manuscript?\n\n---\n\n### W1: Why does distillation robustify unlearning?\n\n*From review text: “The paper does not present much intuition/explanation of why distillation is effective at improving the robustness of unlearning methods.”*\n\nThe student starts knowing nothing and can only learn what the teacher demonstrates. Since the unlearned teacher produces meaningless outputs on forget data, there's nothing for the student to learn about those capabilities.\n\nConsider a teacher that has undergone unlearning to suppress multiplication. Although latent multiplication knowledge remains in its parameters (which might later promote rapid relearning of the capability), when asked to solve multiplication problems, it produces only random noise. A fresh student can only learn from the gradients provided by the teacher's outputs and cannot acquire multiplication, because the teacher provides no meaningful signal.  Though intuitive once stated, the effectiveness of this approach is not obvious, given potential anomalies in the long tail of teacher logits. Our contribution is demonstrating that it works well in practice.\n\nAs an example, the table below shows how the Korean CE Loss stays large despite the fact that the distillation training set includes Korean documents: \n\n*Table: Distillation - Language Unlearning Experiment*\n\n| Step | Student’s Eng CE Loss (Retain) | Student’s Kor CE Loss (Forget)\n|------|----------------|----------------\n| 0    | 12.52          | 12.52         \n| 50   | 7.17           | 12.48          \n| 200  | 6.02           | 12.58      \n| 800  | 4.98           | 12.64       \n\nSimilarly, in arithmetic experiments, the student recovers the teacher's addition/subtraction/English performance while multiplication/division stays completely unlearned.\n\n*Table: Distillation - Arithmetic Unlearning Experiment*\n\n| Step | Student’s Eng CE Loss (Control) | Student’s Subtraction Acc (Retain) | Student’s Addition Acc (Retain) | Student’s Multiplication Acc (Forget) | Student’s Division Acc (Forget) \n|------|----------------|-----------------------|-------------------|----------------|----------\n| 1    | 12.60          | 0%               | 0%            | 0%             | 0%       \n| 200  | 5.59           | 7%              | 3%            | 0%             | 0%       \n| 400  | 4.76           | 18%             | 3%            | 0%             | 0%      \n| 1000 | 4.47          |92%        |80%     | 0%         | 0%   \n\n\n\n---\n\n### W2: Alternative approaches to mixing\n\n*From review text: “Relatedly, are there other ways to \"mix\" the unlearned weights with noise instead of distillation?”*\n\nWe believe that this is a promising topic for future research. Instead of random perturbations, we could use interpretability to identify and corrupt specific weights responsible for harmful behaviors. We think success would depend on two criteria:\n\nCriterion 1: Size of unlearning target - How semantically broad the capability is\n\nCriterion 2: Entanglement - How distributed across the model's weights\n\nFor narrow facts like \"the capital of France is Paris\" (small C1), mechanistic interpretability suggests these are often localized (small C2) [2], making targeted noising promising. However, for complex capabilities like deception (large C1), the required computations are likely distributed across many components (large C2), such as tracking truth, generating alternatives, modeling beliefs, and optimizing outputs [3].\n\nWhile we haven't implemented targeted noising yet, even imperfect targeting could improve retain-forget tradeoffs. It doesn't need to be perfect, just better than global. This is an exciting avenue for future work!\n\n---\n\n### Q1: Unlearning without explicit retain sets\n\n*From review text: “Can unlearning work without a retain set, and are there algorithms in the literature?”*\n\nYes, this depends on the unlearning algorithm. GradDiff computes `loss_forget - loss_retain`, requiring both sets. In practice, many past works use generic retain datasets (WMDP uses Wikitext). During distillation, we don’t require an explicit retain set. The teacher's outputs on any unlabeled data automatically provide the training signal, where meaningful predictions guide learning on retain content while uninformative outputs prevent learning on forget content.\n\n---\n\n### Q2: Distillation dataset\n\n*From review text: “what is the dataset used for distillation? Is it the full training dataset?”*\n\nYes, we use the full pretraining dataset for distillation. Only the initial unlearning step needs small labeled retain/forget sets (typically <0.01% of data). We avoid labeling billions of tokens.\n\n---\n\n### Q3: Full training set partitioning\n\n*From review text: “I'm unclear about whether this claim has been shown empirically, because your approach (line 100 and around) requires the entire training set to be partitioned into the retain and forget sets.”*\n\nNo, we don't require partitioning the entire training set. Lines 204-206 compare against the theoretical \"gold standard\" of filtering everything (impractical at scale). We only need small labeled sets for initial unlearning, then use unlabeled pretraining data for distillation.\n\n---\n\n— We thank the reviewer for pushing us to clarify the conceptual points! —\n\n[1] TOFU: A Task of Fictitious Unlearning for LLMs. COLM 24\n\n[2] Locating and Editing Factual Associations in GPT. NeurIPS 22\n\n[3] The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets. COLM 24"}, {"review_id": "irmQD9Enc3", "reviewer": "Reviewer_TRF9", "summary": "This paper addresses the critical shortcoming of existing machine unlearning methods for large language models, which often only achieve \"behavioral suppression\" rather than true removal of undesired capabilities. The authors demonstrate that these suppressed capabilities can be easily recovered through adversarial finetuning. The core contribution is the finding that distilling an unlearned \"teacher\" model into a new, randomly initialized \"student\" model can robustify the unlearning process, effectively transferring desired behaviors while discarding the latent, undesired capabilities. Building on this insight, the paper introduces Unlearn-Noise-Distill-on-Outputs (UNDO), a practical method that creates a tunable trade-off between unlearning robustness and computational cost by partially noising the teacher model's weights before distillation. Through extensive experiments on language modeling, arithmetic reasoning, and the Weapons of Mass Destruction Proxy (WMDP) benchmark, the authors show that their approach significantly improves resilience to relearning attacks compared to existing methods, establishing a new Pareto frontier for the trade-off between model performance and unlearning robustness.", "questions": "Do the authors have preliminary results for a wider variety of model families and larger model sizes? It would be interesting to see if robustness degrades with scaling.\n\nWMDP is still a proxy. Have you tried latent knowledge probes rather than behavioral scores?\n\nThe performance drop on the MMLU retain task for the WMDP experiments is a key practical hurdle. Do you have other ideas on how to mitigate this issue without needing the original pretraining data?\n\nThe paper argues that UNDO is more efficient than the \"gold standard\" of data filtering with full retraining. Could you provide a more concrete, quantitative comparison of the computational cost (e.g., total FLOPs) between UNDO and full retraining for one of the smaller-scale experiments, such as the language task? This would help readers better contextualize the claimed cost savings.\n\nThe proposed UNDO method introduces noise into the weight space of the teacher model before distillation. Have you considered or experimented with alternative methods of \"damaging\" the teacher to prevent the transfer of latent information?\n\nThe effect of perturbation levels (alpha) seem to vary based on task (e.g. arithmetic vs language). Do you have an intuition for why this is the case? Does this suggest that the optimal alpha is highly task-dependent, and can you offer any guidance for how a practitioner might select a suitable value for a new task without performing an expensive hyperparameter sweep?\n\nSome references are broken (showing as ??) - please fix these.", "limitations": "Yes. The authors have adequately addressed the limitations of their work in a dedicated paragraph and in the appendix. They are upfront about the significant computational cost of distillation and the challenge of maintaining retain performance when the original pretraining data is unavailable. The discussion is fair and provides important context for the results.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper tackles a highly significant and timely problem in LLM safety. The distinction between merely suppressing behavior and genuinely removing a capability is a central challenge for deploying safe models. This work offers a practical and seemingly effective method to bridge this gap, which is of great interest to the community.\n\nThe experimental evaluation is exceptionally thorough and well-designed. The authors construct a compelling narrative, starting by demonstrating the fundamental limitations of output-matching for robust unlearning. They then systematically validate their core hypothesis - that distillation robustifies unlearning - across multiple unlearning techniques and tasks. The proposed UNDO method is analyzed in detail, clearly illustrating the compute-robustness trade-off. The comparison against strong, relevant baselines under increasingly powerful relearning attacks is convincing and highlights the superiority of the proposed approach. Finally, validating the method on a realistic benchmark like WMDP strengthens the claims. The appendices are remarkably detailed, ensuring high potential for reproducibility.\n\nThe paper is very well-written, clearly structured, and easy to follow. The central idea is simple yet powerful, and it is communicated effectively through concise prose and informative figures. The logical flow from problem identification to solution proposal and validation makes the paper a pleasure to read.\n\nWhile its constituent parts (unlearning, distillation) are not new, the core insight - that distillation can act as a selective filter to remove latent capabilities from an unlearned teacher - is novel and impactful. The generalization of this idea into the UNDO framework, which offers a practical knob to tune the trade-off between cost and robustness, is an original and useful contribution. The initial experiment showing that oracle-matching is insufficient for robustness is a valuable finding that helps clarify the nature of the problem for the field.\n\nThe paper contains clear ablation studies and multiple experiments supporting the main hypotheses and also answering potential questions from the readers. For instance, the plots showing effect of noise levels on robustness was very useful.\n\nThe paper uses well known and recent benchmarks to measure effects of the method (e.g. MMLU for retain performance and WMDP for forget performance).\n\nCode will be released on a public GitHub repository, which is clearly stated in the main body of the paper. The paper also lists detailed hyperparameter settings. Both of these enhance reproducibility of this work and its usefulness to the community.\n\nIf the basic observation holds at larger scales, it offers a practical path for open-weight release: distillation is already part of most deployment pipelines, so adding “unlearn then distill” could become standard practice. That is a meaningful contribution to LLM safety.\n\nWeaknesses:\n\nThe primary weakness, which the authors rightly acknowledge, is the high computational cost of distillation. While the paper frames this as a manageable trade-off and notes that distillation is already part of many development pipelines, the cost may still be a significant barrier to adoption, especially for organizations without access to massive compute resources or for applications where frequent unlearning is required.\n\nA key practical limitation is the observed drop in performance on the retain task when applying UNDO to a public model without access to its original pretraining data. This currently limits the \"off-the-shelf\" applicability of the method and suggests that its effectiveness is still somewhat dependent on having access to high-quality, representative data for the distillation phase.\n\nThe paper provides strong empirical evidence that distillation robustifies unlearning but offers a less detailed explanation for why this is the case. The intuition that desired behavior is transferred while latent capabilities are \"left behind\" is compelling but high-level. A deeper analysis - perhaps investigating the information bottleneck properties of distillation in this context or examining changes in the model's internal representations - could further strengthen the paper's contribution, though this may be extensive enough for future work.\n\nThe selection of models and model sizes is very limited.", "was_revised": true, "final_justification": "I have carefully read the author rebuttal and found it helpful in clarifying certain aspects of the submission. While the response addressed some of my concerns, it did not significantly alter my overall evaluation of the paper. As a result, I have decided to retain my original score.", "rebuttal": "We thank the reviewer for the thoughtful and thorough review! We first address the key concerns raised by the reviewer and then proceed to discuss other questions.\n\n---\n\n### W1: Access to pretraining data, off-the-shelf applicability, breadth of models tested\n\nFor the threat models we consider (for example, bad actors finetuning open-weight models), mitigation becomes significantly more difficult once the weights of a model with latent dangerous capabilities are made available to the public. Performing robust unlearning on a model whose weights are already publicly available is not helpful. Therefore we expect robust unlearning methods to be used by model developers after pretraining a model but before releasing it to the public. These developers would have access to the original pretraining dataset. \n\nThe TOFU benchmark gives us another setup to test our method at scale. TOFU's fictional figures don't exist in any pretraining data, making the reference model mimic random initialization (zero latent knowledge of what we want to unlearn). We confirm our method works at larger scales and also test it across different architectures (Gemma-2-2B and Phi-1.5) and unlearning algorithms (GradDiff and NPO):\n\n**Table: TOFU Results on Gemma-2-2B with GradDiff**\n\n| Model | Initial Performance | Performance After 100-step Relearning |\n|-------|-------------------|----------------------------------|\n| **Unlearned Only** (GradDiff) | Forget Q&A: 0.000, MIA: 0.000 | Forget Q&A: 0.378, MIA: 0.822 |\n| **Unlearned+Distilled** (GradDiff+Distill) | Forget Q&A: 0.000, MIA: 0.011 | Forget Q&A: 0.293, MIA: 0.551 |\n| **Control** (Never saw TOFU) | Forget Q&A: 0.057, MIA: 0.310 | Forget Q&A: 0.269, MIA: 0.455 |\n\n\n**Table: TOFU Results on Phi-1.5 with NPO**\n\n| Model | Initial Performance | Performance After 100-step Relearning |\n|-------|-------------------|----------------------------------|\n| **Unlearned Only** (NPO) | Forget Q&A: 0.063, MIA: 0.186 | Forget Q&A: 0.375, MIA: 0.840 |\n| **Unlearned+Distilled** (NPO+Distill) | Forget Q&A: 0.063, MIA: 0.164 | Forget Q&A: 0.282, MIA: 0.692 |\n| **Control** (Never saw TOFU) | Forget Q&A: 0.131, MIA: 0.325 | Forget Q&A: 0.243, MIA: 0.465 |\n\n*Forget Q&A: Probability of correctly answering forget set author questions (lower is better)*\n  \n*MIA: Membership Inference Attack success rate (lower is better)*\n\nBoth our distilled models achieve relearning rates closer to the control baseline (a model that never saw the forget authors), while unlearned-only models rapidly recover the suppressed knowledge (trend is clearer in a graph). In the camera-ready version, we would report an analysis across model families and sizes.\n\nImportantly, we don't necessarily require starting from a random init model if we know for sure from which split of the pretraining data the forget target is from. Consider a case where there are several model checkpoints saved during training based on time intervals (m1, m2, m3, m4...) corresponding to dataset splits (d1, d2, d3, d4...), as is common for many production setups. If we know that the model has become problematic at a certain time point (e.g., harmful content introduced in d4), we can roll back to checkpoint m3 (which doesn’t have latent traces of d4 knowledge) and distill the unlearned m4 onto it. . \n\n---\n\n### W2: Explanation for why distillation robustifies unlearning\n\nThank you for this comment *“The paper provides strong empirical evidence that distillation robustifies unlearning … A deeper analysis … could further strengthen the paper's contribution”*. \n\nWe're particularly excited that our work aligns with emerging insights in the unlearning literature. Recent work [1] found that irreversible unlearning is difficult to achieve without \"large rotations of principal directions, centroid shifts, and vanishing Fisher mass across many layers,\" essentially requiring many layers to undergo coordinated, large-magnitude perturbations. We believe our work provides a timely contribution by demonstrating a concrete mechanism that achieves what these works hypothesize as true forgetting.\n\nOur intuition is that all knowledge must pass through the teacher's outputs to reach the student. When combined with a student that initially has no latent information of the forget set, this distillation operation creates the kind of coordinated, large-magnitude perturbations across many layers that [1] identifies as necessary for irreversible unlearning. By starting from random initialization (or a clean checkpoint), the student undergoes global representational shifts while selectively learning only what the teacher demonstrates.\n\nOur experiments validate this hypothesis. Even when distilling on the full dataset including forget data:\n\n**Table: Selective Learning During Distillation**\n\n| Domain | Metric | Initial → Final |\n|--------|--------|-----------------|\n| Language (Retain: English) | CE Loss | 12.52 → 4.98 |\n| Language (Forget: Korean) | CE Loss | 12.52 → 12.64 |\n| Arithmetic (Retain: Add/Sub) | Accuracy | 0% → 86% |\n| Arithmetic (Forget: Mul/Div) | Accuracy | 0% → 0% |\n\nThe student's performance on forget domains remains at random initialization levels despite seeing the forget domain data in every batch. \n\nAlso, our approach addresses a broader category of capability removal beyond other privacy-focused unlearning use cases. While privacy applications typically target specific data points or narrow facts, our work demonstrates removal of entire capability domains (languages, mathematical operations).\n\n---\n\n### Q1: Preliminary results for wider variety of model families and larger sizes?\n\nWe will prepare a wider variety of model families and larger sizes results for TOFU. \n\nOur main language and arithmetic unlearning experiments were done at pretraining scale, making it harder to scale up. We have initial results for up to 0.9B establishing similar findings, but for the purpose of focused scientific contribution, we chose to present singular model sizes (0.1B for language and 0.3B for arithmetic). Instead, we take an extremely strong adversarial assumption to ensure the validity and replicability of our claims, where we do hyperparameter search across 10 learning rates (1e-6 to 1e-3), substantial data (8M tokens for Korean, 2M tokens for arithmetic), and 500 gradient updates, and report adversarially optimal cases.\n\n---\n\n### Q2: Have you tried latent knowledge probes rather than behavioral scores?\n\nWe have tested prompt-based and activation steering elicitation methods on our bilingual models. However, it was challenging to get clear signals, potentially due to the relatively small model sizes we used (100M-300M parameters) or limitations in applying these techniques to our specific setups. However, we tested alternative elicitation methods. For quantization-based attacks [2]:\n\n**Table: INT4 Quantization Robustness (TOFU, Gemma-2-2B)**\n\n| Method | Forget Q&A Prob | Forget Truth Ratio | Model Utility |\n|--------|----------------|-------------------|---------------|\n| GradDiff (FP16) | 0.000 | 0.000 | 0.495 |\n| GradDiff (INT4) | 0.008 | 0.185 | 0.242 |\n| GradDiff+Distilled (INT4) | 0.003 | 0.017 | 0.391 |\n\nOur distilled models show much better resilience across all metrics, suggesting that distillation provides defense against attack vectors beyond just gradient-based relearning.\n\n---\n\n### Q3: Performance drop on MMLU - how to mitigate without original pretraining data?\n\nSee our comment above regarding threat models. Also, we think parts of our distillation setup could be optimized (e.g., spreading out the noising over distillation may help).  \n\n---\n\n### Q4: Concrete quantitative comparison of cost?\n\nWhile Figure 5 shows that arithmetic reaches gold standard robustness with about 60% of gold standard compute and language with about 80%, we concretely mean data efficiency. The key advantage is avoiding the need to label the entire pretraining corpus. In Figures 5 and 7, UNDO required only 0.01% of the pretraining data to be labeled to match the gold standard robustness of full data filtering. That is, we create a new solution space:\n\n| Method | Robustness | Data Labeling |\n|-|-|-|\n| Data Filtering | High | High |\n| Standard Unlearning | Low | Low |\n| **Unlearn + Distill** | **High** | **Low** |\n\n---\n\n### Q5: Alternative methods of \"damaging\" the teacher?\n\nWe see several promising alternatives like pruning, targeted damaging in ROME-style [3]. While we haven't implemented these yet, even imperfect targeting could improve retain-forget tradeoffs compared to our uniform approach.\n\n---\n\n### Q6: Task-dependent alpha - why does it vary? Practitioner guidance?\n\nThe variation could stem from different evaluation metrics (cross-entropy loss for language vs. accuracy for arithmetic) having different sensitivities to perturbation. After an inflection point, we observe a linear trend between compute requirements and robustness in both domains once the noising starts to remove the latent traces. \n\n---\n\n### Q7: Broken references\n\nThank you for catching this! We'll fix all references showing as \"??\" in the camera-ready.\n\n---\n\n— We thank the reviewer for a very thorough review! —\n\n[1] Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs. Preprint 25\n\n[2] Catastrophic Failure of LLM Unlearning via Quantization. ICLR 25\n\n[3] “Locating and Editing Factual Associations in GPT” NeurIPS 22"}, {"review_id": "HohahDCcgA", "reviewer": "Reviewer_nuYc", "summary": "The paper tackles the problem that most machine-unlearning techniques for large LMs only suppress dangerous capabilities that are to be unlearned, leaving them latent and easy to re-learn with light fine-tuning. The methods presented in the paper are motivated by a simple observation: if you first apply any standard unlearning method to suppress the unwanted behavior and then distill that model into a randomly initialized student, the harmful capability is largely stripped away while the desired skills are retained. They demonstrate this across two unlearning tasks, arithmetic and the WMDP weapons benchmarks, showing that their Unlearn-Noise-Distill-on-Outputs (UNDO) procedure (which adds controlled parameter noising to trade compute for robustness -- i.e., taking a weighted average between a randomly initialized and the pretrained model) closes much of the gap to an expensive \"retrain-from-scratch with perfect data filtering\" oracle (training only on the \"clean\" subset of the data). UNDO consistently sits on the pareto frontier of the utility-unlearning curve, with distilled models relearning harmful tasks no faster than the oracle while preserving most of the original utility.", "questions": "In Fig7a, how come many methods do better than the filtering gold standard?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper is clearly written. The objective is well defined, the problem which is the focus of this paper is interesting, and the presented method is motivated by experiments that show very clear differences in unlearning rate between randomly initialized and pretrained models. \n\nMost of the experiments are carried out in relatively simplified settings, where the forget set is either an entire language or the distinction between two mathematical operators (addition and multiplication). In such a case, the distinction between the examples to be unlearned and the retain set is very significant, as they differ in domain. This contrasts with usual unlearning work that focuses on very specific concepts. Fig 8 reports results on WMDP, but does not present the utility-unlearning curve. What would it look like in this case?\n\nI am not sure the forget-utiltiy comparison in Fig7 is completely fair. If I understand correctly, the baselines just experience unlearning, while UNDO experiences unlearning+retraining on the retain set. It can be the case that this retraining is responsible for better retain-set accuracy. Can it be the case that it’s mainly this retraining, and not the noise, that explains the better tradeoff? Did you try to ablate the noising step? \n\nRobustness is measured via gradient-based relearning attacks (finetuning on the forget set). It would be beneficial to also compare against jailbreaking attacks in inference time.", "was_revised": true, "final_justification": "I am satisfied with the authors response. They answer my concerns, particularly these related to the limited evaluation in the original submission. I hope the new results will be incorporated into the final version of this work.", "rebuttal": "We are glad that the reviewer found our writing clear and findings interesting! \n\n---\n\n### W1: Experiments in simplified settings\n\n*From review text: “Most of the experiments are carried out in relatively simplified settings, where the forget set is either an entire language or the distinction between two mathematical operators (addition and multiplication).”*\n\nWe understand this concern about domain separation. To directly address it, we tested our method on TOFU [1], a benchmark specifically designed to evaluate unlearning of specific concepts like in typical unlearning work. TOFU contains 200 synthetic author profiles with 20 Q&A pairs each. These fictional authors test unlearning where forget and retain sets are closely related.\n\n**Table: TOFU Relearning Attacks (Gemma-2-2B)**\n\n| Model | Initial Performance | Performance After 100-step Finetuning \n|-------|-------------------|----------------------------------\n| **Unlearned Only** (GradDiff) | Forget Q&A: 0.000, MIA: 0.000 | Forget Q&A: 0.378, MIA: 0.822\n| **Unlearned+Distilled** (Ours) | Forget Q&A: 0.000, MIA: 0.011 | Forget Q&A: 0.293, MIA: 0.551\n| **Control** (Never saw forget authors) | Forget Q&A: 0.057, MIA: 0.310 | Forget Q&A: 0.269, MIA: 0.455 \n\n*Forget Q&A: Probability of correctly answering forget set author questions (lower is better)*\n  \n*MIA: Membership Inference Attack success rate (lower is better)*\n\nOur distilled models achieve similar relearning rates to the control baseline (a model that never saw the forget authors), while unlearned-only models rapidly recover the suppressed knowledge. This demonstrates our method also works when unlearning specific concepts. That said, as we explain in our introduction, our primary motivation differs from typical unlearning work. We focus on removing broad capabilities, rather than specific facts or concepts. Our method works for both use cases.\n\n---\n\n### W2: WMDP utility results\n\n*From review text: \"Fig 8 reports results on WMDP, but does not present the utility-unlearning curve. What would it look like in this case?\"*\n\nYou're right that Figure 8 didn’t include complete utility-unlearning curves. We provide these results in Figure 11 in the appendix:\n\n**Table: WMDP-Bio**\n\n| Method | MMLU | WMDP Forget |  |  |\n|--------|-------------|------------------------|------|------|\n| | | Initial | 40 steps | 500 steps |\n| MaxEnt | 0.485 | 0.294 | 0.485 | 0.522 |\n| MaxEnt+UNDO | 0.460 | 0.299 | 0.440 | 0.498 |\n\nWe didn't highlight them in the main paper because interpreting WMDP results is challenging. Gemma-2-2B's pretraining likely already contains weapons and cybersecurity information, making it impossible to establish a true \"never seen\" baseline like we can with TOFU's fictional authors. For clarity, we also evaluated on TOFU, where we have guaranteed control over exposure:\n\n**Table: TOFU Utility-Unlearning Tradeoff**\n\n| Method | Model Utility | Forget Q&A (Initial) | Forget Q&A (50 steps) \n|--------|--------------|---------------------|----------------------\n| Unlearned Only | 0.444 | 0.000 | 0.362\n| Unlearned+Distilled | 0.442 | 0.000 | 0.283\n| Control (Gold Standard) | - | 0.057 | 0.261\n\n*Model Utility is the harmonic mean of 9 metrics measuring performance on retain set, real authors, and world facts (3 metrics each: Q&A probability, ROUGE score, and truth ratio)*\n\nOur method achieves 23% better robustness than unlearning alone while maintaining competitive utility (0.442 vs 0.444 for unlearning only). The distilled model's relearning trajectory closely matches the control, suggesting that the post-unlearning distillation step does promote a deeper knowledge removal while retaining the model's utility. This pattern holds regardless of which unlearning method (GradDiff, NPO, etc.) is used before distillation, confirming the generality of our approach.\n\n---\n\n### W3: Fair comparison concerns\n\n*From review text: \"I am not sure the forget-utility comparison in Fig7 is completely fair. If I understand correctly, the baselines just experience unlearning, while UNDO experiences unlearning+retraining on the retain set.\"*\n\nYou raise an important point, but we believe there may be a slight misunderstanding about how UNDO works. With UNDO, you first select any unlearning method (GradDiff, MaxEnt, etc.) with your desired retain-forget balance (Figure 3). Then, separately, you decide how much compute to invest in distillation to improve robustness and gain back the retain performance. The retain-forget trade-off is bounded by the teacher (unlearning method of choice), while robustness scales with compute (trade compute for robustness, Figure 5).\n\nCrucially, UNDO does not retrain on a labeled retain set. During distillation, we distill on the full pretraining dataset, including both retain and forget data. While the distillation does require more compute than prior unlearning methods, this step enables increased robustness. This is analogous to heavily damaging and then rewriting parts of the model internal while having a teacher who can only properly demonstrate certain capabilities. What the teacher pretends not to know, the student won't learn at all.\n\nFor example, the table below shows our distillation trends where the datasets contained 50% forget data; the student only recovers capabilities the teacher demonstrates:\n\n| Language Task | Initial → Final | Arithmetic Task | Initial → Final |\n|---------------|-----------------|-----------------|-----------------|\n| English CE (Retain) | 12.52 → 4.98 | Addition Acc | 0% → 80% |\n| Korean CE (Forget) | 12.52 → 12.64 | Multiplication Acc | 0% → 0% |\n\nThe student's Korean loss remains at random initialization levels (12.64) despite seeing Korean text in every batch. Similarly, multiplication accuracy stays at 0% even though 25% of the arithmetic distillation data contains multiplication problems. The initially unlearned teacher provides meaningful gradients only for retain capabilities. This property creates a unique space between existing approaches:\n\n| Method | Low Compute | High Robustness | Minimal Data Labeling |\n|--------|-------------|-----------------|----------------------|\n| Data Filtering | X | O | X (All pretraining corpus) |\n| LLM Unlearning | O | X | O (Smaller labeled set) |\n| **UNDO (Ours)** | **Tunable** | **Tunable** | **O (Same as LLM unlearning)** |\n\nRegarding Figure 7 that you specifically mentioned, all UNDO points use the same noise parameters (α=0.6, β=0.1). The different points along the retain-forget tradeoff curve come from varying when we stop distillation based on retain performance thresholds {±0.05, ±0.09, ±0.13, ±0.17, ±0.21, ±0.25, ±0.29, ±0.33}, not from different amounts of noising. \n\n---\n\n### W4: Robustness beyond gradient-based attacks\n\n*From review text: \"Robustness is measured via gradient-based relearning attacks (finetuning on the forget set). It would be beneficial to also compare against jailbreaking attacks in inference time.\"*\n\nWe first note that prior literature established that finetuning is the strongest elicitation technique. [2, 3] found that while fine-tuning consistently performed best across all tested setups, other attack vectors showed less consistent success. This universal vulnerability to fine-tuning, combined with its ease of implementation and replicability across research groups, motivated our focus on fine-tuning.\n\nWe also tested (inference-time) quantization attacks. Recent work [5] showed that quantization can bring back unlearned knowledge:\n\n**Table: INT4 Quantization Robustness (TOFU, Gemma-2-2B)**\n\n| Method | Forget Q&A Prob | Forget Truth Ratio | Model Utility |\n|--------|----------------|-------------------|---------------|\n| GradDiff (FP16) | 0.000 | 0.000 | 0.495 |\n| GradDiff (INT4) | 0.008 | 0.185 | 0.242 |\n| GradDiff+Distilled (FP16) | 0.000 | 0.001 | 0.442 |\n| GradDiff+Distilled (INT4) | 0.003 | 0.017 | 0.391 |\n\nOur distilled models show dramatically better resilience to quantization attacks across all metrics. We highlight forget truth ratio (true utterance about the forget set authors) improvement (0.185 vs 0.017). This suggests that distillation provides defense against orthogonal attack vectors beyond just gradient-based relearning.\n\nAdditionally, we have tested prompt-based and activation steering elicitation methods on our bilingual models. However, it was challenging to get clear signals, potentially due to the relatively small model sizes we used (100M-300M parameters) or limitations in applying these techniques to our specific setups. \n\n---\n\n### Q: Figure 7a performance \n\nThis phenomenon actually has a clear explanation. Our Korean dataset (FineWeb split) contains some English characters, so training on English alone still slightly improves Korean performance, though it remains close to random (CE loss ~11-12).\n\nMethods like MaxEnt explicitly maximize entropy on the forget set, pushing outputs toward uniform distributions. This can achieve a cross-entropy loss closer to the \"random\" baseline than what the gold standard data filtering naturally achieves. As shown in Figure 7a, MaxEnt pushes Korean CE loss close to around 12.5, while the gold standard (trained only on English) settles around 11. \n\nA more disruptive method like GradDiff, which performs a negative gradient update against the forget set, can push Korean CE loss to the regimes of 22~25, which also explains the distillation dynamics in Figure 9. However, CE loss values above ~11 represent essentially non-linguistic outputs. The model isn't producing \"better forgetting\" but rather completely uninformative tokens.\n\n---\n\n— We thank the reviewer for helpful questions! —\n\n\n[1] TOFU: A Task of Fictitious Unlearning for LLMs. COLM 24\n\n[2] WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. ICML 24\n\n[3] The elicitation game: Evaluating capability elicitation techniques. ICML 25\n\n[4] Eight methods to evaluate robust unlearning in llms. Preprint\n\n[5] Catastrophic Failure of LLM Unlearning via Quantization. ICLR 25"}]

Author rebuttal:

Meta-review: This paper addresses a critical robustness problem in machine unlearning for large language models. The authors make the key observation that distilling an unlearned model into a freshly initialized student robustifies the forgetting process, and further propose UNDO (Unlearn-Noise-Distill-on-Outputs) as a practical extension that interpolates between noising and full reinitialization. 

Reviewers broadly praised the core insight, noting that using distillation as a capability filter is a technically and conceptually novel contribution. The evaluation is thorough, with additional experiments during rebuttal. For improvement, the main limitations concern (1) reliance on access to pretraining data for effective distillation, which restricts applicability to settings where data is not available, and (2) limited explanation of why distillation achieves robust unlearning. After the rebuttal, reviewers updated their scores positively. 

The AC concurs with the reviewers that the overall strengths of the work clearly outweigh its weaknesses, and therefore recommends acceptance.
Decision: Accept (spotlight)