From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization
Authors: Shoaib Ahmed Siddiqui
Adrian Weller
David Krueger
Gintare Karolina Dziugaite
Michael Curtis Mozer
Eleni Triantafillou
Venue: NeurIPS 2025 poster ()
Keywords: Unlearning
tamper-resistance
relearning attacks
weight-space analysis
Primary area: deep_learning
TL;DR: We highlight the susceptibility of existing unlearning methods to relearning attacks and analyze the characteristics of robust methods by leveraging the weight-space perspective.

Abstract: Recent unlearning methods for LLMs are vulnerable to relearning attacks: knowledge believed-to-be-unlearned re-emerges by fine-tuning on a small set of (even seemingly-unrelated) examples. We study this phenomenon in a controlled setting for example-level unlearning in vision classifiers. We make the surprising discovery that forget-set accuracy can recover from around 50\% post-unlearning to nearly 100\% with fine-tuning on just the *retain* set---i.e., zero examples of the forget set. We observe this effect across a wide variety of unlearning methods, whereas for a model retrained from scratch excluding the forget set (gold standard), the accuracy remains at 50\%. We observe that resistance to relearning attacks can be predicted by weight-space properties, specifically, $L_2$-distance and linear mode connectivity between the original and the unlearned model. Leveraging this insight, we propose a new class of methods that achieve state-of-the-art resistance to relearning attacks.

From Dormant to Deleted: Tamper-Resistant
Unlearning Through Weight-Space Regularization
Shoaíb Ahmed Siddiqui University of Cambridge
Adrian Weller University of Cambridge The Alan Turing Institute
David Krueger Mila
Gintare Karolina Dziugaite Google DeepMind Mila
Michael C. Mozer Google DeepMind
Eleni Triantafillou Google DeepMind
Abstract
Recent unlearning methods for LLMs are vulnerable to relearning attacks: knowl- edge selovvd-tt-bb-uuleamnee re-emerges by fine-tuning small set of (even evemingly-unrllate examples. We study this phenomenon in a controlled set- ting for example-leved unlearning in vision classifiers. We make the surprising discovery that forget-set accuracy can recover from around 50% post-unlearning nearly 100% with fine-tuning on just the retain set-i.e., forget set. We observe this effect across a wide variety of unlearning methods, whereas for model retrained from scratch excluding the forget set (gold standard), the accuracy remains at 50%. We observe that resistance to relearning attacks be predicted by weight-space properties, specifically, La-distance and linear mode connectivity between the original and the unlearned model. Leveraging this insight, we propose new class of methods that achieve state-of-theaad resistance to relearning attacks
examples of the
Introduction
Machine unlearning is the problem of removing the influence of specific training datapoints, the
Retrain from scratch SCRUB
NegGrad+ Circuit Breakers
forger set, from pretrained model. This was ini-
tially motivated from the perspective privacy, the right-to-be-forgotte
Forget set accuracy
1.00
Unlearned
3 333, data protection
Relearned
policies (32), and recently applied to a range of problems, including removing harmful knowl-
0.50
edge 28 46 8
Exact unlearning refers to completely eliminat- ing the influence of the forget set. This objective can be achieved by retraining the model without those datapoints (i.e., retrain from scratch) (2] However, such method is computationally pro- hibitive as it requires retraining the model for every unlearning request (2). This issue moti- vated the development of approximate unlearn-
0.00
0.88
0.90
0.92
0.94
Test set accuracy
Fine-tuning an unlearned model just the retain set recovers performance on the forget set! Results on CIFAR-10 using a forget set of atypical examples from class 'airplane'.
Figure 1:
"ooreesonneecce to mmsas3ccam ac, uk
"odee reproduce experiments: https: httpa//wwww............../..a.aa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ing methods, where the aim instead is to only approximately remove the influence of the given
datapoints [5] 26 28 46]. in exchange for greater efficiency. While such methods may seemingly succeed at matching the retrained- wii--froomssaare model on some simple metrics, like the accuracy on the forget set, it is unclear they permanently remove the influence of these datapoints [6 In fact,
evaluating whether they do is research problem in its own right [16, 42).
In this work, we take different perspective, focusing on relearing attacks on unlearned models. We consider tampering attacks 4 where the attacker is able to fine-tune the model weights. This direction is inspired by a growing set of observations in the context of unlearning "knowledge" in LLMs: fine-tuning an unlearned model even on seemingly benign data may cause the believed-to- be-unlearned knowledge to re-emerge [20] 31 6 4. However, these studies are carried out under conditions that make it difficult to draw clear conclusions. First, these works study unlearning knowledge, capabilities, or topics, where the problem is inherently under specified. For example, given a dataset containing knowledge necessary for making bioweapons, the goal may be to fully remove the capability of constructing bioweapons, while preserving general knowledge of biology. In this setting, it is hard to draw clear line between forbidden and permissible knowledge and pinpoint all training examples responsible for acquiring different types of knowledge. Furthermore, knowledge is hard to measure, especially due to nuances of natural language, requiring question- answer evaluation which may be sensitive to the particular phrasing. Finally, because LLMs can make complex inferences beyond the training set, not all knowledge that is extractable should be attributed to a failure of unlearning [38]. In many cases, the acquisition of knowledge is natural and unavoidable, hence, making it difficult to distinguish between the two in these problem settings.
To address these issues, we study relearning attacks in a setting that allows for controlled ex- perimentation: unlearning specific training examples from (small) vision classification models, a problem for which a plethora of approximate unlearning methods have been developed and tested 14 15 13 26
45 36). In example-level unlearning, the gold standard is clear, namely, to retrain the model without the forget set. As our models are small enough, we can compute the gold-standard solution and use it as the reference point for comparison. Because we consider clas- sification, we can also use accuracy as a simple and well-undere well-understood measure of performance. These properties combined allow to compare the tamper resistance of different unlearning algorithms with the correct reference point, and therefore draw conclusions about their quality.
41
We evaluate a range of prreaasinnlyccomple unlearning algorithms in this setting and discover a surprising finding: for numerous unlearning algorithms, the accuracy of the forget set jumps from around 50% post-unlearning to nearly 100% after fine-tuning the unlearned models on only the retain set, which is disjoint from the forget set. Fig. shows this phenomenon on CIFAR-10 10 using ResNet- 18, after having attempted to unlearn a subset of atypical instances of class 'airplane'. We note that a model retrained from scratch without the forget set does not exhibit this behaviour, with the accuracy remaining at 50%. Therefore, the recovery of forget set accuracy can be safely interpreted as a failure of these algorithms to fully remove the influence of the datapoints in the forget set.
Our extensive analyses leads to multiple insights. First, unlearning and relearning of typical examples trivial, and unlearning methods behave similarly to the gold standard. However, we see stark contrast in their respective patterns of behaviour on forget set of atypical examples. Furthermore, taking a weight-space view [2], we discover a key characteristic of unlearning algorithms that are better at resisting these attacks: they yield an unlearned model that is distant from the pretrained model in the weight-space Based on this insight, we propose a new class of unlearning algorithms that are superior in terms of resisting relearning attacks by incorporating terms in their objective that encourage the unlearned model to move far away from the pretrained model in the weight-space.
To summarize, we make the following contributions in this work:
We show that unlearning algorithms fail to delete the influence of the forget set, which stays dormant and can resurface by fine-tuning even on just the retain set.
We identify a key characteristic of methods that are more robust against relearning attacks, namely: the unlearned model is distant from the pretrained model in weight space.
Leveraging this insight, we propose a new class of unlearning methods that attempt to push the unleamed model far away from the pretrained model. These methods are significantly more robust against relearning attacks in comparison to unlearning methods that operate only at the output-level (26) or the representation--lvell [28 46).

---

2 Background and Related Work
Unlearning. The problem of machine unlearning was introduced by B. The goal is to remove the influence of a "forget set" from model that was trained on a dataset including that set. This was motivated by privacy and hight-to-be-foroonn
policies [2). The perfect unlearning method, from the perspective of fully erasing the influence of the forget set, is to simply retrain the model excluding that set. However, the computationally prohibitive training costs make such an approach infeasible in most practical cases. [2] propose to shard the dataset, and train an ensemble model over it, allowing to selectively retrain only the affected parameters. However, the computational cost is still prohibitive in the worst case, while also leading to poorer performance in some cases due to the use of specialized architectures. These issues motivated the development of approximate methods that accept imperfect unlearning in exchange for greater efficiency. This is the family of methods focus on in this work. The goal in approximate unlearning is to post-process the trained model as efficiently possible in order to closely match the model which is retrained from scratch using only
a small amount of model fine-tuning [21 I 26 28 46
42]. This is a challenging problem as imperfect attempts erase the influence of the forget set post-hoc may have number of unwanted side-effects, such as harming the overall utility of the model 42
45
Unlearning quality metries. Since most approximate unlearning methods that applicable to deep models do not come with theoretical guarantees about the quality of their approximation, we are required to estimate how well they approximate retraining from scratch empirically. This is research problem in and of itself, and current rigorous metrics are very computationally expensive 42 16]. Furthermore, unlearning entails fundamental trade-offs, such between forgetting and maintaining the model's utility. This requires multifaceted evaluation metric that captures relevant factors aside from forgetting quality. Commonly, in vision classification, the accuracy on the retain set and the accuracy on the test set are used to measure model utility. In similar spirit to our work, time to relearn 14] quantifies the strength of unlearning by the number of optimization steps required to reacquire forgotten information by directly fine-tuning on it. We instead show that we can restore forget set accuracy even when fine-tuning on only a subset of it, or solely on the retain set.
Re-emergence of models showed that subset of the forget set or even on eeemiggyyunnrlltee data [20, 31 6 4]. Relatedly, it has also been shown that fine-tuming
knowledge via fine-tuning, Recent work in language knowledge re-emerge by fine-tuning
small
language model on benign inputs can reverse the safety tuning of the model 334 27]. A key distinction sets our work apart from all prior efforts. They study unlearning knowledge, or capabilities, rather than specific training examples. Their goal is to remove unwanted knowledge beyond the specific instances in the forget set, e.g., fully remove a dangerous capability (such as bioweapon construction) after having unlearned on a specific dataset containing related knowledge [28]. This problem is inherently less well-specified compared to unlearning specific examples where we have a clear definition of the ideal solution, namely, retraining from scratch without the specific examples. In LLMs, measuring knowledge is also nuanced, requiring question-answering answering tools, for instance, where the success of extracting knowledge may depend on the phrasing [47]. We study relearning attacks for example-level unlearning in vision classifiers, a setting where the forget set is speciifed and the goal well-defined and simple to measure.
3 Problem Formulation
Let D,y denote a training set and A a learning algorithm. Let Mp = A(MI, Der) denote the "pretrained model", obtained by training Dtr. starting from random initialization M,. Now, let DF C. Der denote a forget set that we want to unlearn, and let DR = Dzr \ Dr denote the retain set.
The goal of an unlearning algorithm 2, is to post-process the pretrained model Mp to remove the influence of Dr. Specifically, we denote an unlearning algorithm by U M X DR x Dr H M that takes in model, retain set DR. forget set Dr. and returns an unlearned model My (MMp, DR, Dr). Ideally, the unlearned model My should match the gold-standard "retrained- from-scratch" model Mns- A(M,, DR) which starts from a random initialization and trains on only the retain set, fully eliminating the influence of Dr. We desire unlearning algorithms that can approximate that solution but are much more efficient than retraining from scratch.
3

---

In this work, we study relearning attacks that apply further fine-tuning phase attempting to reintro- duce the forget set. Such attacks that are able to modify the model's 's weights are referred to in the literature as tampering attacks 4 40 We carry out these attacks by fine-tuning the model on the union of Dr and a subset of "relearning examples" Dr... C Dr. We denote the relearned model as MRL = (M,, DR U FF.,), ), where A' denotes fine-tuning algorithm used for relearning (which might be similar to A with slightly different hyperparameters) and M can be either My or .MRs.
We measure performance on the held-out portion of the forget set (held-out from the perspective that it was not used during releamingg), denoted D Pr... = Dr D Dr... We vary the size of Dr,, and measure the effect on relearning. We also measure performance on a test set Die. to measure utility.
An ideal unlearning algorithm is one that is tamper resistant: upon relearning. its accuracy on the forget set does not increase more than it would by learning the relearning set anew. In other words, the forget set accuracy of A' A'Muu, Dn U 'Dr..) should not be higher than that of A'(Mns: ns; Dn U U'Dr,..) At the same time, an ideal unlearning algorithm would not sacrifice the test accuracy.
Threat model. Similar to tampering attacks considered for LLMs 40 31 20
we assume that the defender has access to a pretrained model, and performs unlearning using any algorithm of their choice. Furthermore, we assume that the attacker has white-box access to the unlearned model provided by the defender, the retain set DR. and limited access to the forget set (i.e., the relearning set D...). The goal of the attacker is to recover performance on the full forget set Dr, while minimizing the number of unlearned examples needed Dr., (as relearning becomes trivial if DF., = Dr). We also consider extreme-and perhaps more realistic- case of access to only Dn.
4 Experimental Setup
Models and Datasets. We use two different models for our evaluation from the ResNet model family [17], namely 18 and ResNet-34 [7.. In terms of datasets, we use CIFAR-10 and CIFAR- 100 datasets [25] with 10 and 100 classes respectively, and a total of 50k training instances in each case (5k instances per class for CI and 500 instances per class for CIFAR-100
Evaluation. All models are evaluated in terms of accuracy on the held-out part of the forget set Drao (same subset between all models), as well as the test set Dre. While we can report accuracy on the full forget set Dp for the unlearned model, we instead report accuracy on the remaining subset in order to enable direct comparison of the impact of the relearning attack. The line plots show accuracy every 10 optimization steps. When reporting results using a scatter plot, we average the test set accuracy as well as the accuracy on the forget set for the last 50 steps reported in the line plots.
Pretraining. We pretrain the model for 300 epochs using Adam optimizer (4 with a learning rate of le-4, cosine learning rate decay with a decay factor of 0.1, batch size of 128, and a weight decay of le-4 in all configurations.
Unlearning We consider two unlearning settings: sub-class unlearning, where the forget set consists of 10% of the class instances (here, sub-class means a subset of the complete class), and class-agnostic unlearning, where we select 1% of the data set regardless of class labels. This ensures that we use the same number of examples in the forget set for both settings on CIFAR-10 (we only evaluate sub-class unlearning on CIFAR-100). We use smaller learning rate of le-5 without any weight decay and optimize the model for a 100 epochs during the unlearning phase.
Relearning attack. During this phase, we fine-tune on a combination of the retain set Dn and a subset of the forget set for relearning (Dr..) ). We explore the impact of different choices for relearning examples in Appendix G We again use a small learning rate of le-5 without any weight decay, and optimize the model for just 10 epochs (except Fig. [T where we optimized the model for 300 epochs). Similar to the pretraining stage, we use cosine learning rate decay with decay factor of O.
4.1 Baseline Unlearning Methods
We consider a range of different baseline unlearning methods. Each method has its own set of hyperparameters. We attempted to select the hyperparameters to achieve good trade-off between the test accuracy and the forget set accuracy for each unlcarning method.

---

Retrain from scratch
Sparse
Weight Drapout Weight Distortion
Random Relabeling
Catastrophic Forgetting
SCRUB
SSD
CBFT
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget set accuracy
unlearned model
relearning examples
10 relearning examples
100 relearning examples
1.00
0.50
0.00
0.80
0.90
0.80
0.90
0.80
0.90
0.80
0.90
Test set accuracy
Forget set accuracy
unlearedd model
relearming examples
relearning examples
releaming examples
1.00
0.50
0.00
0.70
0.80
0.90 0.70
0.80
0.90
0.70
0.80
0.90
0.70
0.80
0.90
Test set accuracy
Figure 2: Scatter-plots with test-set accuracy the x-axis and accuracy the held-out portion of the forget set, Pr... on the y-axis. The left-most subplot indicates performance immediately following unlearning. The next three subplots are following relearning attack with instances of the retain set, DR and a varying number of instances of the forget set, Dr. (0, 10, and 100, respectively). Each point the average performance of the last 50 steps (see Fig. 1T for the whole trajectory for sub-class unlearning and Fig. T2 for the trajectory for class-agnostic unlearning). The forget set is comprised of atypical examples (from the 'airplane' class, i.e., sub-class unlearning for the top row and all classes, i.e., class-agnostic unlearning in the bottom row) in CIFAR-10. The figure indicates that many methods achieve near-perfect recovery unlearned knowledge with only small amount of model fine-tuning, even with 0 relearning examples (fine-tuning on only the retain set). Weight Distortion, CBFT, and Weight Dist Reg are introduced in Section 6
SCRUB 26 uses two-phase training procedure where it interleaves iterations the forget set and the retain set. The loss function minimizes KL-divergence between the pretrained model and the unlearned model output distributions the retain set, along with cross-entropy the true labels. For unlearning, maximizes the KL-divergence on the forget set between the distributions of the pretrained model and the unleamed model.
Circuit Breakers 46 28 was proposed particularly to unlearn knowledge in language models. The training procedure attempts to push the representations apart by minimizing cosine similarity with the pretrained model on the forget set, while minimizing the Euclidean distance the representations on the retain avoid model collapse. We apply circuit breaker loss layer 4 and layer 7 of our models, which is motivated by the fractional depth considered in the original work.
NegGrad+ [26] maximizes the cross-entropy loss the forget set, while minimizing the loss on the retain set. used the alternating variant (similar to SCRUB) instead joint optimization of two losses, as it resulted in better test accuracy as well as lower forget set accuracy.
Catastrophic Forgetting [42 uses repeated fine-tuning on the retain set with a weight decay, which naturally leads to a decay in the magnitude of the parameters that are unimportant for the forget set. We a weight decay of 0.001 for all our models.
L1-Sparse 21 is similar to the Catastrophic forgetting in our case, except that minimizes L Lorm instead of the Ly-norm employed in weight decay.
Selective Synaptic Dampening (SSD) [T] identifies model weights to dampen based on their importance for the retain set and forget set, quantified using the Fisher information matrix. In contrast to the original paper, we follow this process with fine-tuning on the retain set to consistent with other baselines, giving SSD better shot at repairing test accuracy.
5

---

Retrain from scratch
TAR
CBFT
Weight Dist Reg
Forget set accuracy
SCRUB
Circuit Breakers
Weight Distortion
1.00
0.50
0.00
0.85
0.90
0.85
0.90
0.85 0.90
Test set accuracy
Figure 3: Comparison between test set accuracy and accuracy the held-out part of the forget set Dr.. after relearning, for subclass unlearning of atypical examples CIFAR-10. We consider two-phase unlearning methods: first, an initial safeguard (unlearning phase) is applied, with the unlearning algorithm mentioned as the subplot title. Then, each of TAR, CBFT, and Weight Dist Reg applied as second phase for increasing the tamper-resistanee. The '+' symbol represents the performance of the initial safeguard for reference. We observe that TAR fails to add any tamper- resistance in addition to that of the initial safeguard despite being designed for this.
Random Relabeling [5] relabels every example in the forget set with random label. Note that we used an aggressive version of random relabeling where we re-assign a new label at every fine-tuning step. Hence, this can be considered analogous to minimizing divergence to a uniform distribution.
Weight Attenuation attenuates all model weights with a fixed attention factor of 0.5, followed by simple fine-tuning on the retain set.
Weight Dropout performs random (unstructured) pruming with dropout factor of 0.2 (i.e., zeroing out 20% the model weights), followed by simple fine-tuning on the retain set.
Tampering Attack Resistance (TAR) 40), originally proposed for LLM unlearming, defines a bi- level optimization, starting from an already unlearned model, with the aim to make resistant against tampering attacks. It uses first-order approximation of inner adversaries, and attempts to maximize the entropy of the model predictions after fine-tuning of the unlearned model. TAR also uses a representation alignment loss which minimizes the Euclidean distance between the representations of the initially unlearned model and the current model to avoid model collapse. Similar to Circuit Breakers, we apply the representation alignment loss on layer and layer of our models. Note that TAR relies on an unlearned model as a starting point, making it two-phase approach.
5 Recent Unlearning Algorithms are not Tamper-Resistant
We begin by investigating the tamper-resistane of current state-of-the-ar thr unlearning methods. We use CIFAR-10 10 and CIFAR-100 with R8sNet-18, and attempt to unlearn instances of a single class ('airplane' class for CIFAR-10, and 'apple' class for CIFAR-100), across classes (class agnostic).
The role of typicality for learning and unlearning. Not all examples are equally easy to unlearn, and not all examples are equally easy to (re)learn. Typical examples can be particularly easy unlearn as even a no-op unlearning already yields similar predictions on these examples as the retrained from scratch model (21). This is because they are, by definition, easy to predict regardless of whether they are part of the training set or not [TO 22 II 39). Generally, because typical examples are more likely to be predicted correctly, they casier to relearn well. Based this, we hypothesize that the typicality of examples is a key property that will determine relearning patterns and differences between unlearning algorithms compared to retraining from scratch.
To investigate this, we study three different settings, characterized by different typicality levels, where the forget set contains instances of the 'airplane' class that are: (i) most likely to be typical, (ii) randomly selected, and (iii) most likely to be atypical. We leverage pre-computed consistency scores from 22 to separate typical and atypical instances, by treating instances with the highest consistency scores as typical instances, while treating instances with the lowest consistency scores as atypical instances. Other scoring schemes are also equally applicable [10] 39 I.

---

When the forget set consists of typical examples, relearning is trivial and uninteresting, The reason that accuracy of the retrain-fromm from-scratch model is essentially perfect on the forget set, even before relearning is applied, and all unlearning methods exhibit similar behavior (for the sake of completeness, we show the results for this case in Fig- 7 Appendix A). Similarly, when the forget set consists of randomly selected examples, performance of retrain-from frommscratch is nearly perfect because randomly selected examples are predominantly typical [10 2
[9] (again, for completeness, this case is shown in Fig. 8 Appendix B). Consequently, we focus on atypical forget set items for the remainder of the paper: in this case, retrain-from-scratee will not predict correctly prior to relearning and thus we can measure the effectiveness of relearning attack.
Relearning attacks succeed against several unlearning baselines. We present the results for sub-class unlearning for the forget set of atypical examples in Fig. (top). In this and subsequent figures, existing methods are indicated by pastel colors. New methods, to be introduced shortly, are represented by saturated colors. For the most part, the existing methods all behave similarly and the reader need not attend to the individual methods. The accuracy of retrain from scratch is less than 50% on that forget set, and remains almost exactly the same when subjected to the relearning attack of fine-tuning only on the retain set (i.e., relearning examples). The accuracy of this model shifts up slightly as we increase the number of relearning examples from the forget set (going from left to right). In stark contrast, the different unlearning methods we evaluate show a qualitatively different trend. We make the striking observation that some methods (such as Circuit Breakers, SCRUB, and Random Relabeling) are very susceptible to relearning attacks. For these methods, forget-set accuracy drops down after unlearning, near the desired reference point of the retrained model. However, upon relearning even on just the retain set, the model achieves near-perfect accuracy on the forget set-a jump from near 50% post-unlearning to nearly 100% after relearning.
Sub-class vs class-agnostic unlearning. We further attempt to understand if such differences exist in class-agnostic forget-set setting (with the same number of forget-set examples, i.e., 500), which is now comprised of atypical examples across classes. Fig. 2 (bottom) shows that class-agnostic unlearning better differentiates among methods compared to sub-class unlearning. We see wider pre-relearning). since aggnosic
spread among methods, even in the first subplot gpost-unleaminng unlearning is harder. We also observe that while for sub-class unlearning, the performance of retrain- from-scratch (black star) on Dr.,, shifts upwards as more relearning examples are used, this shift does not occur with class-agnostic unlearning. This result is expected since relearning on a larger number of atypical examples does not improve performance on a disjoint set of other atypical examples (by definition of atypicality (1O). There is again stark contrast between the performance of retrain- from-scratch model on Dr. where post-relearning accuracy remains very low, and many unlearning algorithms, where post-relearning accuracy again reaches nearly 100%. Importantly, the relative ranking between methods in the sub-class and the class-agnostic case remains consistent.
Relearning attacks succeed even against methods designed for tamper-resistance. We further compare two-phase methods that assume an initial unlcarning round, followed by a subsequent train- ing round in order to reduce susceptibility to relearning attacks. This is inspired by the methodology of TAR [4), which is explicitly designed to increase the resistance of the unlearned model against mme-tuning--bsee relearning attacks. Despite TAR's success in the case of language models, Fig. shows that it struggles to provide any resistance against relearning attacks in our case.
6 A Weight-Space View on Understanding and Improving Tamper-Resistand
In the previous section, we demonstrated the susceptibility of existing prominent unlearning methods to tampering (Fig. 2) However, it is unclear what makes a method vulnerable or robust to these relearning attacks. In this section, we shed light on this question through a weight-space view. Specifically, we hypothesize that the susceptibility of unlearned model to relearning may be associated with failing to move 'far enough' from the pretrained model in weight-space. We explore this hypothesis from two perspectives: (i) by measuring distances in weight-space, and (ii) through Linear Mode Connectivity analysis [12]. We then use these tools to interpret the rasitrere profiles of proppssed unlearning algorithms (Fig, 2 that show better tamper- resistacee (such as Catastrophic Forgetting and L1 Sparse) compared to the ones that exhibit worse tamper- resistance (such as Random Relabeling, Circuit Breakers, and SCRUB).
7

---

Breakers
Weight Reg
1.0
1.0
1.0
1.0
1.0
Accuracy
0.5
0.5
0.5
0.5
0.5
Test Set Retain
Farget Set
0.0
0.0
0.0
0.0
0.0
0
0
0
Mixing weight
Figure 4: Linear mode connectivity analysis on CIFAR-10, where the forget set is comprised of atypical examples. We construct linear path between the pretrained and the unlearned (or retrained- from-scratch) model by interpolating the model parameters and batch-norm statistics using different mixing weights (shown the x-axis). We report accuracy the y-axis. 0 the x-axis represents the pretrained model, while I represents the unlearned or retrained model. Retrain-from-scratch not linearly connected the pretrained model, whereas for unlearning algorithms, the resulting unlearned model is in many cases still linearly connected to the pretrained one.
Weight-space distance. Fig. 5 plots L2 dis- tance between the pretrained model and the un- learned model. We see that existing methods that we previously showed be susceptible to relearning (pastel colors) induce only small movement in parameter space, indicated by the small L2 distance.
M|Mp-MMllz
||M Mullz
10*
10a
Retrain from scratch
Random Relabeling
SCRUB-
NegGrad +-
Circuit Breakers
55D
CBFT
L1 Sparse
Weight Dropout
Weight Attenuation
Catastrophic Forgetting
Weight Distortion
Weight Dist Reg
that methods with higher Lg distance exhibit higher robustness against re- learning attacks presented in Fig. 2 Notably, two methods based on these insights which we describe shortly, Weight Distortion and Weight Dist Reg, have the highest L2-norm and higher samper-resistanee We remark that, out of pre- vious methods, those with increased tamper- resistance (such as Catastrophic Forgetting and L1 Sparse) have comparatively higher distance norm compared to methods with very poor tam- per resistance (such as Random Relabeling, Circuit Breakers, and SCRUB). We note that both Catastrophic Forgetting and L1 Sparse use weight-space regularizers (L2 and L1, respec- tively). Fine-tuning without any regularizer failed to unlearn the forget set in evaluations, which aligns with our weight-space interpretation. We consider more elaborate analysis regarding layers that are particularly important for robustness against relearning attacks an interesting direction for future work.
Generally,
Figure 5: L2 norm of the difference between the parameters of the pretrained and the unlearned models induced by different unlearning methods. We consider only the difference in the parame- ters, while ignoring the batch-norm statistics for ResNet-1 8 trained on CIFAR-10 10, where the forget set is comprised of atypical examples.
Linear Mode Connectivity (LMC). We further investigate the relationship in weight-space be- tween the unlearned and the retrained from scratch model via the lens of LMC [12. We plot the accuracy when interpolating between the two models (where we interpolate both model parameters as well model batch-norm statistics) along linear path. We compare the pretrained and retrained from scratch model (which are trained from the same initialization), different unlearned models in Fig. 4 Looking at the retrain from scratch plot, we see that there is high-loss barrier between the two models, meaning that they are not in linearly connected modes. The same holds for methods with some tamper resistance (like Catastrophic Forgetting and L1 Sparse) but not for those vulnerable to tampering attacks like SCRUB, where observe no such barrier.
well the pretrained and
6.1 A New Class of Taapee-Resistant Unlearning Methods
now leverage these insights to propose a class unlearning methods that are designed with lamper-resistance in mind. We achieve this through objectives that either aim to induce large

---

distance in the weight-space, or a loss barrier between the pretrained and unlearned models. Hence, any method that directly or indirectly attempts to separate out the pretrained model and the unlearned model is an instantiation of this framework.
Weight Distortion. This is very simple method that adds isotropic zero-mean Gaussian noise to all model weights with fixed standard deviation of 0.02, followed by simple fine-tuning on the retain set. We hypothesize that the addition of noise facilitates moving away from the pretrained model.
Weight Dist Reg. We directly aim to maximize the distance between the pretrained and unlearned models, by explicitly adding a term that quantifies the Euclidean distance between the two models. We maximize this loss during training, while minimizing the loss on the retain set.
Connectivity-Bant Based Vine-Tuuiigg (CBFT). We employ the method from [30], which was originally proposed to obtain models focusing on distinct recognition mechanisms. We maximize the loss on the midpoint between the pretrained model and the current unlearned model on examples from the retain set as well as the forget set, while only minimizing the loss on the retain set for the unlearned model. This attempts add high-loss barrier in between, while still retaining model utility on the retain set for the final unlearned model. We use a small weighting factor of 0.001 on the loss maximization term. Similar to (00, we ignore the loss maximization term if the loss magnitude exceeds value of 50.
Findings. Fig. 2 shows that Weight Distortion and Weight Dist Reg are significantly more tamper- resistant comparison to prior approaches. However, we observe that CBFT is less effective than Weight Dist Reg and Weight Distortion across the board. We hypothesize that this is due to acting on model outputs and only indirectly inftuencing the weight-space. Indeed, while CBFT does create a larger loss barrier than other methods (Fig. 4. the L2 norm between the pretrained and unlearned parameters is relatively low (Fig.5) Overall, out of the two weight-space diagnostic tools, we find that the L2 norm of the difference in model parameters is more reliable in predicting tamper-resistance. Fig. 3 shows that both Weight Dist Reg and CBFT, when applied as a second phase on an initial safeguard, can substantially improve its tamper- resistanee unlike TAR. The only exception is if the initial safeguard is Weight Distortion, which already has sufficient tamper-resistanee
7 Analysis and Discussion
On the trade-off between respereree and test accuracy. As discussed previously, there inherent trade-offs in unlearning. between forgetting the specified examples while maintaining utility (measured via test accuracy) [26 6 42 Here, we discover different trade-off between resisting relearning attacks, and test accuracy. Indeed, Fig. irg-2ishon shows that the methods that are best at defending against these attacks are the ones with the lowest test accuracy. Having surfaced this fundamental tension, we hope future work improves on the current Pareto frontier formed by our new methods.
Findings hold across datasets/archittecuue The results presented in the paper consistent across models i.c., ResNet-34 on CIFAR-IO as presented in Fig. IO Appendix E Furthermore, these results also consistent across different datasets, i.c., CIFAR- 100, as highlighted in Fig. 9 Appendix C
Finally, our findings are also consistent on a higher-resolution Imagenette dataset [9 as highlighted in Table Appendix [
Efficiency of Unlearning Algorithms. We used a fix budget of 100 epochs for unlearning to avoid
confounding from differences in optimization efficiency. In order to justify the use of approximate unlearning algorithms instead of exact unlearning (i.e., retrain from scratch) with a training budget comparable to pretraining, we evaluate the efficiency of unlearning algorithms in Table Appendix ! The results show that our methods are much more efficient at unlcarning (and consequently resisting re-emergence of knowledge) even after single epoch of unlearning compared to other approaches.
Difference between Quantization and Relearning Attacks. Language models have been shown to be susceptible to quantization attacks (4), where quantization after unlearning is sufficient to recover unlearned knowledge from the model. We evaluate susceptibility to quantization in Fig. 6 Interestingly, our vision models are not vulnerable to these quantization attacks: forget set accuracy stays at the same level as the full-precision unlearned model regardless of the number of bits until it gets close to 8 bits, at which point the model collapses. Thus, the spontaneous-recoveey phenomenon we explore via relearning attacks is distinct from the recovery obtained via quantization. We hypothesize that the primary reason why quantization attacks succeed in language models is due

---

Rctrain From Scratch
SCRUB
Circuit Breakers
CBFT
Weight Reg
1.0
1.0
1.0
1.0
1.0
Accuracy
0.5
0.5
0.5
0.5
0.5
Text Set
Retbin Set
0.0
0.0
0.0
0.0
0.0
Forget Set
10
20
30 10
20
30
10
20 30
10
20
30
10
20
30
Number of quantized bits
Figure 6: Comparison between the number of quantized bits and accuracy of the model for different unlearning methods on CIFAR-I0 and ResNet-18, where the forget is comprised of atypical examples. In contrast to language models, where quantization has been shown to be responsible for the re-emergence of unlearned knowledge [44] we observe that quantization has little no impact on forget set accuracy until the point where full performance is recovered. Note that the accuracy of 100% the start in most cases is an artifact of the model prediction collapsing to just class 0 (whose instances have been selected as our forget set).
to their larger number of parameters (on the order of 100 billion parameters 7 22 ). resulting in very small changes in weights during unlearming. Hence, models easily revert to their previous values with quantization. This problem is further exacerbated with models that are trained with quantization in mind. Such models can immediately latch onto the previous values that were regularized to be easily quantizable (23]. Similar to our findings, 44 hypothesized that small differences are the cause of this susceptibility quantization attacks. They further proposed increasing the learning rate as potential mechanism to induce larger differences between the pretrained and unlearned models 44] while work introduces more principled approaches for the same purpose.
The role of the retain set for relearning. Given the surprising finding that we can recover forget set accuracy while fine-tuning on only the retain set, we ask: are the retain set examples necessary for this to occur, or could we have used any other examples from the same distribution? We investigate this in Appendix G where we replace the retain set with different sets, and fine-tune the unlearned model on those different sets (combined with the 'relearning set', which is subset of the forget set, as before). Notably, replacing the retain set with test examples (that the model was not previously trained on) causes the relearning effect to be a bit less pronounced, especially when 0, or few, relearning examples are used, highlighting the importance of using training rather than held-out data for inducing relearning. This observation relates to other recent findings on anticipatory knowledge reawakening when exposing the model to a repeated sequence of documents. In this scenario, the model processes documents in a fixed order, it unexpectedly begins to recover an increasing amount of information about a previously seen example even before encountering that example again [43]
8 Conclusion
We showed that unlearning methods susceptible to relearning attacks where the forget set accuracy can be recovered simply by fine-tuning on the retain set. For atypical examples in particular, there is stark contrast between the relearning patterns of unlearning methods compared to retraining from scratch. Based on weight-space analysis, we suggest two diagnostic tools for understanding tamper- resistance, and propose simple methods that yield state-of-the new pathways for improving unlearning. The authors of (36) argued that unlearning algorithms that operate on representations rather than the level of outputs may be more robust at defending some types of attacks. Our findings take this discussion further, showing that unlearning methods that operate at either the level of the model outputs or the representations, without any constraint on model weights (which include methods such as SCRUB, Circuit Breakers, and Gradient Ascent, etc) should struggle to induce robustness against relearning attacks. On the other hand, methods that directly or indirectly push the pretrained and the unlearned models apart via any intervention (which includes distorting model weights, regularizing the model to decay parameter magnitude, or even directly pushing the models apart using explicit loss term) should be significantly more robust to relearning attacks. Our proposed methods for increasing tamper-resistance exemplify this and we hope future work builds on these further.
ramper-resistant results, revealing
10

---

Acknowledgements
The authors would like acknowledge useful discussions with Yanzhi Chen and Iia Shumailov regarding unlearning, and susceptibility to relearning. We are also very thankful to the anonymous reviewers for their useful feedback on the initial submission. AW acknowledges support from Turing AI Fellowship under grant EP/VO22279/1, the Alan Turing Institute, and the Leverhulme Trust via CFI.
References
[1]
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876-10889,
2021.
[2] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot, Machine unlearning, In 2021 IEEE symposium on security and privacy (SP), pages 141-159, IEEE, 2021.
[3] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463-480, IEEE, 2015.
[4] Zora Che, Stephen Casper, Robert Kirk, Anirudh Satheesh, Stewart Slocum, Lev E McKinney, Rohit Gandikota, Aidan Ewart, Domenic Rosati, Zichu Wu, et al. Model tampering attacks enable more rigorous evaluations of Ilm capabilities. arXiv preprint arXiv:2502.05200 2025.
[5] Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 37, pages 7210-7217, 2023.
[6] Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.0882/ 2024.
[7] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv-2407, 2024.
[8] Ronen Eldan and Mark Russinovich. Who's 's harry potter? approximate unlearning in Ilms. arXiv preprint arXiv:23 2023.
[9] Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium Theory of Computing, pages 954 959,9999990 2020.
[10] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891 2020.
[11] Jack Foster, Stefan Schoepf. and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 12043-12051, 2024.
[12] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. International Conference on Machine Learning, PMLR, 2020.
pages
[13] Shashwat Goel, Ameya Prabhu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, and Ponnurangam Kumaraguru. Towards adversarial evaluations for inexact machine unlearning. arXiv preprint 2rXiv:2201.066400 2022.
[14] Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. pixeac-rivacy forgetting deep networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 792-801, 2021.

---

[15] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516-11524, 2021.
[16] Jamie Hayes, lli Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid false sense of privacy. arXiv preprini arXiv:2403.012/68 2024.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 100:: 2019.
[19] Jeremy Howard. Imagenette, https://gthhbb.cco//sssa//ggggge 2019, Version 2.
fast.ai.
[20] Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Jogging the memory of unlearned model through targeted relearning attack. arXiv preprint arXiv:2406. 13356, 2024.
[21] Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51584-51605, 2023.
[22] Ziheng Jiang. Chiyuan Zhang, Kunal Talwar, and Michael Mozer. Characterizing structural regularities of labeled data in overparameterizee models. arXiv preprint arXiv:2002.03206, 2020.
[23] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4350-4359, 2019.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards un- bounded machine unlearning. Advances in neural information processing systems, 36, 2024.
[27] Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:23 2023.
[28) Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski. Shashwat Goel, Long Phan, et al. The wmdp benchmark: Mcasuring and reducing malicious use with unlcarning. arXiv preprint arXiv:2403.03218 2024.
[29] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19443 19437, 2024.
[30] Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In International Conference on Machine Learning, pages 22965-23004. PMLR, 2023.
[31] Jakub Lucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramer, and Javier Rando. An adversarial perspective on machine unlearning for ai safety. arXiv preprint arXiv:2409. 18025 2024.
[32] Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 29(3):229-235, 2013.
12

---

[33] Seth Neel, Aaron Roth, and Saeed Sharifi-Malvaipee Descent-to-delete: Gradient-base methods for machine unlearning. In Algorithmic Learning Theory, pages 931-962. PMLR, 2021.
[34] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.0
2023.
[35] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 12-222,, 2015.
[36] Nazanin Mohammadi Sepahvand, Eleni Triantafillou, Hugo Larochelle, Doina Precup, James J Clark, Daniel M Roy, and Gintare Karolina Dziugaite. Selective unlearning via representation erasure using domain adversarial training. In The Thirteenth International Conference on Learning Representations.
[37] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[38] Ili Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-limenez, Nicolas Paper- not, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai. arXiv preprint arXiv:2407.00106, 2024.
[39] Shoaib Ahmed Siddiqui, Nitarshan Rajkumar, Tegan Maharaj. David Krueger, and Sara Hooker. Metadata archaeology: Unearthing data subsets by leveraging training dynamics. arXiv preprini arXiv:2209. 10015, 2022.
[40] Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang. Ron Arel, et al. Tamper-resistant safeguards for open- open-weight Ilms, 2024. URL https://arxiv. org/abs/2408.007
[41] Reihaneh Torkzadehmahani, Reza Nasirigerdeh, Georgios Kaissis, Daniel Rueckert, Gintare Karolina Dziugaite, and Eleni Triantafillou. Improved localized machine unlearn- ing through the lens of memorization. arXiv preprint arXiv:24/2.022 2024.
[42] Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Jumior, Ioannis Mitliagkas, Jun Wan, et al. Are we making progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint arXiv:2406.00007, 2024.
[43] Yanlai Yang, Matt Jones, Michael C Mozer, and Mengye Ren. Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training. arXiv preprint arXiv:2403.09613. 2024.
[44] Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Catastrophic failure of Ilm unlearning via quantization. arXiv preprint arXiv:2410. 16454, 2024.
[45] Kairan Zhao, Meghdad Kurmanji. George-Octavinn Bárbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 37: 12233-3333,,, 2024.
[46] Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[47] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 20045 2023.
13

---

A Tamper-Resistance with Typical Examples for the Forget Set
As highlighted in Section typical examples can be particularly easy to unlearn as simply ignoring these examples during training (i.e., no-op) yields similar predictions as the retrained from scratch model [2I]. This is because they are, by definition, easy to predict regardless of whether they are part more likely to be predicted correctly, they are casier to relearn as well. For the sake of completeness, we include the results for typical examples in Fig. 7 As evident from the figure, even the retrained from scratch model already achieves perfect accuracy on the forget set, highlighting that unlcarning, and in turn, relearning are both trivial for typical examples. Therefore, we primarily focused atypical examples in Section 55] We see that methods such as Random Relabeling, while completing forgetting the forget set, deviates in a very significant way from the retrain from scratch baseline, which is what compare against.
of the training set not [10 22 [9]. Generally, because typical examples
B Tamper-Resistanee with Random Set of Examples for the Forget Set
We further evaluate performance in the case of random selection of examples, rather than just typical examples evaluated in Appendix A We visualize these results in Fig. 8 Since randomly selected examples are predominantly typical [10 9 39]. retrain from scratch already achieves near- rear-perfect accuracy. Furthermore, we observe only minor differences between methods. Consequently, we focused on atypical examples for the forget set in the main paper (Section [5) as these examples are hard to predict without being part of the training set, marking clearly the impact of relearning attacks which goes beyond model generalization.
C Tamper-Resistance on More Complex Dataset (CIFAR-100)
We predominantly focus on CIFAR-10 dataset in the main paper (Section 5) We evaluate the susceptibility on more complex CIFAR- dataset in Fig. in order to understand the impact the dataset. We only look at sub-class unlearning in this case, focusing on the 'apple" class for unlearning. Due to the higher complexity of the dataset, we observe lower test accuracies as evident from the scale of the x-axis. Retrain from scratch baseline achieves a low forget set accuracy. Furthermore, see a wider spread of different methods on this more complex dataset, with the trend looking similar to the class-agnostic results presented in Fig. 2 (as it provided more complex forget set comprising of the most atypical examples from the dataset). is interesting to note that CBFT [30], which was not tasper-resistant CIFAR-10, demonstrated significant resistance CIFAR-100.
D Tamper-Resistance on a Higher-Resolution Dataset (Imagenette)
All our prior evaluations focused on small resolution images, i.e., 32 x 32 for both CIFAR-10 and CIFAR-100. In order to understand the generalizability of our approaches to higher resolution images, we evaluate the higher-resolution ImageNette dataset [19], which is a subset of the ImageNet dataset (35). The results are presented in Table [ Similarly to our findings on both CIFAR-10 10 (Section 5) and CIFAR-100 (Appendix C we that the accuracy of retrain from scratch on the forget set after relearming stays close to 50%, while the accuracy of unlearning algorithms is significantly higher, except for our proposed weight dist reg algorithm among the tested methods.
E
Tamper-Resistanee of a Larger Model
All our prior results focused on the smaller model, i.c., ResNet-18 17]. In order to understand the impact of model size, we evaluate the susceptibility to relearning attacks on ResNet-34 (with almost double the number of parameters compared to ResNet-18, 18, going from ~ 11M to 21M). The results are visualized in Fig. 10, Note that we directly transfer the hyperparameter settings from our ResNet- 18 experiments. Hence, we found that Weight Dist Reg, which was the most competitive method in terms of robustness, became susceptible to relearning. It is worth noting that the test set accuracy as well as the forget set accuracy for Weight Dist Reg in this case deviates significantly from all prior results, where demonstrated consistently lower accuracies, highlighting potential
14

---

Model
Label
Test Acc (%)
Forget Set Acc (%)
Unlearned
Retrain from Scratch
87.34%
67.53%
Unlearned
SCRUB
85.81 %
85.71%
Unlearned
Weight Distortion
83.16%
74.03%
Unlearned
Weight Dist Reg
83.72%
84.42%
Relearn
Retrain from Scratch
86.40%
63.53%
0 Relearn
SCRUB
86.47%
93,40%
Relearn
Weight Distortion
84.71 %
78.86%
O Relearn
Weight Dist Reg
84.31 %
73.87%
10 Relearn
Retrain from Scratch
86.73%
64. 16%
Relearn
SCRUB
86.65% %
94.21
10 Relearn
Weight Distortion
84.62%
81.09%
10 Relearn
Weight Dist Reg
84.83%
75.77%
Table Test and forget set accuracy across unlearning and relearning conditions when evaluating on the higher-resolution Imagenette dataset [19] with atypical examples for the forget These results are primarily consistent with our findings on both CIFAR-10 10 as well as CIFAR-100 datasets, where models learned with proposed unlearning algorithm are much robust against relearning attacks.
deficiency of the selected hyperparameters. Catastrophic Forgetting and Weight Dist Reg exhibit high robustness against relearning, which are instantiations of our framework.
Evolution of Forget Set Accuracy During Relearning
we visualize the forget set accuracy evo- lution in Fig. [T. We see that methods that most stable nearly unaffected by increasing training time, highlighting that more training time might not be sufficient to increase their susceptibility further.
In order to complement the results in Fig. 2 (top) and Fig. 3
Similarly, Fig. 12 complements the results on class-agnostic unlearning in Fig. 2 (bottom), where visualize both the line plots well as the results from two-phase training strategies. We visually observe best separation between different methods on the line plot in this case of class-agnostic unlearning.
G Relearning with Examples from the Same Distribution, which are Distinct from the Retain Set
While demonstrated that unlearned knowledge be recovered through repeated fine-tuning on the retain set, it remains unclear whether we can also recover performance on the forget set by fine-tuning
examples from the same data distribution that unseen by the model. The question that we ask is: are the same examples that the model was trained on particularly important for relearning, other examples from the same data distribution equally effective?
In order to simulate this scenario, we use examples from the test set (which are unseen by the model) for relearning instead of the retain set. We further evaluate using corrupted version of the test set from CIFAR- 10-C [8]. In particular, we use the JPEG corrupted examples with the highest severity level of 5. Note that still use examples from the relearning set when considering scenarios where the number of relearning examples is > 0.
The results are presented in Fig. 13 We observe that using the retain set DR achieves higher accuracy on the forget set compared to other choices. However, these differences diminish as the number of relearning examples increases, since these examples directly represent the unlearned knowledge and may become the dominant factor recovery. When the test used for relearning (instead of the retain set), unsurprisingly observe perfect accuracy on the test set, as it serves as the training set
15

---

Unlearning Method
MIA Acc (%)
Retrain from Scratch
52.2%
LI Sparse
54.2%
SCRUB
51.6%
Circuit Breakers
55.0%
Gradient Ascent
54.2%
Random Relabeling
94.8%
Catastrophic Forgetting
54.2%
SSD
53.8%
Weight Attenuation
53.6%
Weight Distortion
53.2%
Weight Dropout
54.8%
Weight Dist Reg
48.4%
Table 2: Membership-Inferes Attack (MIA) accuracy, where 50% indicates chance-level per- formance. Note that most of the methods are close to chance-level performance, except random relabeling. This indicates that MIA attacks are not sufficient to understand susceptibility to relearning attacks.
in this scenario. These results indicate that relearning more effective when using examples that were seen during training, rather than held-out set of examples.
These findings are also related the other recent findings on anticipatory knowledge recovery when exposing the model to a repeated sequence of documents. In this scenario, as the model processes documents fixed order, unexpectedly begins to recover an increasing amount of information about a previously seen example even before encountering that example again 43
H
Membership--nferenee Attacks (MIA) on Unlearned Models
Membership-Inferenee Attacks (MIA) attempt to evaluate it is possible identify instances that were part of the model's training distribution from held-out instances (77 Being able to distinguish instances leaks private information regarding an instance's membership. We follow the setup from (26 for membership inference ienferensippifffrnncee attacks with balanced bos--theeshold-basee classifier. The results are presented in Table 2 We observe that, is the case in our initial findings with the forget set accuracy metric, this particular type of MIA can give a false sense that unlearning successfully erased all traces unlearned data, even in cases where we know that the unlearned model is prone to relearning attacks. Note that we evaluated particularly simple variant of MIA based on prior work (26). Therefore, is possible to use more complex variants of MIA, which might be better able to predict this vulnerability against relearning.
I Understanding the Efficiency of Unlearning Algorithms
order control for efficiency effects of the unlearning algorithm, all previous results were reported at a fixed unlearning budget of 100 epochs, which is comparable to the budget allocated for pretraining. This raises concerns regarding the utility of approximate unlearning algorithms, as one should use the gold-standard unlearning algorithm, i.c., retrain from scratch in cases where the computational budget is directly comparable to the pretraining budget.
In order to understand the significance of this selection, we vary the number of unlearning epochs and report test accuracy and forget set accuracy after relearning, while using the same set of hyperpa- rameters. The results are presented in Table 3 It is clear from the table that while both catastrophic forgetting and weight distortion are successful resisting relearning at the maximum budget of 100 epochs, catastrophic forgetting is still susceptible to relearning with smaller number of unlearning epochs. On the other hand, weight distortion is more resistant to relearning even after a single epoch of unlearning.
16

---

Model
Method
Unlearning Epochs
Test Acc (%)
Forget Set Acc (%)
Unlearned
Catastrophic Forgetting
93.79%
100.00%
Unlearned
Catastrophic Forgetting
3
93,429
100.00%
Unlearned
Catastrophic Forgetting
10
92.52%
94.50%
Unlearned
Catastrophic Forgetting
30
91.68%
77.75%
Unlearned
Catastrophic Forgetting
50
90.25%
60.00%
Unlearned
Catastrophic Forgetting
100
88.23%
41.75%
Unlearned
Weight Distortion
1
84.49%
43.50%
Unlearned
Weight Distortion
3
86.21% %
49.00%
Unlearned
Weight Distortion
86.27% 10
52.75%
Unlearned
Weight Distortion
30
86.29%
53.25%
Unlearned
Weight Distortion
50
86.31%
50.50%
Unlearned
Weight Distortion
100
85.97%
45.75%
Relearn
Catastrophic Forgetting
93.71%
99.99 .9.9%%
0 Relearn
Catastrophic Forgetting Catastrophic Forgetting
93.65% 3
99, 99.99%
Relearn
93.37% 10
99.28%
0 Relearn
Catastrophic Forgetting
30
93.22% 50
93.63%
Relearn
Catastrophic Forgetting
93.08%
82. 16%
0 Relearn
Catastrophic Forgetting
100
92.80% 1
66.14%
0 Relearn
Weight Distortion
90.14%
61.07%
0 Relearn
Weight Distortion
90.14% 3
61.66%
0 Relearn
Weight Distortion
10
89.86%
61.38%
0 Relearn
Weight Distortion
30
89.62%
60.69%
0 Relearn
Weight Distortion
50
89.44%
59.64%
0 Relearn
Weight Distortion
100
89.64%
58.59%
Table 3: Test and forget-set accuracy of the unlearned model after relearning attack on CIFAR-10 with atypical forget set examples. is evident from the results that even small number of unlearning epochs are sufficient to achieve robustness against relearning attack with weight distortion. In contrast, other methods such as catastrophic forgetring only work the largest epoch setting. Note that all previous results were reported with 100 unlearning epochs, masking any impact of the efficiency of the unlearning algorithm.
J Computational Resources
We used NVIDIA RTX 3090 for each of our experiments, with the GPU equipped with 24GB of high-bandwidth memory (HBM) - we only use tiny fraction of it as we train small ResNet models on CIFAR-10/100.
The pretraining takes about 3 hours. Each unlearning method takes about 2.5 hours, except TAR which takes about 9 hours. Each setting required training 21 unlcarned models (3 x 3 combinations for two-phase training as we evaluate three methods with three different initial safeguards). This equates to about 18 x 2.5 + 9 x 3 = 72 hours per setting. We evaluated 6 main settings: ResNet- on CIFAR-10 (typical, atypical, and random subset for the forget set), ResNet-34 on CIFAR-I 10 (atypical), ResNet-18 18 on CIFAR-10 with Mlass-agnostic unlearning (atypical), and ResNet- 88 on CIFAR-100 (atypical). Hence, this equates about 450 hours. Grid evaluation of each model further takes one hour, resulting in additional 130 hours (21 6 models).
Finally, taking into account all further evaluations well failed experiments, we expect have invested about 1000 GPU hours on the project.
K Societal Impact
We highlight limitations of existing unlearning techniques in the context of visual recognition, showing that such techniques are prone to relearning attacks, where unlearned knowledge can be casily recovered with access to only retained knowledge. Deployment of such methods therefore
17

---

poses risks in the context of privacy or model safety. We further propose a new class of methods that are more robust against such attacks.
Developing better tamper-resistance to relearning attacks for unlearned models attempts to reduce the potential harm from these existing systems.
18

---

Retrain from scratch
L1 Sparse
Weight Dropout
Random Relabeling
Catastrophic Forgetting
Weight Distortion CBFT
SCRUB
SSD
Neg Grad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget set accuracy
unlearned model
relearning examples
10 releaming examples
100 relearning examples
1.00
0.50
0.00
0.40
0.60
0.80
0.40
0.60
0.80
0.40
0.60
0.80
0.40 0.60 0.80
Test set accuracy
Retrain from scratch Random Relabeling
L1 Sparse
Weight Dropout
Catastrophic Forgetting
Weight Distortion
SCRUB
SSD
CBFT
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget set accuracy
relearning examples
10 relearning examples
100 relearning examples
1
100
101
102
100
101
102
100
101
102
Number of training steps (log-scale)
Retrain from scratch
TAR
CBFT
Weight Dist Reg
SCRUB
Circuit Breakers
Weight Distortion
Forget set accuracy
1.00
0.50
0.00
0.85
0.90
0.85
0.90
0.85
0.90
Test set accuracy
Figure 7: Comparison between test set accuracy and accuracy on the held-out part of the forget set Dr.., CIFAR-10 and ResNet-18, where the forget set is comprised of typical examples from the 'airplane' class. The figure indicates that all methods achieve perfect recovery of unlearned knowledge, which is consistent with the retrain from scratch baseline, the model can directly generalize to these examples without having them as part of the training set by definition.
19

---

Retrain from scratch
L1 Sparse
Weight Dropout Weight Distortion CBFT
Random Relabeling
Catastrophic Forgetting SSD
SCRUB
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget set accuracy
1.00
unlearned model
releaming examples
relearning examples
100 relearning examples
0.50
0.00
0.80
0.90
0.80
0.90
0.80
0.90
0.80
0.90
Test set accuracy
Retrain from scratch Random Relabeling
L1 Sparse
Weight Dropout Weight Distortion CBFT
Catastrophic Forgetting
SCRUB
SSD
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget set accuracy
relearning examples
10 relearning examples
100 relearning examples
1
100
10
102
100
10
102
100
10
102
Number of training steps (log-scale)
Retrain from scratch
TAR
CBFT
Weight Dist Reg
SCRUB
Circuit Breakers
Weight Distortion
Forget set accuracy
1.00
0.50
0.00
0.85
0.90
0.85
0.90
0.85
0.90
Test set accuracy
Figure 8: Comparison between test set accuracy and accuracy on the held-out part of the forget set DF. on CIFAR-10 and ResNet-18, where the forget set comprised of random subset of the examples from the 'airplane" class. The figure indicates that retrain from scratch baseline can already achieve near near-pertee near-perfect accuracy, the examples class are predominantly comprised of typical examples.
20

---

Retrain from scratch
Sparse
Weight Dropout
Random Relabeling
Catastrophic Forgetting
Weight Distortion
SCRUB
SSD
CBFT
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
unleamed model
relearning examples
10 relearning examples
Forget set accuracy
1.00
0.50
0.00
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
Test set accuracy
Retrain from scratch
Sparse
Weight Dropout
Random Relabeling
Catastrophic Forgetting
Weight Distortion
SCRUB
SSD
CBFT
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget set accuracy
0 relearning examples
10 relearing examples
0
100
10
102
100
10
102
Number of training steps (log-scale)
Retrain from scratch
TAR
CBFT
Weight Dist Reg
SCRUB
Circuit Breakers
Weight Distortion
Forget set accuracy
1.00
0.50
0.00
0.50
0.75
0.50
0.75
0.50
0.75
Test set accuracy
Figure 9: Comparison between test set accuracy and accuracy on the held-out part of the forget set Drno on CIFAR-100 and ResNet-18, where the forget is comprised of atypical examples from the "apple" class. We see significant variation among different methods on the more complex CIFAR-100 dataset, which comparable the results we observed with class-agnostic unlearning on CIFAR-10 in Fig. 2 (bottom).
21

---

Retrain from scratch
L1 Sparse
Weight Dropout Weight Distortion
Random Relabeling
Catastrophic Forgetting
SCRUB
SSD
CBFT
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget accuracy
unlearned model
0 releaming examples
10 relearning examples
100 relearning examp es
1.00
0.50
0.00
0.85
0.90
0.85
0.90
0.85
0.90
0.85
0.90
Test set accuracy
Retrain from scratch Random Relabeling
L1 Sparse
Weight Dropout Weight Distortion
Catastrophic Forgetting
SCRUB
SSD
CBFT
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget set accuracy
relearning examples
10 relearning examples
100 relearning examples
100
10
102
100
10
102
100
101
102
Number of training steps (log-scale)
Retrain from scratch
TAR
CBFT
Weight Dist Reg
Forget set accuracy
SCRUB
Circuit Breakers
Weight Distortion
1.00
0.50
0.00
0.88
0.90
0.93
0.88
0.90 0.93
0.88
0.90
0.93
Test set accuracy
Figure 10: Comparison between test accuracy and accuracy on the held-out part of the forget set Dr. on CIFAR-10 and ResNet-34, where the forget set is comprised of atypical examples from the 'airplane' class. We still see the distinction between robust and non-robust methods. However, the relative ranking between different methods changes as we use the same set of hyperparameters, which should be tuned considering larger number of parameters in the model. We find that using large model, i.c., ResNet-34 34 instead of ResNet-18 induces slight shift in relative method ranking, particularly for methods that require careful tuning of hyperparameters.
22

---

Retrain from scratch
L1 Sparse
Weight Dropout
Random Relabeling
Catastrophic Forgetting
Weight Distortion
SCRUB
SSD
CBFT
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget set accuracy
relearning examples
10 relearning examples
100 relearning examples
100
101
102
100
10
102
100
101
102
Number of training steps (log-scale)
Figure 11: Comparison between relearning time and accuracy the held-out part of the forget set Dr. on CIFAR-IO and ResNet-18, where the forget set is comprised atypical examples from the 'airplane' class. The figure indicates that many methods achieve near-perfect recovery of unlearned knowledge with only a small amount of model fine-tuning, without even assuming access to the unlearned examples. This figure is extension of the results presented in Fig. (top).
23

---

Retrain from scratch
L1 Sparse
Weight Dropout
Random Relabeling
Catastrophic Forgetting
Weight Distortion
SCRUB
SSD
CBFT
NegGrad+
Weight Attenuation
Weight Dist Reg
Circuit Breakers
Forget set accuracy
relearning examples
10 releaming examples
100 relearning examples
100
10
102
100
10
102
100
10
102
Number of training steps (log-scale)
Retrain from scratch
TAR
CBFT
Weight Dist Reg
SCRUB
Circuit Breakers
Weight Distortion
Forget set accuracy
1.00
0.50
0.00
0.85
0.90
0.85
0.90
0.85
0.90
Test set accuracy
Figure 12: Comparison between test set accuracy and accuracy on the held-out part of the forget set Drn. on CIFAR-10 and R8sNet-18, where the forget set comprised of atypical examples from all classes. This figure is an extension of the results presented in Fig. (bottom). We find that atypical examples, when selected from all the classes, are significantly harder in comparison to examples selected from particular class, and hence, achieve better separation between different methods in comparison sub-class unlearning in Fig. (top).
24

---

[Dete] SCRUB
[DA] SCRUB
[Dre]] SCRUB
relearning examples
10 relearning examples
100 relearning examples
Forget set accuracy
1.00
0.50
0.00
0.95
1.00
0.95
1.00
0.95
1.00
Test set accuracy
[Dam]] SCRUB
[DR] SCRUB
[Da] SCRUB
Forget set accuracy
relearning examples
10 relearning examples
100 relearning examples
100
10
102
100
10
102
100
101
102
Number of training steps (log-scale)
Figure 13: Comparison between training time and accuracy on the held-out part of the forget set Drao on CIFAR-10 10 and ResNet-18, 18, where the forget set is comprised of atypical examples, and using SCRUB as the unlearning method. We evaluate the use of different including the retain set DR, test set Die. corrupted test set Date taken from CIFAR-10-C [18] (using JPEG compression with severity level of 5), along with the selected number of relearning examples. The figure indicates that there can be significant differences in relearned model accuracy based on the selected examples used for reminding the model.
to remind the model,
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We highlight that the aim the paper is to evaluate and propose new methods that resistant against tampering attacks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We highlight the limitations including the trade-off between clean accuracy and the tamper-resistance Section 7, We also highlight the limitation of one instantiation of our framework (Mode Connectivity) in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [NA]
Justification: The paper is based on empirical evidence, as we primarily evaluate the limitations of existing approximate unlearning methods.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We list all the main hyperparameters and settings in the paper, which are sufficient to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will provide code to reproduce our main results along with a README file. We use standard datasets i.c., CIFAR-10 and CIFAR-100 which should be directly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips-. cc/
mublic//uiiee/CodeSuutiii oot.oooeeaaaaissssoooooooooooooo oiliccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//abps.cc/pullic/uuiiee
swssssssiioyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We list all these minor details, including method-specife hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Since we are testing vulnerability, which is quite stable, we only did a single run. While extending it to multiple runs is desirable, this would incur significant additional cost without any major insight as the differences are already high.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We list details about the hardware used for experiments and experimental runtime in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httppp////.aaree...ccppaa///aaaa
Answer: [Yes]
Justification:
The conducted research does comply with NeurIPS code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We highlight the societal impact of our work in Appendix K
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't release data or models that pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use publicly available datasets and model architectures.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No crowdsoureing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM was not used for the core method development.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neuree http///rrppscccoooeeeeereeee//// Cc/Connferences/2225L/II. for what should or should be described.
32


Reviews:
[{"review_id": "vH4Uxqu9nc", "reviewer": "Reviewer_JnT6", "summary": "The paper studies the relearning attack phenomenon in example-level image classifier unlearning, where fine-tuning an unlearned model causes previously forgotten knowledge to re-emerge, even when the fine-tuning data contains no examples from the forget set. The authors analyze this phenomenon through the lens of L2 distances and linear mode connectivity in weight space, and find that models farther from the original pretrained model tend to be more robust to such attacks. Building on this finding, the paper proposes novel unlearning methods that involve either randomly distorting the model weights or directly maximizing the weight-wise distance from the original model. These methods are shown empirically to better resist relearning attacks.", "questions": "- [Q1] I agree with the authors that distinguishing every single method in Figures 1 and 2 may not be necessary, but I still believe each method should be distinguishable for the sake of full readability. That being said, could those figures be revised to clarify the different methods? One suggestion would be to use colors together with different marker shapes for existing vs. new methods?", "limitations": "The authors have adequately addressed limitations of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- [S1] **Interesting direction.**  The finding that fine-tuning an unlearned model on seemingly benign data can revive forgotten knowledge is an important concern in unlearning, given how easy such attacks are to perform. Developing methods that are resistant to this type of attack is of growing interest to the unlearning research community.\n- [S2]  **Strong analyses.** The empirical studies involving L2 distance and linear mode connectivity clearly support the paper’s core claims, and could become practical tools for evaluating post-unlearning models in the future.\n\n## Weaknesses\n- [W1] **Narrow novelty and scope.** As mentioned in Lines 38-40, the vulnerability of unlearned models to relearning attacks has already been studied in the context of LLM unlearning. While the authors justify a focus on image classification by mentioning the difficulty of \"drawing a clear line between forbidden and permissible knowledge\" (Lines 45-46), this justification is not very compelling. In fact, existing LLM unlearning benchmarks do provide well-defined forget/retain splits and interpretable performance metrics [A, B, C]. Given that unlearning is arguably more pressing for LLMs and diffusion models, due to their generative nature and the high cost of pretraining, the contribution here feels somewhat narrow. Prior work has already shown that unlearned knowledge in LLMs can re-emerge under quantization, and that robustness can be improved by increasing weight distance [D].\n- [W2] **Evaluation limited to classification accuracy.** Another major limitation is that all evaluations are based on classification accuracy. The study does not consider Membership Inference Attack (MIA) accuracy, which is a standard and widely accepted metric in the image classifier unlearning literature [E, F]. This makes it difficult to fully assess how well the proposed methods achieve the goal of unlearning.\n- [W3] **Confusing structure.** The structure of the paper, particularly from the experimental section onward, is difficult to follow and would benefit from a major revision. The proposed methods (Weight Distortion, Weight Distance Regularization, and CBFT) are only detailed near the end, even though their results are discussed in earlier sections. Additionally, significant results currently buried in the conclusion should be moved up to the relevant sections. A more effective structure might present the preliminary and analytical findings first, then introduce the proposed methods, followed by the full experimental results. Section 4.1, which over-explains the baselines, could be substantially shortened to make room for a clearer explanation of the authors’ own methods.\n\n[A] TOFU: A Task of Fictitious Unlearning for LLMs. COLM 2024.\\\n[B] The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning. ICML 2024.\\\n[C] RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS 2024.\\\n[D] Catastrophic Failure of LLM Unlearning via Quantization. ICLR 2025.\\\n[E] Towards Unbounded Machine Unlearning. NeurIPS 2023.\\\n[F] SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. ICLR 2024.", "was_revised": true, "final_justification": "Thank you authors for sharing further experimental results and insights. The results convincingly resolve most of my prior concerns. Though I am still a bit concerned of the paper's applicability especially since unlearning is much more crucial with larger models trained on larger datasets, I feel the paper's current positioning makes sense and would be of good interest to unlearning researchers. Hence I am leaning more towards an accept and raise my score.", "rebuttal": "We are very thankful to the reviewer for their valuable and detailed comments. We address the concerns raised by the reviewer below.\n\n## Response to W1 (novelty and scope)\n\nAlthough recovery of forgotten knowledge has been demonstrated in the LLM unlearning literature, we believe  our work is the first to show, under controlled conditions, that fine-tuning on only the retain set can undo unlearning. Furthermore, the strength of this effect is quite striking, yielding almost complete recovery of the forget set in some cases (Figure 1).  We’ve cited all relevant works we could find, including many works in the LLM setting, but we still consider our analysis to be novel and complementary to those interesting works.\n\nSpecifically, LLMs are not ideal for experimentation for two reasons: (i) computational cost, and (ii) it’s difficult to isolate factors at play, which limits conclusions about the causes and remedies of vulnerability to relearning. To elaborate:\n\n- The goal of unlearning in LLMs is vaguely defined since the model defines a joint probability over the entire sequence (materialized via conditional probabilities of the next token conditioned on all preceding tokens). This means paraphrasing of text complicates evaluation since we would like to minimize the joint probability of any arbitrary paraphrasing of the text to be forgotten. In the case of classification, the goal is readily quantified.\n- Powerful LLMs can in many cases acquire knowledge “anew” rapidly, even just at in-context learning time, by synthesizing existing knowledge in novel ways. This makes it harder to separate knowledge acquired by relearning from knowledge acquired/synthesized “on the fly”.\n- The pretraining → finetuning methodology in LLMs further complicates matters. Having pretrained on a vast corpus of text leads to ambiguity about what knowledge was acquired at pretraining time versus at finetuning time, and when the knowledge was acquired can make a difference for unlearning/relearning dynamics, complicating the analysis.\n\nTo eliminate the above complicating factors, we study unlearning (and relearning) of a specific set of examples in a model that is trained (from random weights) on a dataset including those examples. In our setup (small models, no prior pretraining phase), we are also able to compute and compare against the oracle unlearning method of retraining from scratch without the forget set, and study relearning in that model as a reference point for how much information can be acquired “anew” about the forget set in a model that was truly never trained on that forget set before.\n\nOur analysis in these controlled conditions is the first to show that, beyond doubt, relearning attacks succeeding are a failure of unlearning. Our analysis also led us to clearly explain this phenomenon from a weight space perspective and propose new methods to address it, both of which are important contributions that also showcase how fruitful it can be to conduct experiments in our setting.\n\nWe are grateful to the reviewer for highlighting reference D which we weren’t aware of at the time of submission. We found that our vision models aren’t vulnerable to these quantization attacks: **forget set accuracy stays at the same level as the full-precision unlearned model regardless of the number of bits until it gets close to 8 bits at which point the model collapses**. Thus, the spontaneous-recovery phenomenon we explore is distinct from the recovery obtained via quantization. Below are some sub-sampled quantitative results due to space constraints:\n\n| Unlearning Method         | Bits | Test Acc (%) | Forget Set Acc (%) |\n|--------------------------|------|--------------|--------------------|\n| Catastrophic Forgetting  | 2    | 10.00        | 0.00               |\n| Catastrophic Forgetting  | 4    | 10.00        | 0.00               |\n| Catastrophic Forgetting  | 8    | 13.19        | 0.00               |\n| Catastrophic Forgetting  | 16   | 88.23        | 39.60              |\n| Catastrophic Forgetting  | Full   | 88.23        | 42.60              |\n| Weight Dist Reg          | 2    | 10.00        | 0.00               |\n| Weight Dist Reg          | 4    | 10.06        | 0.00               |\n| Weight Dist Reg          | 8    | 78.75        | 27.40              |\n| Weight Dist Reg          | 16   | 82.25        | 31.20              |\n| Weight Dist Reg          | Full   | 82.26        | 32.20              |\n\n\nWe have two hypotheses for why they are not effective in our setup: (i) Language models consist of billions of parameters (compared to a few million in our case). Hence, the deltas are really small. This results in easily knocking off the weights to their previous pretrained values, which isn’t the case in our simple vision models. (ii) This problem is further exacerbated with models that are trained with quantization in mind. Such models can immediately latch onto the previous values which were regularized to be easily quantizable.\n\nFurthermore, D showed that small weight distances (between the pretrained and unlearned model) are the cause of quantization attacks. We instead show this for relearning attacks for the first time. Given that not all models suffer from quantization attacks (an interesting finding in and of itself), it is important to understand how weight space distances affect each type of attack and this was not previously understood. Finally, the approach taken in D for increasing the distance between the unlearned and pretrained model is to increase the learning rate (in existing unlearning objectives/algorithms). While this is a very valuable first step in the exploration, the authors themselves admit that their framework is “highly sensitive to hyperparameter selection, leading to an unstable unlearned model”. Instead, we propose unlearning objectives that are designed to handle this natively, and we study the trade-offs of these new unlearning algorithms compared to the old ones.\n\nBased on the above, we hope the reviewer agrees that our work is complementary to D. We added these new results and discussion of D to our paper. \n\n\nOverall: **our study of relearning in vision classifiers complements the existing findings in the LLM unlearning literature** and **is the first study to** (i) **show beyond doubt that the relearning phenomenon is due to imperfect unlearning** (rather than due to other factors that remained entangled in LLM studies); (ii) **show that relearning is possible without access to any examples from the forget set**; (iii) **show that vulnerability to relearning is strongly correlated with weight space distances** between the pretrained and unlearned models, complementing D that shows this for quantization but not for relearning attacks (this is important since, as discussed, not all models that suffer from relearning attacks also suffer from quantization attacks); (iv) **derive more robust algorithmic principles**, going above and beyond prior work in D that tried to increase weight space distance by increasing learning rates, leading to an unstable system that is highly sensitive to hyperparameter selection.\n\nWe consider it important future work to translate these advancements to other modalities, architectures and different sizes of models and datasets. We have **added results on a higher-resolution ImageNette dataset** which further validates the generality of our findings; please see response to **reviewer JQcR**.\n\nWe hope that we have convinced the reviewer about the novelty and significance of our work, but we would be very happy to discuss this more during the upcoming reviewer-author discussion phase.\n\n\n## Response to W2 (MIA attacks)\n\nWe are thankful to the reviewer for the useful suggestion regarding MIA attacks. Based on the reviewer’s feedback, we present the results for MIA below. We chose the MIA from Kurmanji et al. (reference E) according to the reviewer’s suggestion. We observe that, as is the case in our initial findings with the forget set accuracy metric, this MIA can “fool us” into thinking that unlearning successfully erased all traces of unlearned data, even in cases where we know that the unlearned model is prone to relearning attacks. We will add these results in the appendix.\n\n| Unlearning Method       | MIA acc |\n|-------------------------|---------|\n| l1_sparse               | 54.2    |\n| weight_dropout          | 54.8    |\n| weight_dist_reg         | 48.4    |\n| ssd                     | 53.8    |\n| weight_attenuation      | 53.6    |\n| weight_distortion       | 53.2    |\n| alt_scrub               | 51.6    |\n| circuit_breakers        | 55.0    |\n| alt_gradient_ascent     | 54.2    |\n| random_relabeling       | 94.8    |\n| catastrophic_forgetting | 54.2    |\n| retrain from scratch    | 52.2    |\n|-------------------------|---------|\n\n\n> Confusing structure\n\nThanks for raising this. We believe that this structure offers greater compactness compared to presenting “preliminary results” first without our proposed methods, and then repeating those plots later with the addition of more methods. We would be happy to hear any suggestions from the reviewer regarding how this can be more effectively communicated without replication.\n\n\n> Q: use colors together with different marker shapes for existing vs. new methods?\n\nThank you for the useful suggestion. We will add all the results in a tabular form in the appendix for curious readers who are interested in looking at precise values. Please let us know if this would be helpful.\n\n—-\n\nOverall, we really appreciate the points raised by the reviewer as we feel these are important discussions to have, and reflecting these in the paper has strengthened our contribution. We also feel that the addition of the quantization and MIA results is very valuable. We are looking forward to hearing any further thoughts from the reviewer and discussing more."}, {"review_id": "EFY7kuxa5Z", "reviewer": "Reviewer_JQcR", "summary": "This paper proposes a defense method against the relearning/tampering attack in machine unlearning. The authors find that keeping the model farther away from its original weights makes unlearning more robust. They introduce a weight-space regularization technique that pushes the model away in weight space, improving resistance to relearning on vision benchmarks.", "questions": "1. The paper reports that the proposed unlearning method takes 2.5 hours, while training from scratch takes about 3 hours. Given that training from scratch guarantees complete forgetting, could the authors clarify why the proposed method would be preferable in practice when its computational cost is comparable?\n2. Do the authors have any insights or preliminary results on how well this method would generalize to other domains?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n1. The motivation and methodology are written clearly and are easy to follow.  \n2. The proposed weight-space regularization method is intuitive and empirically effective.\n\n## Weaknesses\n\n1. The method is evaluated only on vision datasets (CIFAR datasets and ResNet architectures); it remains unclear how well it generalizes to other domains or larger models.\n2. The unlearning problem can be unequivocally solved by training from scratch, so the computational overhead introduced by the regularizer is especially important. If the cost of unlearning exceeds that of training from scratch, the proposed approach may lose practical value.", "was_revised": true, "final_justification": "Based on the authors' response during the rebuttal phase, I have revised my assessment of the paper's significance, raising my score from 2 to 3. I now believe this paper will bring moderate impact to the machine unlearning sub-area of AI research, addressing meaningful challenges in this emerging field.", "rebuttal": "We are very thankful to the reviewer for their valuable and encouraging comments. We address the concerns raised by the reviewer below.\n\n> The method is evaluated only on vision datasets (CIFAR datasets and ResNet architectures)\n\nThat’s a great remark. We indeed leveraged simple vision datasets with smaller-scale architectures for our exploration. This was by design, to allow us to answer some fundamental questions about relearning attacks: why do they occur and how can we address them. This is a different direction that complements work on relearning attacks that were carried out in LLMs.\n\nHowever, we agree with the reviewer that presenting results in an additional larger dataset would lead to more convincingly showcasing the generality of our findings. Based on this suggestion, we present additional results here on the well-known Imagenette dataset [1] (a subset of ImageNet). Similar to other results in the main paper, we focused on using atypical examples for the forget set. The results for three different unlearning methods are presented below (note that tuning hyperparameters for all baselines is difficult in such a tight timeline, hence, we refrain from reporting sub-optimal numbers for other baselines – we will add all results in the next revision of the paper).\n\n| Model | Label | Test Acc (%) | Forget Set Acc (%) |\n|---|---|---|---|\n| unlearned | Retrain from scratch | 87.34 | 67.53 |\n| unlearned | SCRUB | 85.81 | 85.71 |\n| unlearned | Weight Distortion | 83.16 | 74.03 |\n| unlearned | Weight Dist Reg | 83.72 | 84.42 |\n|---|---|---|---|\n| 0 relearn | Retrain from scratch | 86.40 | 63.53 |\n| 0 relearn | SCRUB | 86.47 | 93.40 |\n| 0 relearn | Weight Distortion | 84.71 | 78.86 |\n| 0 relearn | Weight Dist Reg | 84.31 | 73.87 |\n|---|---|---|---|\n| 10 relearn | Retrain from scratch | 86.73 | 64.16 |\n| 10 relearn | SCRUB | 86.65 | 94.21 |\n| 10 relearn | Weight Distortion | 84.62 | 81.09 |\n| 10 relearn | Weight Dist Reg | 84.83 | 75.77 |\n\nWe see that the forget set accuracy of SCRUB, similar to our previous results, shoots up upon relearning, even with 0 relearning examples, in line with our findings on CIFAR with an increasing number of relearning examples. On the other hand, variants of our tamper-resistant framework, weight distortion, and weight dist reg, similar to our previous results, are more resistant against these relearning attacks.\n\n\n[1] https://github.com/fastai/imagenette\n\n\n> The unlearning problem can be unequivocally solved by training from scratch, so the computational overhead introduced by the regularizer is especially important. If the cost of unlearning exceeds that of training from scratch, the proposed approach may lose practical value.\n\nThank you for raising this. We would like to highlight that our regularizer isn’t more expensive than any other methods commonly employed for approximate unlearning (that commonly rely on two different forward passes through the network), while being significantly more efficient at the unlearning task itself: see our response below where we can unlearn effectively with just a single epoch compared to catastrophic forgetting which significantly benefits from a larger number of unlearning epochs.\n\n> The paper reports that the proposed unlearning method takes 2.5 hours, while training from scratch takes about 3 hours. Given that training from scratch guarantees complete forgetting, could the authors clarify why the proposed method would be preferable in practice when its computational cost is comparable?\n\nThat’s a great question, and an important observation. We gave all methods a very large training budget in order to ensure that they have the best chance of succeeding at unlearning. However, note that not all methods are equally effective at the task. In order to make this concrete, we present the unlearning/relearning performance of the model with a different number of unlearning epochs. We see that forget set accuracy goes down with an increasing number of unlearning epochs as it only relies on weight decay for unlearning. On the other hand, for weight distortion, we see that the forget set accuracy starts quite low even after a single epoch, and doesn’t benefit much from even a 100 unlearning epochs (which is the consistent budget we used in all our previous experiments for comparison). We also highlight the accuracy after the relearning attack (using 0 relearning examples; larger number of relearning examples from the table were removed as the results were very similar) in order to establish that the unlearning in this case was truly successful. Therefore, this highlights that our methods are particularly efficient at unlearning. The large unlearning budget we chose (which is infeasible in practice due to the cost approaching that of retraining from scratch as the reviewer correctly pointed out) ensures that methods that are slow in unlearning are not penalized. Analyzing the efficiency of unlearning is an interesting question in its own right, and worth investigating in more detail separately.\n\n| Model | Method | Unlearning Epochs | Test Acc (%) | Forget Set Acc (%) |\n|---|---|---|---|---|\n| unlearned     | catastrophic_forgetting |   1  | 93.79    | 100.00         |\n| unlearned     | catastrophic_forgetting |   3  | 93.42    | 100.00         |\n| unlearned     | catastrophic_forgetting |  10  | 92.52    | 94.50          |\n| unlearned     | catastrophic_forgetting |  30  | 91.68    | 77.75          |\n| unlearned     | catastrophic_forgetting |  50  | 90.25    | 60.00          |\n| unlearned     | catastrophic_forgetting | 100  | 88.23    | 41.75          |\n| unlearned     | weight_distortion       |   1  | 84.49    | 43.50          |\n| unlearned     | weight_distortion       |   3  | 86.21    | 49.00          |\n| unlearned     | weight_distortion       |  10  | 86.27    | 52.75          |\n| unlearned     | weight_distortion       |  30  | 86.29    | 53.25          |\n| unlearned     | weight_distortion       |  50  | 86.31    | 50.50          |\n| unlearned     | weight_distortion       | 100  | 85.97    | 45.75          |\n| 0 relearn     | catastrophic_forgetting |   1  | 93.71    | 99.99          |\n| 0 relearn     | catastrophic_forgetting |   3  | 93.65    | 99.99          |\n| 0 relearn     | catastrophic_forgetting |  10  | 93.37    | 99.28          |\n| 0 relearn     | catastrophic_forgetting |  30  | 93.22    | 93.63          |\n| 0 relearn     | catastrophic_forgetting |  50  | 93.08    | 82.16          |\n| 0 relearn     | catastrophic_forgetting | 100  | 92.80    | 66.14          |\n| 0 relearn     | weight_distortion       |   1  | 90.14    | 61.07          |\n| 0 relearn     | weight_distortion       |   3  | 90.14    | 61.66          |\n| 0 relearn     | weight_distortion       |  10  | 89.86    | 61.38          |\n| 0 relearn     | weight_distortion       |  30  | 89.62    | 60.69          |\n| 0 relearn     | weight_distortion       |  50  | 89.44    | 59.64          |\n| 0 relearn     | weight_distortion       | 100  | 89.64    | 58.59          |\n\n\n> Do the authors have any insights or preliminary results on how well this method would generalize to other domains?\n\nSusceptibility to relearning has already been analyzed for language models as we list extensively in the paper. The fact that relearning is observed across these different models is a clear sign that it is a more general phenomenon, beyond a single modality/domain.\n\nThe focus of our paper is to design a controlled study to gain a deeper understanding of causes and remedies for vulnerability to relearning. Based on this, we have identified weight space distances as an important factor affecting the robustness of different unlearning algorithms and proposed different unlearning objectives to directly address this.\n\nWe consider it exciting future work to experiment with our unlearning objectives in LLMs too for increasing their robustness. But there is no doubt, based on the existing literature, that this phenomenon does also occur in LLMs, making this an important and impactful direction.\n\n\n—\n\nOverall, we believe that we have addressed all comments and questions brought up by the reviewer and we’re curious if any other concerns remain. If the reviewer is satisfied, we would be really grateful if the reviewer can show stronger support for acceptance by increasing their score."}, {"review_id": "UUPceRBGIy", "reviewer": "Reviewer_qmRS", "summary": "This paper argues that existing studies on relearning attacks have been limited to large language models (LLMs), and that a more precise analysis requires investigation in well-defined settings such as vision classification models. In response, the paper presents the first analysis of relearning attacks targeting vision classification models that have undergone unlearning. Specifically, it shows that when a model is unlearned using existing methods on atypical instances, its performance on the forget set can be substantially recovered by fine-tuning the model using only the retain set. The authors analyze the cause of this phenomenon from a weight-space perspective and, based on this insight, propose new methods to enhance the tamper-resistance.", "questions": "While reading this paper, I had the following questions, which are closely related to the weaknesses already mentioned:\n\n\n1) What exactly is the criterion used to define atypical instances? If this criterion were changed (e.g., by modifying the threshold that separates typical and atypical samples), would the analysis and conclusions presented in the paper still hold?\n\n\n2) What is the exact reason behind the drop in test accuracy observed with Weight Dist Reg? Additionally, is there a potential way to improve tamper-resistance without sacrificing test accuracy?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n\n1) This paper is the first to formally define the problem of relearning attacks in the context of vision classification tasks, and the motivation for addressing relearning attacks in this setting is generally clear.\n2) The problem definition and experimental setup for evaluating relearning attacks are well-designed, with appropriate methodologies and baselines selected for comparison.\n3) The empirical finding that the performance on the forget set can be recovered solely by fine-tuning the unlearned model on the retain set is particularly intriguing and provides a novel insight into the limitations of existing unlearning methods.\n\nWeaknesses\n\nThis paper is generally strong in its problem formulation and experimental design. Also, it makes a novel and interesting empirical observation, that performance on the forget set, which includes atypical instances, can be suppressed solely by fine-tuning on the retain set. However, there are several important issues that should be addressed:\n\n\n1) **Definition of atypical Instances**: The paper lacks a clear explanation of how typical and atypical instances are distinguished. While it mentions that the score from [1] is used, it does not specify the exact threshold applied to categorize samples discretely. Moreover, it is unclear how many images are considered atypical and what proportion they represent out of the total. Providing such information would help readers better understand the experimental setup.\n\n\n[1] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural\nregularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206,\n2020.\n\n\n2) **Analysis of Weight Dist Reg**: The proposed method, Weight Dist Reg, is reasonably derived from the analysis of weight distances and does show improved tamper-resistance. However, it exhibits noticeably lower test accuracy compared to other methods. This suggests that the model’s weights change substantially, leading to performance degradation on the retain set. The authors should discuss why this trade-off occurs and propose directions to address this issue in future work.\n\n\n3) **Figure Readability and Alignment**: The overall readability of the figures needs improvement. Specific issues include:\n\n\n- In Figure 1, the legend at the top assigns circular markers to all unlearning methods, but the plot legend only labels \"Relearned,\" making the meaning of the markers ambiguous.\n- In Figure 2, it is not clear what differentiates the top and bottom rows. The main text only describes the bottom plot, leaving the top plot unexplained.\n- Figures 2, 3, 4, and 5 appear far from the text where they are discussed. For example, Figure 2 appears on page 5 but is not mentioned until page 7, which disrupts the flow of reading.\n\n\nWhile these limitations do not undermine the main contribution of the paper (i.e., the discovery of the vulnerability in existing unlearning methods) they are important issues that require revision. Taking all of this into consideration, I give this paper a borderline accept recommendation.", "was_revised": true, "final_justification": "The authors provided clear clarifications on the definition of atypical instances, the trade-off in Weight Dist Reg, and figure readability.\nTheir response adequately addressed my concerns, so I am satisfied with the current version and will maintain my borderline accept recommendation.", "rebuttal": "We are very thankful to the reviewer for their valuable and encouraging comments. We address the concerns and answer the questions raised by the reviewer below.\n\n> Definition of atypical Instances\n\nThank you for highlighting this. The details about the selection of typical and atypical examples are important to understand the overall results, we will make sure to add them to the paper comprehensively. \n\nWe used the consistency scores from Jiang et al. (2020) that provides the notion of an example's typicality. We selected instances from the unlearning class with the highest typicality/atypicality scores in the class as our typical/atypical examples for the forget set, while using all the remaining examples as the retain set. Hence, we didn’t need to use a particular threshold, but rather, just sorted them w.r.t. the scores and took top-k/bottom-k examples for typical/atypical examples. We picked the instances with highest/lowest scores from the entire dataset for class-angostic unlearning, and only the target-class for specific-specific unlearning. We will update the manuscript to highlight all these details for curious readers.\n\n\n> Analysis of weight dist reg, drop in test accuracy and ideas for future work to improve it.\n\nThis is a great point, thank you for raising it.\n\nAs we discuss in our discussion section, and as the reviewer pointed out, our results reveal an interesting trade-off between robustness to relearning attacks and test accuracy. It’s natural to expect that methods that add noise to the weights (in attempts to push the unlearned model further away from the pretrained model) destroy useful information in the model that is valuable for generalization. In general, the pretrained model has good generalization capabilities (strong accuracy on the test set). Therefore, moving away from it risks losing those, in favor of increasing robustness to relearning. This is related to a fundamental trade-off in unlearning: removing unwanted information while maintaining permissible knowledge and good model utility (e.g., accuracy on the retain and test sets). We will discuss this further in the updated paper.\n\nMoreover, the reviewer’s comment prompted us to think about actionable paths for addressing this and obtaining a better trade-off between generalization/utility and robustness. We outline some ideas here that future work can pursue.\n\nRather than adding noise to all parameters of the model, we can consider doing this selectively on only parameters that are determined to be disproportionately important for the forget set compared to the retain set (using criticality criteria, such as the one proposed in the SSD unlearning method [1] – such techniques are essential when considering extremely large models such as language models). More broadly, we can try to find a better balance between retaining desired knowledge of the pretrained model while moving away from it more selectively. This can be operationalized by freezing a subset of (cleverly chosen) parameters during the unlearning phase.\n\nWe would love to hear the reviewer’s thoughts on this and keep discussing.\n\n[1] Foster, J., Schoepf, S. and Brintrup, A., 2024, March. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence (Vol. 38, No. 11, pp. 12043-12051).\n\n\n> Figure readability and alignment.\n\nThank you for these suggestions which we have adopted in the paper. Regarding Fig 2, we do actually describe top vs bottom. The caption says: “The forget set is comprised of atypical examples (from the ‘airplane’ class i.e., sub-class unlearning for the top row and all classes i.e., class-agnostic unlearning in the bottom row”. We will however work on improving readability, e.g. by writing “top” and “bottom” in bold to make it easier to spot. If the reviewer has other suggestions, we would be happy to apply them.\n\n\n—\n\nOverall, we believe that we have addressed all comments and questions brought up by the reviewer and we’re curious if any other concerns remain. If the reviewer is satisfied, we would be really grateful if the reviewer can show stronger support for acceptance by increasing their score."}, {"review_id": "YQXlYpFTDg", "reviewer": "Reviewer_22p1", "summary": "The paper presents an analysis of relearning attacks. In these attacks a model has been forced for \"forget\" certain information which it has learned and the goal is to recover the models lost capability. This can be done with no or some subset of the examples which were removed from the model. The paper shows that in some cases, even with none of the forgotten examples, accuracy can be recovered. The subsequent analysis indicates that weight-space regularization, forcing the unlearned model weights to be far from the pre-trained model, can make the model more resistant to relearning attacks. The paper then proposes a method based on weight-space regularization which performs better than prior art.", "questions": "1. Evaluate on more datasets (like ImageNet) or justify why that wouldnt change the conclusion (may require another round of submission)\n2. Are the training iterations always fixed? \n3. How can we extend this to modern models and tasks?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall I like this paper. \n\nThe presented analysis was fascinating: the idea that unlearned example accuracy can be recovered with *only* an orthogonal set of data is surprising and a very nice finding. That information by itself is extremely useful and the new method adds to the contributions of the paper. According to the analysis presented in the paper, this proposed method is effective at solving some of the issues identified with prior art. \n\nFurthermore I see the potential for real-world impact. With models being released frequently by companies these days, I can imagine a scenario where a model was trained on sensitive, illegal, or privacy violating data. The influence of that data could be \"scrubbed\" in a later release of the model using a forgetting algorithm. However, if that information is easily recoverable by an attacker, there could be serious consequences. \n\nThe analysis does have some limitations, however. The biggest issue is with the dataset. CIFAR{10,100} are very simple and may have some unusual properties (see https://arxiv.org/abs/2111.00861 for example). While the method shows analysis on multiple models it does not consider data outside of CIFAR. The analysis *needs* to include a more complex dataset to be convincing. It was also not clear to me how many training iterations were being used for each model, I think it may be important to ensure that the models which were trained from scratch used the same number of total iterations as the models which were fine-tuned on the retain set. Lastly, and this is touched on in the introduction, the models here are very simple and may not be representative of modern LLM behavior, confounding as they are to study. \n\nSo my final thought is that there is *definitely* something good here but it may be too early to accept it as is without further analysis.", "was_revised": true, "final_justification": "After discussing this paper with the authors and reading their rebuttals/the other reviews I am increasing my rating to accept. I think the authors did a good job of clarifying the intent to study relearning in a controlled environment and I still find the analysis that models are able to re-learn with none of the original data to be a significant finding. While I stand by my primary issue: that the results are not directly applicable to real-world models, I am convinced now that this is not a major issue since the analysis itself is still useful. I think this paper could be the starting point of an expansion of the authors definition of the re-learning problem in follow-up work which will eventually lead us to an understanding of relearning in modern multi-modal LLMs. Since the potential future impact is high and the findings are interesting (and there are no methodological flaws that I could find) I think this paper should be published in NeurIPS 2025.", "rebuttal": "We are very thankful to the reviewer for their valuable comments. We are happy that the reviewer found our analysis fascinating, and we are looking forward to discussing more with the reviewer in light of our rebuttal and new experiments. We address the concerns and answer the questions raised by the reviewer below.\n\n> Evaluate on more datasets (like ImageNet) or justify why that wouldn’t change the conclusion (may require another round of submission)\n\nIndeed, we focused on only simple vision datasets with smaller-scale architectures (mainly motivated by prior work which focused on CIFAR-10 and CIFAR-100 as the canonical datasets for unlearning in vision classifiers). Using these smaller-scale datasets allowed us to answer some fundamental questions about relearning attacks: why do they occur and how can we address them. This is a different direction that complements works on relearning attacks that were carried out in LLMs (where the possible exploration is limited due to the sheer cost associated with such experiments, and where the complex setup makes it difficult to draw clear conclusions).\n\nHowever, we fully agree with the reviewer that showing results in only those specific datasets was limiting. Based on this suggestion, we present additional results here on the well-known Imagenette dataset [1] (a subset of ImageNet) to showcase the generality of our findings on more complex and higher-resolution images. Similar to other results in the main paper, we focused on using atypical examples for the forget set. The results for three different unlearning methods are presented below (note that tuning hyperparameters for all baselines is difficult in such a tight timeline, hence, we refrain from reporting sub-optimal numbers for other baselines – we will add all results in the next revision of the paper).\n\n| Model | Label | Test Acc (%) | Forget Set Acc (%) |\n|---|---|---|---|\n| unlearned | Retrain from scratch | 87.34 | 67.53 |\n| unlearned | SCRUB | 85.81 | 85.71 |\n| unlearned | Weight Distortion | 83.16 | 74.03 |\n| unlearned | Weight Dist Reg | 83.72 | 84.42 |\n|---|---|---|---|\n| 0 relearn | Retrain from scratch | 86.40 | 63.53 |\n| 0 relearn | SCRUB | 86.47 | 93.40 |\n| 0 relearn | Weight Distortion | 84.71 | 78.86 |\n| 0 relearn | Weight Dist Reg | 84.31 | 73.87 |\n|---|---|---|---|\n| 10 relearn | Retrain from scratch | 86.73 | 64.16 |\n| 10 relearn | SCRUB | 86.65 | 94.21 |\n| 10 relearn | Weight Distortion | 84.62 | 81.09 |\n| 10 relearn | Weight Dist Reg | 84.83 | 75.77 |\n\nWe see that the forget set accuracy of SCRUB shoots up upon relearning, even with 0 relearning examples, in line with our findings on CIFAR. On the other hand, variants of our tamper-resistant framework, weight distortion, and weight dist reg, similar to our previous results, are more resistant against these relearning attacks. These evaluations highlight that our results are general and hold across different datasets.\n\n\n[1] https://github.com/fastai/imagenette\n\n\n> Are the training iterations always fixed?\n\nThanks for checking this. Yes, we fixed the number of training iterations in order to make sure that discount any effects of (un)learning efficiency for different methods. Hence, we assigned a large pretraining, unlearning, and relearning budget. We show that some methods are more computationally effective, and can **get away with a significantly smaller training budget** in response to **reviewer JQcR**.\n\n\n> How can we extend this to modern models and tasks?\n\nSusceptibility to relearning has already been analyzed for language models as we list extensively in the paper. The fact that relearning is observed across these different models is a clear sign that it is a more general phenomenon, beyond a single model/task.\n\nThe focus of our paper is to design a controlled study to gain a deeper understanding of causes and remedies for vulnerability to relearning. Based on this, we have identified weight space distances as an important factor affecting the robustness of different unlearning algorithms and proposed different unlearning objectives to directly address this.\n\nWe consider it exciting future work to experiment with our unlearning objectives in LLMs too for increasing their robustness. But there is no doubt, based on the existing literature, that this phenomenon does also occur in LLMs. Hence, we believe that these results should directly extend to modern models and tasks.\n\n\n> The \"retrain\" terminology was a little confusing: \"retrain from scratch\" vs. \"fine-tuning\" various models. \n\nRegarding terminology, thanks for raising this potentially confusing issue. We reserve the term “retrain from scratch” to refer to the ideal unlearning method: training from scratch on only the retain set (guaranteeing by construction that the unlearned model is not influenced by the forget set). This is denoted as the black star in our plots and is the reference point for how high or low we expect the forget set accuracy to be under ideal unlearning. This is an important reference point for studying relearning attacks: if the forget set accuracy of even this ideal reference point (black star) shoots up upon relearning attacks, then a higher forget set accuracy upon relearning is not due to imperfect unlearning in the first place. What we observe in our plots though, is that the forget set accuracy of various approximate unlearning algorithms shoots up much higher than the black star upon relearning attacks, allowing us to safely conclude that their susceptibility to relearning is due to not having fully erased the knowledge of the forget set during their unlearning phase. On the other hand, we use the term “finetuning” to refer to further post processing of unlearned (or attempted-to-be-unlearned) models. \n\nWe attempted to lay out the terminology clearly in Section 3 but we agree with the suggestion of ensuring we always refer to retrain-from-scratch as such, rather than shortening to “retrain”, to ensure clarity. Thank you for the suggestion.\n\n\n> In the Fig 2. captions i recommend replacing \"abscissa\" and \"ordinate\" with \"x-\" and \"y-axis\".\n\nThank you for the very useful suggestion. We will update the Fig. 2 caption to make it consistent with all other figures.\n\n\n—-\n\nOverall, we really appreciate the points raised by the reviewer. We also feel that the addition of a new dataset is very valuable to highlight the generality of our results, particularly on high-resolution images, and the clarifications and feedback about writing to improve the presentation of our work. We believe that we have addressed all concerns and questions raised and we are looking forward to hearing back from the reviewer."}]

Author rebuttal:

Meta-review: This paper revisits the idea of the re-learning attack on unlearned models that was proposed in prior work, mainly in the context of LLMs. The authors instead focus on a more controlled experimental setting in image classification. The reviewers acknowledge that even though the experimental setup is far from the real-world use cases of unlearning, the results here shed light on our understanding of unlearning methods. The authors have also actively engaged in the discussion with reviewers and addressed many of the questions and concerns.
Decision: Accept (poster)