What makes unlearning hard and what to do about it
Authors: Kairan Zhao
Meghdad Kurmanji
George-Octavian Bărbulescu
Eleni Triantafillou
Peter Triantafillou
Venue: NeurIPS 2024 poster (2024)
Keywords: machine unlearning
Primary area: other
TL;DR: We present the first investigation into factors that affect the difficulty of an unlearning problem, surfacing previously-unknown behaviours of SOTA methods, and propose a framework that improves unlearning pipelines.

Abstract: Machine unlearning is the problem of removing the effect of a subset of training data (the ``forget set'') from a trained model without damaging the model's utility e.g. to comply with users' requests to delete their data, or remove mislabeled, poisoned or otherwise problematic data.
With unlearning research still being at its infancy, many fundamental open questions exist: 
Are there interpretable characteristics of forget sets that substantially affect the difficulty of the problem? 
How do these characteristics affect different state-of-the-art algorithms?
With this paper, we present the first investigation aiming to answer these questions. We identify two key factors affecting unlearning difficulty and the performance of unlearning algorithms. Evaluation on forget sets that isolate these identified factors reveals previously-unknown behaviours of state-of-the-art algorithms that don't materialize on random forget sets.
Based on our insights, we develop a framework coined Refined-Unlearning Meta-algorithm (RUM) that encompasses: (i) refining the forget set into homogenized subsets, according to different characteristics; and (ii) a meta-algorithm that employs existing algorithms to unlearn each subset and finally delivers a model that has unlearned the overall forget set. 
We find that RUM substantially improves top-performing unlearning algorithms. 
Overall, we view our work as an important step in (i) deepening our scientific understanding of unlearning and (ii) revealing new pathways to improving the state-of-the-art.

What makes unlearning hard and what to do about it
Kairan Zhao' University of Warwick
Meghdad Kurmanji University of Cambridge
George-Octavian Barbulescu
University of Warwick
Eleni Triantafillou' i Google DeepMind
Peter Triantafillou'
University of Warwick
Abstract
Machine unlearning is the problem of removing the effect of subset training data (the "forget set") from trained model e.g. comply with users' requests delete their data, or remove mislabeled, poisoned or otherwise problematic data. With unlearning research still being its infancy, many fundamental open ques- tions exist: Are there interpretable characteristics forget sets that substantially affect the difficulty of the problem? How do these characteristics affect different state-of-the- algorithms?? We present the first investigation into these questions. We identify two key factors affecting unlearning difficulty and the performance of unlearning algorithms. Our evaluation on forget sets that isolate these identified factors reveals previously-unknu don't 't materialize on random forget sets. Based on our insights, we develop a framework coined Refined-Unled Unlearnin Meta-algorithm (RUM) that encompasses: (i) refining the forget set into homogenized subsets, according to different char- acteristics; and (ii) a meta-algorithm that employs existing algorithms to unlearn each subset and finally delivers model that has unlearned the overall forget set. RUM substantially improves top-performing unlearning algorithms. Overall, we view our work as an important step in deepening our scientific understanding of unlearning and revealing new pathways to improving the state-of-the-art.
behaviours of state-of-the-e algorithms that
Introduction
Deep learning models have generated impressive success stories recently by leveraging increasingly large and data-hungry neural networks that are also increasingly expensive to train. This trend has led to reusing previously-traines models for a wide range of tasks more than ever before. However, the heavy reliance of deep models on training data, together with the difficulty of removing data from trained models after-the-fact, has exacerbated concerns perpetuating harmful or outdated information, violating user privacy and other issues. Specifically, deep networks are highly non- convex, making it difficult trace (and thus attempt to remove) the effect of given subset of training data on the model weights. We are therefore faced with important technical challenges when comes building machine learning pipelines that performant while efficiently supporting deletion requests. Machine unlearning 29 a growing field that aims to address this important issue.
While unlearning is receiving increasing attention 34 33
it is still young area of research and the factors affecting the success of different approaches remain poorly-understood. Understanding
Correspondence to Kairan.Zhaaa wwaiee@@@awwwck.uukkkkk 1 Equal senior contribution
"Code is available at: htttp:///ggstuuu..cmm//aaaaaaaee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

what makes an unlearning problem easy or hard is crucial for several reasons, First, knowledge of behaviours of unlearning algorithms on different types of forgetting requests may inform which unlearning method to choose for given request. In fact, for some requests it may be that all current methods are inadequate, suggesting that one should pay the cost of retraining from scratch rather than opting for "approximate unlearning" that imperfectly removes information after-the-face Further, deepening our understanding of unlearning can illuminate pathways for improving both unlearning algorithms well as evaluation protocols by focusing on relevant factors that affect difficulty.
To this end, we present the first investigation into different factors that characterize the difficulty of an unlearning problem. We find that the unlearning problem becomes harder i) the more entan- gled the retain and forget sets are and ii) the more memorized the forget set examples are. Our investigation reveals that different unlearning algorithms suffer disproportionately as the difficulty behaviours and failure modes of state-of-the-ard
level increases and surfaces rrviiuuslyunknowee unlearning algorithms. Inspired by our findings, we propose a Silineddnnneeainng Meta-algorithm (RUM) for improving unlearning pipelines. RUM contains two steps: i) a refinement procedure that divides the given forget set into subsets that are homogeneous with respect to relevant factors that influence algorithms' behaviours, and ii) a meta-algorithm that dictates how to unlearn each of those subsets and compose the resulting models to arrive at one that has unlearned the entire forget set. Our thorough investigation shows that RUM boosts unlearning performance of several state-of-the-ar algorithms and addresses issues that our investigation of unlearning difficulty has uncovered.
2 Preliminaries
2.1 Unlearning problem formulation
Let 0 = A((( be the weights obtained by applying training algorithm A on a training dataset Dirain. We will refer 0 the "original model". Further, let S C Drroin denote subset of the training data referred to as the "forget set".. For convenience, we will refer to its complement as the "retain set" = train S. Informally, the goal of an unlearning algorithm UI is utilize 6", S and R to produce an unlearned model 6th = U(0", S, R) from which the influence of S is removed.
This idea has been formalized by considering the distributional similarity between the model 6 produced by 2/ and the model 0" produced by the optimal unlearning approach: retraining from scratch on an adjusted training dataset that excludes the forget set: 0" A(Dtrain \ ().. Note that we refer to distributions here since rerunning U and A with different random seeds (that control e.g. the initialization and the order of mini-batches) will lead to slightly different weights. The ideal unlearning algorithm then, according to this viewpoint, is one that yields the same distribution of weights as retraining from scratch. Of course, for an unlearning algorithm be practical, we would additionally desire it to be significantly more computationally efficient than retraining the model.
The following definition, borrowed from 333 (and similar to [14 formalizes this idea in framework inspired by differential privacy [].
Definition 2.1. Unlearning, An unlearning algorithm 2 is an (e, (), --uullearnrr (for A, Dirain and S) the distributions of A(Drrain SS and U((0 ,, Derain \. 5) are (e, -close.
where we say two distributions μ1, v are ()-lose if ((B) S c + 5 and v(B) S (( + for all measurable events B.
According to the above definition, an unlearning algorithm is said to be exact unlearning if it satisfies the above definition for e = 5 = 0, i.e., it yields distribution of models identical to that of retraining from scratch. For neural networks, the only known exact solutions involve retraining, either naively, or in the context of mixtures where one retrain only a subset of models affected by the deletion request B. These approaches unfortunately are inefficient; in the worst-case, even clever schemes suffer inefficiency similar naive retraining, and may also yield poorer performance. To address this, a plethora of approximate unlearning algorithms have been recently proposed, whose e and o values aren't 't known in general, but substantially more efficient and may have higher utility.
Evaluating approximate unlearning. Since the success of (most) approximate unlearning algo- rithms cannot be proved within tight (e. bounds, the community has considered various empirical measurements of success, guided by three desiderata: i) good forgetting quality, 2) high utility, and 3) efficiency. An unlearning algorithm thus is faced with complex balancing act, as there are

---

well-known trade-offs both between forgetting quality and utility, as well as forgetting quality and efficiency, and good unlearning metric should capture these nuances.
Utility and efficiency are straightforward to measure, and, in the context of classifiers, can be represented by the accuracy the retain and test sets, and time in seconds, respectively. Measuring forgetting quality, on the other hand, is more complex and several proxies have been proposed. The simplest one is to inspect the accuracy on the forget set, with the goal of matching the accuracy on the forget set that would have been obtained by retraining from scratch. Alternatively, inspired from privacy literature community [25] 26 20 to measure forgetting quality. In essence, an MIA is designed to infer from the model's characteristics (e.g. loss, confidence) whether data point has been used in training, and then unlearned, versus was never trained on in the first place. Intuitively, the failure of an attacker to tell apart unlearned examples from never-seen examples marks a success for the unlearning algorithm in terms of this metric. We will consider both of these proxies in our experimental investigation.
Membership Inference Attacks (MIAs) have been adopted by the unlearning
To holistically evaluate an unlearning algorithm, we desire a single metric that captures both forgetting quality and utility. We will later introduce a "lug-of-war" metric for this purpose, inspired by [33
2.2 Memorization
Deep neural networks are known to "memorize" (a subset of) their training data, with recent theory showing that label memorization is in fact necessary for achieving dlosetto-optimal generalization error in classifiers [I] when the data distribution is long-tailed.
Definition 2.2. Memorization score [1]. The memorization score for an example i E DD with respect training dataset ID and training algorithm A is
mem(AA, i) = f~A(D) 1 [fx;) = vil f~A(D())
Pr [f(x,) = yil
where and y/ are the feature and label, respectively, of example i.
The first term in the above equation considers models trained on all of D whereas the second term considers models trained on D excluding example i. Intuitively. the memorization score for an example is high if including in training yields a different distribution of predictions on that example than excluding it from training would have. Recent works [1] 12 23 identify atypical examples or outliers of the data distribution as examples that are more highly memorized: if an example has noisy incorrect label, the model is required to memorize it in order to predict it correctly.
3 Related Work
Approximate unlearning algorithms. A plethora of algorithms have been proposed that aim to identify effective data scrubbing procedures post-training. We now describe representative methods.
Fine-tune [36] 15] relies on catastrophic forgetting to diminish the confidence of the original model 8 on S. Catastrophic forgetting induced by simply fine-tuning on the retain set Dtrain \ SSS On the other hand, NegGrad [15] [7 31 instead directly maximizes the loss on S. This approach has been found empirically to cause large drop in the utility of the model. To address this, NegGrad+ (25) combines fine-tuning and gradient ascent, by jointly minimizing the loss function on the retain set, and maximizing the loss function with respect to the forget set. SCRUB, proposed by the same authors as NegGrad+, extends the contrastive learning behind NegGrad+ by framing as student-teacher problem. Concretely, SCRUB is a bi-optimization algorithm, where the student aims to mimic the teacher's behaviour on R and to disobey the teacher's output with respect to S. Li-sparse (26] infuses weight "sparsity" into the unlearning algorithm by fine-tuning on the retain set with an Ll-penalty, drawing inspiration from the model pruning literature [13] 27]. Influence Unlearning 21 24 arrives at the important model's 's weights by estimating how removing a data point affects 8 via influence functions 77. and draws connections to (c, 8) forreetnnn [9].
methods that trick the model to learn new labels for This can be achieved by finetuning the model with respect to dataset Drotahet (Xs, Y), where s are the features and labels Y are sampled from a prior distribution of the label space. Saliency Unlearning (SalUn) [10 learns Dreiabel by optimising only the salient parameters of the model.
different line of work "relabelling
3

---

Concretely, the authors argue that the model's weights 8 can be decomposed into salient weights and "intact" model weights, by investigating the weight space with respect the forget set S ala 30
Difficulty of Unlearning. The closest research to ours is the contemporaneous work of [9] where the authors study adversarial unlearning cases, i.e. "worst-case" forget sets. M arrives at difficult
forget sets by solving bi-level optimization based on fine-tuning (i.e. catastrophic forgetting). Instead, we arrive at difficult partitions through the lens of interpretable factors: the degree of entanglement between the retain and forget set and memorization. While the primary aim of (9 is to construct more pessimistic evaluation benchmarks, our primary aim is to deepen our understanding of unlearning problems and of the behaviour of state-of-the-arr algorithms when operating on forget set of different identified characteristies, ultimately improving unlearning pipelines.
Catastrophic forgetting, atypical examples and privacy.
122 and 32 study catastrophic forget-
ting during training. [22] finds that, when training on large datasets, examples that were only seen early in training may enjoy better privacy, in terms of MIAs and extraction attacks, compared to examples seen recently. [32 investigate "forgetting events", where an example that was previously correctly predicted becomes incorrectly predicted later in training. They find that examples with noisy labels witness a larger number of these forgetting events. Further, [5] find that models trained with Differential Privacy (DP) find it primarily hard to correctly predict atypical examples. We build on this literature by studying the difficulty unlearning after-the-face rather than (passive) forgetting during training and draw connections to memorization, notion closely related to atypicality in the data distribution.
What Makes Unlearning Hard?
In this section, we identify and empirically examine two factors that affect the difficulty of unlearning. Before diving in, we first define a simple proxy for unlearning difficulty that we will use in this section. Our goal is to capture the difficulty of performing the "balancing act" of forgetting S while retaining the ability to perform well on R and generalize well to the test set. We propose a metric to capture this "tug-of-war (ToW) using the relative difference between the accuracies of the unlearned and the retrained model on the forget, retain and test sets, in a manner inspired by [33]
TOW 0", Drest) (1 da(0", 10000,0" 0", ()) da 0", R)) (1 da(l", 0", Drest))
where a(0, D) = T27 E (a.))) (f 8) = y] is the accuracy on D of a model f parameterized by 0 and da(O", 0", D) = la D) a(0", D) is the absolute difference between the accuracy of models l and f on D. Therefore, ToW rewards unlearned models that match the accuracy of the model, each of the forget, retain, and test sets. ToW ranges from 0 to 1, with higher values associated with better unlearning.
4.1 The more entangled the forget and retain sets are, the harder unlearning becomes
Prior research (e.g., by Feldman et al 11 and Carlini et al (5) has tried to identify prototypical (or atypical) examples and their impact on learning. Primarily, this depended on the position of examples within the overall data space distribution. In contrast, here we focus the embedding space, as unlearning depends heavily on how the model has learned to represent training data. Furthermore. instead of looking isolated examples, we delve into how "entangled" the retain and forget sets are in embedding space. We hypothesize that higher "entanglement" leads to harder unlearning: if the two sets are highly entangled, attempting erase S will cause accidentally erasing R too.
We propose measure entanglement between the retain and forget sets via the below Entanglement Score (ES), inspired by measure previously introduced in [16] to study learned representations.
(S [ EiER(d
Ejes(0s 2
[(URR - -1² + (us - 11)2)
where di = g( 00) is the embedding of example x; according to the "original model" f. parameter- ized by 0º; where g denotes the forward pass through of up till the penultimate layer, i.c. excluding the

---

1.0
1.0
0.8
1000
0.8
0.4
orggnal fombung Lisper
Heg Hog hegganat nesgondetyou wileesse
Randem
0.2
Roual orggnn tor sode Hosp "egeae Wegenntsaan whuets saton
Randerr
Unieaming method
Unleaming method
(a) Entanglement Score (ES) ToW
(b) Memorization vs ToW
Figure I: Uncovering two factors that affect unlearning difficulty according to ToW (where higher is better). Left: the more entangled the retain and forget sets are in the embedding space, the harder it is to unlearn. Right: the less memorized a forget set is (thus having influenced the model less), the easier it is to unlearn (for most algorithmss. Error bars correspond to 95% confidence intervals from running each algorithm 3 times (6 times for relabelling-basee that had higher variance).
classifier layer. Further, pa TR] EiEr de is the mean embedding of the retain set, and analogously, Iss the mean embedding of the forget set, while y is the mean embedding over all of Dirain - RUS. S.
1
Intuitively, ES measures entanglement between S and R in the embedding space of the original model (before unlearning begins). The numerator measures intra-class variance, capturing the tightness of each of those two sets, independently, while the denominator measures inter-class variance between those two sets. Higher ES score corresponds to higher entanglement in the embedding space.
Our investigation hinges on generating three different forget/retain partitions with different degrees of entanglement: low, medium, and high. But, Equation (2) does not directly suggest procedure for generating retain/forget partitions with desired ES score. Hence, we achieved this indirectly using a proxy. Specifically, let d(i, p.; 09) = ||;; 2112 denote the 12-distance in the original model's embedding space between example and centroid as defined above. We compute this distance for each example in Dirain and sort those examples according their d-values. We then form each forget set to contain contiguous subset of examples from different ranges of that sorted list. We find that this procedure allows us construct retain/forget partitions of varying ES values. The ES values for our low, medium and high partitions are 309. 98. 56, 2011224-558.66 63, and 3876.90 42-6999--222.2 for CIFAR- 100. We
64, 1612.21+110.82
for CIFAR- 10, and 963. 1 include details of this procedure in the Section A.3 along with visualizations and Maximum Mean Discrepancy (MMD) analysis to further validate ES, confirming that the degree of retain/forget entanglement aligns with the computed scores. We experiment with four dataset/architeccuue settings: MEAAR-O//ReSee-I1 18, MUTAR--00/ResNet-5500 Tiny-ImageNer/ResNee-- and Tiny-ImageNe//VGG 16, using |S] = 3000. Refer to Section A.2 for implementation details and Section A.8 for more detailed results on all datasets.
We observe from Figure la that it is harder to unlearn when the retain and forget sets are more entangled: all unlearning algorithms have poorer performance for highly-entangled vs lower-entangled settings. Further, we notice that different unlearning algorithms suffer disproportionately as the entanglement increases. Notably, methods based on relabelling (SalUn and Random-label) perform very poorly when the entanglement is high. We hypothesize that this is because, if two examples i and j are close neighbours in embedding space, with in the forget set and j in the retain set, forcing example i to be confidently predicted as incorrect class (as relabelling al algorithms do) will also cause j to be predicted as that incorrect class, too, thus causing drop in retain accuracy, which is captured by ToW. This effect will be less pronounced if and are far from each other.
4.2 The more memorized the forget examples are, the harder unlearning becomes
Feldman et al || have already established that models must memorize some atypical examples in order to perform well. Further, prior literature has also established that noisy examples (that are more likely to be memorized) witness more "forgetting events" during training (their predicted label flips to an incorrect one) (32 and that models trained with Differential Privacy (DP), a procedure where noise is added to the gradients (making it harder to memorize), find it primarily hard to correctly predict atypical examples [5] In this section, we build upon these prior insights by investigating the connection between the degree of memorization of the forget set and difficulty of unlearning.
5

---

Let's begin by inspecting Definition 2.2 if an example is not really memorized, the predictions of the model on that example will not change much whether the example was included in training or not. This implies that even the original model (no unlearning) is similar to cerrainfrromscratee of predictions on those examples, making unlearning unnecessary or trivial. On the other hand, for examples, the predictions between the original and retrained models will differ significantly, implying that an unlearning algorithm has "more work" to do to turn the original model into one that resembles the retrained one. We now investigate how the level of memorization of the forget set affects the behaviour of state-of-the-art unlearning algorithms. We hypothesize, based on our above intuition, that unlearning is easier when the forget set contains less- memorized examples.
in terms
To investigate this, we first compute the memorization score mem (A, Dtrein: i) of each example i E Dtrain and we sort all examples according to their scores. We then use that sorted list to create three different forget sets, corresponding to the lowest N scores ("low-mem"), the highest N ("high-mem"), and the N that are nearest to 0.5, i.e. the midpoint of the range of memorization scores ("medium-men"s of these forget sets and compute ToW. We perform this experiment on C1FAR-10 using ResNet-18 and on CIFAR-100 using ResNet-50. Refer to Section A.2 for implementation details.
where N = 3000. We then apply different unlearning algorithms on each
We first emphasize two key sets of conclusions. First, in terms of ToW, Figure 1b shows that, indeed, for most algorithms, the lower the memorization level of the forget set, the easier the problem is. In line with our prior discussion, even the original model performs well on ftw-mem",, but performs very poorly on "high-mens
Interestingly though, the two relabelling-buss -based algorithms (SalUn and Random-label) follow an inverse trend: they perform better for nigher-memorized forget sets. Second, breaking down ToW into its parts, we find interesting trends in terms of the forget set accuracy, in Figure [2] Specifically, for several unlearning algorithms, the forget accuracy for "low-mem" is still very high after unlearning them, as the model can infer the correct labels for such examples even when they weren't 't included in training; this follows directly by Definition 2.2 if unlearning is done by retraining, and is shown here for the first time for approximate unlearning algorithms. On the other hand, we find that, for "high-mem" different unlearning algorithms can (to varying degrees) cause the forget set accuracy to drop substantially; this is consistent with both 32] and (5). but shown here for the first time for approximate unlearning algorithms. Notably, find that relabelling-basse algorithms cause a larger drop in the accuracy of the forget set, relative to other approaches. This benefits ToW in the case of "high-mem" forget sets, where retraining has poor accuracy on this set (so they get rewarded by matching it), but it hurts on "low-mem", since it causes a large discrepancy from retraining, which has high accuracy this set (since it makes similar predictions to the original model on this set, by definition, and the original model has high accuracy on all of Derain).
Overall, we have presented the first investigation into the behaviour of unlearning algorithms applied on forget sets of different degrees of memorization. A key finding is that different algorithms outshine others for different forget sets. Most notable is the failure of relabelling-baae based algorithms the "low-mem" " forget set, which is casy for other algorithms and, in fact, even no unlcarning in that case might be acceptable solution. We intuit that this is due to their aggressive unlearning strategy yielding "overforgetting" (producing a forget set accuracy that is lower than that of retraining from scratch) discussed above. Furthermore, observe that different unlearning algorithms work best for the "low-mem", "medium and "high-mem" forget sets. Concretely, from Figure| [T] we note that Finetune is best for "medium-mem" SalUn is best for "high-mem", and number of algorithms are top-performers for "low-mem" (including no unlearning). This reveals a possible pathway for improving unlearning based on using different algorithms for different forget sets. So, how can one build on these insights to further improve unlearning algorithms performance?
5
Refined-Unleariinn Meta-algorithm (RUM) for Improved Unlearning
Previously, we observed that unlearning algorithms have different behaviours on forget sets with different properties. For example, while "low mem" forget sets are almost trivial to unlearn (and even doing nothing may be acceptable), SalUn and Random-label perform poorly on them. On the other hand, SalUn and Random-labe evidently outperform other unlearning algorithms on "high mem". These observations suggest that the optimal unlearning algorithm to use is dependent on the properties of the forget set. One could therefore pick the best unlearning algorithm for each unlearning request, based on these factors. However, in practical scenarios, forget sets may be distributed differently than in our preliminary experiments, that were designed to cleanly separate different factors of interest.

---

Indeed, real-world forget sets would likely contain mixture of examples from different modes of the data distribution, some rare or highly-memorized while others common and not memorized at all. So, what can be done about these expected heterogeneous forget sets? How can our insights above be leveraged to improve unlearning for such cases?
To address this, we first propose a refinement procedure that divides forget sets into homoge- neous subsets (with respect to the factors that we have found to affect the difficulty of unlearning and behaviours of existing algorithms), Second, we propose utilize pool of state-of-the- art algorithms to unlearn different subsets. Put to- Meta-
Foot on LEIVEIINg A.yrmmr
RUM
27 mich
IV,
5)
gether, we propose a Refined-Unlearee algorithm (RUM), comprised of two steps: 1) Refinement and 2) Meta-unlearning. Figure 2 overviews RUM.
Dedinn dor was
o sovee
S
U USS
Step 1: Refinement. We introduce a function FF that partitions the forget set S into K subsets: {5 = F(S) such that each forget set ex- ample appears in exactly one such subset. The intention of F is to generate homogeneous subsets w.r.t factor(s) that affect difficulty / algorithm's behaviours.
Figure 2: Overview of RUM.
Step 2: Meta-Unlearning. Having obtained the subsets {S of S, we now require "meta- algorithm" M that dictates how to perform the individual unlearning requests and how to compose the resulting unlearned models to arrive to model that has unlearned all of S. In this work, we focus on meta-algorithms that tackle unlearning of subsets in a sequence, leaving other designs for future work. It therefore remains for our "meta-algorrthm" to decide: i) what unlearning algorithm to apply for each subset, ii) what order should the unlearning requests be executed in.
More concretely, we assume access to pool of existing unlearning algorithms 24 .N.,, like the ones described in related work, for instance. Let (M(SS.) denote a procedure that takes as input a subset S; and returns an unlearning algorithm 2/ E {Uh [ } that will be used for that subset. This selection can be done by leveraging insights such as those in Section 4 Further, let M denote a
procedure that takes input the K subsets of S and returns sorted list S (FF)F))) containing the subsets in the desired order execution.
Given the above ingredients, RUM proceeds by executing K unlearning requests in a sequence, with step of that sequence corresponding to unlearning subset [i] by applying (1 ('] R,) 0% where 0h denotes the unlearned model up step and U4 (M (i)) and R, = R J (S"i 1],....S' [K]} }} is the retain set for step i, containing R as well as all other subsets of S that have not yet been unlearned in the sequence far. We finally return the unlearned model of the last step 0%-
Our RUM framework is meant as an analysis framework, surfacing new problems to be solved and offering pathways into future of-t--of-the-ar algorithms. Nonetheless, contribute below specific top-performing RUM instantiations, with specific choices for F and for M.
6 Experimenting with RUM flavours
We now present RUM instantiations using a refinement strategy based on memorization scores and experimental evaluations answering the following questions: Q1: How useful is refinement alone? That is, for given unlearning algorithm 44, does applying U sequentially on the K homogeneous subsets of S outperform applying 2/ once all of S? Q2: Can we obtain further gains by additionally selecting the best-performing unlearning algorithm for each forget set subset? Q3: Are there interpetable factors behind the boost obtained by sequential unlearning of homogeneous subsets?
Experimental setup We experiment with refinement strategy based on memorization scores where K 3. Specifically, we study unlearning a forget set that the union of the three sets containing the N lowest, the N closest to 0.5, and the N highest memorized examples in the dataset,
7

---

Liinaming methed
Uninarring
(a) CIFAR-10 with ResNet-18
(b) CIFAR-10 with R8sNet-18
(c) CIFAR-100 with ResNet-50
From subplots a and c, we observe that RUM improves each unlearning algorithm. Vanilla corresponds to unlearning S in one go, whereas Shuffle and RUM operate sequentially on subsets of S. In the case of RUM the 3 subsets are the result of applying J and the order is low + medium - high, whereas Shuffle uses equal-sized random subsets, serving Further, from subplot b, we observe that full RUM, equipped with the best algorithm for each subset (do nothing - Fine-tune - SalUn), yields the overall best results (note: in CIFAR-100 100, NegGrad+ is the best algorithm, so full RUM corresponds to the RUM variant of NegGrad+).
Figure 3:
control experiment.
where N = 1000, making the overall size of the forget set 3000. We conduct experiments on two different datasets: CIFAR-10 and CIFAR- 100, using ResNet-18 18 and ResNet-50, respectively. We evaluate unlcarning algorithms both in terms of ToW, before, and using commonly-used MIA [6] that is a binary classifier trained to separate from Drest and then queried on examples from S. Following previous work, report "MIA" the fraction of examples from S that were predicted to be held-out. The goal is to match the "MIA" score of retrain-from from-scratch. For convenience, we report the "MIA gap": the absolute difference of the MIA score of unlearning from the MIA score of scramsc (lower is better). Further details on MIA setup are in Section A.4 We also introduce new ToWMMA" which mirrors ToW but replaces its "forget quality" component (i.e., its accuracy on S) with the MIA score. Results for ToW-MIA are presented in Section A.4 and A.8 and largely follow the the unlearned and retrained models, we additionally analyze the number of examples where their predictions disagree (see Section A.7]. We also consider and analyze the effect of two different orderings: i) low + medium - high and i) high + medium + low.
trends. To gain finer understanding of the differences between
How useful is refinement alone? To investigate this, we selected a subset of perfhesnppr al- gorithms and apply each algorithm Uí in three ways: i) "vanilla", i.e. applying U1(0", S R) as usual, ii) "shuffle" where Lí is applied sequentially equal-sized subsets of S that were determined ran- domly, and iii) RUM where L/ is applied sequen- tially on the subsets obtained by F(S), i.c. utilizing only the refinement step of RUM and applying the
CIFAR-100
"WAN <t>
MIA
ToW
1.00::00000
0.000
0,000
Fine sune vanilla
0.120
6.732:0025
0.139
three
Fine suee startle
BL0SS
C35P-100
0.345
Fine RUM
0L999
0.764-0069
0,093
Li RUS""
0.716500025
0.577
0604=0022
0,355
11900=900200
0
Negeieate venila
0.230
0.159
Neytitat shuffle
100022000222
0.520
0813=10054
0,417
NegGrade RUM*
18879:900668
0.134
0.21-00054
0,059
same U on each subset. We include "shuffle" as
8.727-000300
0134
0.538:00019
a control experiment, so that any gain of RUM over shuffle is due to homogenization rather than simply reducing the size of the forget set or other effects of sequential unlearning. We observe from Figure 3 and Tables [6a]and 16b that, for four dif- ferent unlearning algorithms and on two different datasets, RUM significantly outperforms "vanilla" and "shuffle", indicating that operating sequentially on homogenized subsets according to memoriza- tion can boost the performance of unlearning algo- rithms. Interestingly, in most cases, "shuffle" actu- ally performs worse than "vanilla", indicating that the difficulty of the problem may increase rather than decrease given poor refinement strategy.
RUM
1886729.069
0614:00057
RUM
00054
0.921-80644
0.059
Table 1: Each algorithm 4 is applied in three
ways: i) in one-go ("vanilla"), ), ii) on a random partition of S into equal-sized subsets, se- quentially ("shuffle"), and iii) on three equal- sized subsets obtained by F in low + med - high order ("RUM""). In last row, RUM ad- ditionally chooses the best algorithm for each subset: none + Fine-tune + SalUn in CIFAR- 10 and NegGrad+ RUM in CIFAR-100.
Can we further boost performance by per-subset algorithm selection? To answer this, we leverage our findings from Section 4 to identify the best unlearning algorithm for each of the "low-
8

---

1.0
08
0.7-
orignal
Rostuna
13 1i-pane
Seglaadt
scRus
Sanon
Rne-unne
Suth
Neyoled4
Li-sparze
Unicarning mothod
Unieerning method
(a) C-proxy vs ToW
(b) RUM on CIFAR- 10 using C-proxy
Figure 4: Replacing mem scores with the efficient C-proxy yields similar trends and performance gains, carving a path for practical deployment of RUM. Left: forget sets with lower C-proxy values (i.e., higher mem scores, since C-proxy and memorization unlearn, consistent with the trend in Figure lb Right: RUM using C-proxy in the refinement step enhances unlearning performance across algorithms, comparable using the memorization score in Figure 3al Error bars correspond to 95% confidence intervals, with each algorithm run 3 times.
negatively correlated) are harder to
mem", "medium-ment and "high-mem" scenarios. On CIFAR-10, this corresponds to doing nothing for low-mem (i.e. using the original model directly), using Fine-tune for medium-mem and SalUn for high-mem (based on Figure lb From Figure 3b (and Table 16ab, we observe that get the overall best results by far, both in terms ToW and MIA, by applying RUM with "nothing - Fine-tune + insights into unlearning pipelines. In fact, lets
SalUn", demonstrating the value of incorporating revisit our previous observation that SalUn and Random-Label perform uncharacteristically poorly on the "low-mem" forget set. In line with our hypothesis, we notice that "nothing - SalUn - SalUn" outperforms applying SalUn on all three subsets (a. SalUn RUM ). Note that, for CIFAR-100, the best algorithm for all subsets is NegGrad+, full RUM corresponds to NegGrad+ RUM
Can we obtain similar performance boosts with compute-efficient proxies? Given the compu- tational cost of calculating memorization scores, we aim to find a more efficient proxy to enable practical deployment of RUM. We discuss a confidence- based memorization metric, termed "C- proxy", as proxy for memorization [23] 32). Section provides detailed explanation of the proxy and the complete results across various datasets and architectures. Figure 4al shows that the observed unlcarning difficulty sptcims-pecifiialll. that forget examples with higher memorization scores (i.e., lower C-proxy values, as they are negatively correlated; see Table 4 are harder to unleam--remins consistent when using C-proxy, paralleling the results in Figure 1b This pattern is further confirmed on ImneemmaeNNet with both ResNet- 18 and the VGG-16 architectures, as shown in Figure
9 We then examine whether using C-proxy achieves similar performance gains within RUM as the original memorization score. Figure(4bppe presents results on CIFAR- 10 with ResNet-18, using C-proxy in place of memorization score during the refinement step. This analysis is also extended to Imagemm with both ResNet-18 and VGG-16 architectures, as shown in Figure IO and Table [7 Together, these results suggest that C-proxy is practical and compute-efficient alternative. delivering significant performance gains comparable to those achieved with the memorization score. Detailed analysis of the use and appropriateness of various memorization proxies and their effect on RUM can be found in (37.
Analysis of sequence dynamics We report ToW and MIA with different orderings in Tables 16a and T6b We find that, while ToW is similar for different orderings, MIA vary. To better understand these dynamics, we inspect the accuracies on S and its subsets after each step in Figure 5 for SalUn on CIFAR-10 10 and in Figure 13 for NegGrad+ CIFAR-100. The former reveals why SalUn" with low - med + high order greatly outperforms vanilla SalUn. Recall that we identified Section 4.2 that SalUn "overforgets" low-mem examples (its forget accuracy is lower than that of retraining). We observe from Figure that future steps of the sequence neutralize that overforgetting effect on low-mem, leading to better ToW (see Figure 3). Interestingly, in line with our previous insights (Section 4.2). we find (from both Figures 5 and 13) that it is hard to cause the "low-mem" accuracy to become low and stay low. NegGrad+ does not drop for any order of execution; SalUn drops it in the first ordering, but that drop is later reversed. We leave it to future work to further study these sequential dynamics and their helpful or harmful effects on Tow and MIA. The fact that MIA results differ based on the sequence may tie in with the recently-identinn "privacy onion effect" [6]

---

SUP
May Drw mordd
Gee 2ongh nord
Step DOG moott
RUN brsuher mele!
May doe mwn.
(8) low + med + high (MIA gap: 0.031)
(b) high + med low (MIA gap: 0.221)
Figure 5: Sequence dynamics for SalUn RUM on CIFAR-10. We report the accuracy on overall S, R and Drest, and subsets of S after each step. Both orderings yield similar Tow (see Table T6a
7 Discussion and conclusion
We presented the first investigation into interpretable factors that affect the difficulty of unlearning. We found that unlearning gets harder i) the more entangled the forget and retain sets are in embedding space and ii) the more memorized the forget set Our investigation led into uncovering previously- unknown behaviours of state-of-the-ar algorithms that were not surfaced when considering random forget sets: when do they fail, when do they excel and when do they exhibit different trends from each other. Notably, discovered that relabelling-ben based methods suffer disproportionately as the embedding-sppce entanglement increases, and exhibit a reverse trend compared to other methods in their behaviour for different memorization levels. Armed with these insights, we then proposed the RUM framework surfacing new (sub)problems within unlearning, whose solution may lead to greater performance. Finally, derived specific instantiations of RUM and analyzed how its different components can improve performance. We found that sequential unlcarning of homogenized forget set subsets improves all considered fate-of-the-an art unlcarning algorithms and investigated the dynamics of sequential unlearning to glean insights as to why that is. We also found that we can further boost performance by selecting the best unlearning algorithm per subset.
Efficiency How this important aspect affected in our sequential framework? We remark that depends on the unlearning algorithm. For instance, applying Fine-tune three times is much more expensive than applying once, because Fine-tune performs (at least) one epoch over the entire retain set. But for other algorithms the overall cost does not increase significantly. Further, key observation from our results is that we can do well by actually doing nothing on a subset of the forget set, which can really boost efficiency (especially since the vast majority of examples are "low mem").. Additionally, our results with the C-proxy demonstrate that significant performance improvements in unlearning be achieved with minimal computational cost, avoiding the heavy expense of computing memorization scores.
Data-space vs embedding-space outliers How does the embedding space entanglement interact with the level memorization of the forget set? We analyzed this and found in Table that all of our memorization buckets have relatively high ES, indicating that separating out (data-space) outliers in the forget set doesn't 't lead to lower entanglement between the two sets in the embedding space. We leave it to future work to study the interaction of these factors.
Limitations and future work We hope future work explores other refinement strategies (e.g. for a notion that captures embedding sequential RUM, c.g. in terms of the "privacy onion effect" (6. We also hope to see how our RUM framework can be adopted and adapted for unlearning in LLMs, especially given the findings from the contemporaneous paper [2] where unlearning is performed only on the highest-memonte in the forget set (albeit, memorization is defined differently for LLMs). We hope our framework continues to enable progress in understanding and improving unlearning and that our identified factors of difficulty and associated behaviours of existing algorithms continue to improve the state-of-the-ar and inform the development of strong evaluation metries that consider forget sets that vary in terms of relevant identified characteristics.
entanglement) and investigates privacy implications of
examples
10

---

8 Acknowledgements
We thank Vincent Dumoulin and Fabian Pedregosa for valuable conversations and feedback various stages of the project.
References
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
[2] George-Oetavian Barbuleseu and Peter Triantafillou. To each (textual sequence) its own: Improving memorized-dale unlearning in large language models. In International conference on machine learning, 2024.
[3] Lucas Bourtoule, Varun Chandrasekaram, Christopher A Choquette- Chane, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159. IEEE, 2021.
[4] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914. IEEE, 2022.
[5] Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv:1910.13422 2019.
[6] Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. Advances in Neural Information Processing Systems, 35:13263-13276 2022.
[7] R Dennis Cook and Sanford Weisberg. Criticism and influence analysis in regression. Sociolog- ical methodology, 13:313-361, 1982.
[8] Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pages 1-12. Springer, 2006.
[9] Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. Challenging forgets: Unveiling the worst-case forget sets in machine unlearning. arXiv preprint arXiv:2403.07366, 2024.
[10] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. arXiv preprint a3Xiv: 10. /2508, 2023.
[11] Vitaly Feldman. Does learning require memorization? short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 2020.
[12] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891 2020.
[13] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.
[14] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making forget you: Data deletion in machine learning. Advances neural information processing systems, 32, 2019.
[15] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304-9312, 2020.
[16] Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Goldstein. Unraveling meta-learning: Understanding feature representations for few-shot tasks. In International Conference Machine Learning, pages 3607-3616. PMLR, 2020.
11

---

[17] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516-11524, 2021.
[18] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[19] Chuan Guo, Tom Goldstein, Awni Hamnun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv: 1.03030, 2019.
[20] Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy, arXiv preprint arXiv:2403.01218, 2024.
[21] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008-20166 PMLR, 2021.
[22] Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. Mcasuring forgetting of memorized training examples. arXiv preprint arXiv:2207.000990 2022.
[23] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206, 2020.
[24] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
[25] Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards un- bounded machine unlearning, Advances in Neural Information Processing Systems, 36, 2024.
[26] Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, PRANAY SHARMA, Sijia Liu, et al. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36, 2024.
[27] Xiaolong Ma, Geng Yuan, Xuan Shen, Tianlong Chen, Xuxi Chen, Xiaohan Chen, Ning Liu, Minghai Qin, Sijia Liu, Zhangyang Wang, et al. Sanity checks for lottery tickets: Does your winning ticket really win the jackpot? Advances in Neural Information Processing Systems,
1111279--12600. 2021.
[28] Justus Mattern, Fatemchsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor via neighbourhood comparison. arXiv preprint ar0iv: 18462, 2023.
Membership inference attacks against language models
[29] Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint 02X.v9220..02299, 2022.
[30] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint arXiv:1706. 706.03825. 2017.
[31] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors inftuencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303-33 319. IEEE, 2022.
[32] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning, arXiv preprint
2018.
[33] Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, Lisheng Sun Hosoya, Sergio Escalera, Gintare Karolina Dziugaite, Peter Triantafillou, and Isabelle Guyon. Are we making progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint arXiv:2406.09073 2024.
12

---

[34] Eleni Triantafillou, Fabian Pedregosa, Jamie Hayes, Peter Kairouz, Isabelle Guyon, Meghdad Kurmanji, Gintare Karolina Dziugaite, Peter Triantafillou, Kairan Zhao, Lisheng Sun Hosoya, Julio C. S. Jacques Junior, Vincent Dumoulin, Ioannis Mitliagkas, Sergio Escalera, Jun Wan, Sohier Dane, Maggie Demkin, and Walter Reade. Neurips 2023 machine unlearning. 2023.
[35] Junxiao Wang, Song Guo, Xin Xie, and Heng Qi. Federated unlearning via class dass-discriminative pruning. In Proceedings of the ACM Web Conference 2022, pages 622-632, 2022.
[36] Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine un- learning of features and labels. arXiv preprint 1rXiv:2108.11577, 2021.
[37] Kairan Zhao and Peter Triantafillou. Scalability of memorization- machine unlearning. In NeurlPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability.
13

---

A Appendix/ supplemental material
A.1 Broader impact
Unlearming research can have profound broader impact in allowing users to request their data to be deleted from models or making models safer or more accurate by removing harmful or outdated information. Our work is exploratory; we identify factors affecting difficulty that are interpretable and can drive improvements to unlearning pipelines, As such, we don't 't see any direct harmful impact of our work.
A.2 Implementation details
Datasets and models We use the CIFAR-10 10, CIFAR-100, and Tiny-ImageNet datasets for our evaluation. CIFAR-10 consists of 60,000 32x32 color images across 10 classes, with 6,000 images per class. CIFAR-100 contains 100 classes, each with 600 32x32 color images. Tiny-ImageNo includes 100,000 64x64 color images across 200 classes, with each class containing 500 training images, 50 validation images, and 50 test images (for a total of 100,000 training. 10,000 validation, and 10,000 test images). For CIFAR-10 10 and CIFAR-100 100, the train, validation. and test set sizes 45,000, 5,000, and 10,000 images, respectively. We use ResNet-18, ResNet-50, and VGG- 16 as model architectures. Specifically, we use ResNet-18 for CIFAR-10. 10, ResNet-50 for CIFAR-100, and both ResNet-18 and VGG-16 for Tiny-ImageNet.
Training details for original models We trained four models across different datasets and archi- tectures: ResNet- 18 on CIFAR-10, 10, ResNet-50 on CIFAR-106 ResNet-18 on Tiny-Iman IinyImmaeNNet, and VGG-16 on -iny-ImageNet. For CIFAR- 10, the ResNet- 18 model was trained for 30 epochs using the SGD optimizer. The learning rate was initialized at 0.1 and scheduled with cosine decay. For CIFAR-100, the ResNet-50 model was trained for 150 epochs using the SGD optimizer, with the learning rate initialized at 1.1 and decayed by a factor of 0.2 at epochs 60 and 120. For Ting- the R8 model was trained for 80 epochs, and the VGG-16 model was trained for 100 epochs, both using the SGD optimizer with an initial learning rate of 0.1 and cosine decay. All models were trained with a weight decay of 0.0005, momentum of 0.9, and batch size of 256. Additionally, data augmentations, including random cropping and random horizontal flipping, were applied during training on CIFAR-100 and Imag-em All training was conducted on Nvidia RTX A5000 GPUs.
Training details for machine unlearning To ensure optimal performance, we carefully set the hyperparameters for each unlearning method across different datasets and architectures. For retrain- crom-scratch, followed the exact same training procedure as the original model, but only trained on the retain set, excluding the forget set. For Fine-tune, we trained the model for 10 epochs with a learning rate in the range [0.01, 0. LI-sparse was run for 10 epochs with a learning rate in the range [0.005, 0.1] and a sparsity- promoting regularization parameter in the range [10-6, 10-*] NegGrad was also trained for 10 epochs. with a learning rate in the range [10-1 0.1]. For NegGrad+, we used a 3 value in the range [0.85, 0.99] as weighting factor that balances two components of the loss, training for epochs with a learning rate in the range [0.01, 0.05). Influence unlearning involved tuning the parameter a for the woodfisher Hessian Inverse approximation within the range [1, 100]. SalUn was trained for 5-10 epochs with a learning rate in the range [0.005, 0.1] and sparsity ratios in the range [0.3, 0.7]. Random-label was trained for 10 epochs with a learning rate in the range [0.01, 0.1].
In our experiments with forget retain set partitions at varied levels of memorization or ES, we tuned the hyperparameters to achieve the best ToW performance for each unlearning algorithm. The results are reported as averages with 95% confidence intervals over 3 runs, except for relabeling-bai based methods, which had higher variance and were therefore run times. For the RUM experiment, we adjusted the hyperparameters at each step to ensure that the accuracy after each step closely matched the accuracy obtained by retraining from scratch. This procedure was repeated for all algorithms, with results reported as averages over 3 runs with 95% confidence intervals.
14

---

A.3 Procedure for creating retain / forget partitions with varying ES
To create retain / forget partitions with varied levels of ES, we followed systematic procedure. Initially, we trained the original model ( on the entire training dataset Ptrain. Using 0º, we then extracted embeddings for each data point in Draain. The global centroid for Drainn denoted as μ., was determined by calculating the mean of all example embeddings. For each example in Drrain. we then computed its (2-distance from the global centroid μ in the original model's embedding space as follows:
d(i, μz; = 110 - -A12
We ranked these distances for all data examples in Dtrain and selected the 3000 examples with the highest distances to form the low ES bucket. Subsequently, we moved further down the ranked list, selecting 3000 examples with progressively lower distances form the medium and high ES buckets, until we achieved the desired levels of ES variation. This approach allowed us to form forget sets, each with a size of 3000, categorized into low, medium, and high ES levels. The rationale behind this selection is that examples with high distances from the global centroid are considered "distant" from the overall data distribution in the embedding space and are therefore less entangled with the rest of the dataset, i.c., the retain set. Various unlearning algorithms were then deployed on 6 across different forget retain partitions. Their performance was measured using ToW along with forget, retain, and test accuracy, well as MIA.
This procedure enabled us to create retain forget partitions with varying ES values. The ES values for our low, medium, and high ES partitions are shown in Table 2 As observed from the table, the ES values increase from low to high ES partitions for both CIFAR-10 and CIFAR-I00, confirming the effectiveness of our procedure. We also use Maximum Mean Discrepancy (MMD) [8]] widely-used metric, to further validate ESS. The MMD is computed as follows: Given pre-trained model 6o (the original model in case) trained retain set R using 0% denoted as (SS) and o (RR), respectively. We then apply an RBF kernel to map these features to another space H such that o' dS(), o' (R) E H. The MMD between ((S) and (RR) is then calculated using the following formula:
Drrain.
first extract features for the forget set S and the
MMD (O(S) =
8
S
R
iES
JER
H
Our experiments reveal a negative correlation between ES and MMD scores, reported in Table - For CIFAR-10/ResNet-18 and CIFAR- 100, low (high) ES values correspond to high (low) MMD values, supporting the consistency of the ES.
Additionally, Figure 6 presents the data representation of low, medium, and high ES partitions, confirming that the degree of entanglement between the retain and forget sets aligns with the computed ES values. As we move from low to high ES partitions. the forget set (yellow) and the retain set (blue) become increasingly entangled. This indicates that higher ES partitions reflect greater complexity in distinguishing between the two sets.
A.4 Description of MIA and ToW-MIA
We adopted MIA based on prediction confidence, following the procedure described by (26). To conduct this attack, we first sampled equal-sized data from the retain set and the test set, using these to train binary classifier as the MIA model. This model is designed to distinguish whether data example was involved in the training stage or not.
Next, we applied this attack model to the forget set to evaluate unlearning performance during the testing phase, after an unlearning method was implemented. Intuitively, for successful unlearning, we want forget set examples to be classified as "non-training" data. We define "training" data as the positive class and "non-training" data as the negative class, and measured the performance of MIA by calculating the ratio of true negatives (i.c., the number of the forgetting samples predicted as
15

---

Table ES and MMD scores for the low, medium and high forget retain partitions for CIFAR-10 and CIFAR-100.
Low ES
Medium ES
High ES
ES value
309.941-98.56
1076.99478.64 14.07:t..9))x00+
1612.210 (2.84:0.75)x100 10
MMD value
(5.15::0.26)x10-
(a) CIFAR-10
Low ES
Medium ES
High
ES value
963.82+113.53
2831.244:558.63
3876.90=4426.2
MMD value
(0
(1892.13=0.05)x100
10891.441..88)x00
(b) CIFAR-100
(a) low ES
(b) medium ES
(c) high ES
Figure 6: Data representation visualization for forget / retain partitions with low, medium, and high ES. We used PCA to reduce the dimensionality for visualization. In each figure, the data examples from the forget set are shown in yellow, while those from the retain set are in blue. The global centroid is marked in red the center of the figures.
non-training examples) predicted by the MIA model the total size of the forget set, as shown in following the same procedure as prior work 26
MIA Performance = TN$ [S]
In this context, S C Dtrain represents the forget set and [S] is the total size of this set. The term TNs denotes the number of true negatives predicted by the MIA model, indicating examples identified as "non-training" data. This metric ranges from 0 1, with higher values signifying a larger portion of the forget set predicted as thon-training data. The ideal MIA score, for an unlearing algorithm, one that matches the MIA score of retraining- wrrtiinn--ro--sseeee from-scratchh. Note that, even if applying this MIA on caannn-foo--ccatttt some portion the forget set will classified as "training data" due to heavily resembling the retain set (e.g. examples that
really memorized may have similar confidences regardless on whether or not they were trained on). And we want the model confidences of the unlearned model resemble closely possible those of retrain-frome from--sratchh. For this reason, we additionally report the "MIA gap" (the absolute differentce between the MIA score for unlearning compared to that of foaiii--roo--rratch as casier-to-interppee metric, where lower is better, and the ideal score there is 0.
Building the MIA setup above, introduce another metric "ToW-MIA" to evaluate unlearning performance, similar to ToW described in Section 4 The metric is defined as follows:
ToW-MIA (0th 0", S, R, Drest) (1 dm(0", 0r, ST)) (1 - da(0", 0", R)) (1 da(0", (0"00" "',,,"DDesti Dtest))
where m00, D) represents the MTA performance of the model trained on D. as defined in Equation
33. The first term, dm(0 0, S) mm0,., S) m(0,. S)}, denotes the absolute difference in MIA
16

---

Septradt
(a) TOW-MIA vs ES
(b) ToW-MIA VS memorization
(c) TOW-MIA vs RUM
Figure 7: Evaluating performance with To T-W-MIA (higher is better) for unlearning difficulty analysis
(a, b) and RUM (c). Error bars represent 95% confidence intervals from 3 runs per algorithm.
performance on S between the unlearned model O,, and the term is the key distinction between ToW and ToW-MIA, where "forget quality" is measured by MIA (ToW-MIA) rather than accuracy (ToW). The other two terms remain identical to those in ToW (see the equation in Section 4. representing the absolute difference in accuracy between 0 and 0" on R and Drest respectively, which reflect "model utility". Like ToW, ToW-MIA rewards unlearned models that approximate the performance of the retrained model across the forget, retain, and test sets. ToW-MIA ranges from O to 1, with higher values indicating better unlearning performance. Figure 7 show the same trends with ToW-MIA for ToW, both for unlearning difficulty analysis and RUM. More detailed results using ToW-MIA can be found in Table 14
model 0,. This
A.5 Data-space vs embedding-space outliers
Building on our discussion in Section 4 we identify two factors that affect the difficulty of unlearning: the entanglement between the forget and retain sets, and the memorization level of the forget examples. This raises the question: how do these two factors interact with each other? Are the outliers in data space the same as those in embedding space? Our findings in this section suggest that these factors are indeed distinct.
Memorization within ES-based partitions We analyzed the memorization scores within each for- get set categorized by different ES values. Figure 8 shows the distribution and average memorization scores in the forget sets of low, medium, and high ES categories. It can be seen that each ES category's s forget set contains a mix of memorization levels. Specifically, while the mean memorization score of forget examples increases from low to high ES categories, the memorization scores still span the entire range from to
Entanglement within memorization-basee partitions We examined the ES values for each mem- orization category to understand the entanglement between the forget and retain in the embedding space. Table 3 presents the ES values corresponding to low, medium, and high memorization buckets. The findings reveal that all the memorization bucket exhibits relatively high ES, suggesting that separating outliers in the data space does not reduce the entanglement between the forget and retain sets in the embedding space.
Table 3: ES values for forget retain partitions across varied memorization levels for CIFAR- and CIFAR-100.
Low memorization
Medium memorization
High memorization
ES value
21734.127
32785.711
14736.591
(a) CIFAR-10
Low memorization
Medium memorization
High mamorization
ES value
30028.924
58683.180
20528.561
(b) CIFAR- 100
17

---

2500
793
ree:
500
00
Memoriaation
Memoriaation
0.1
(a) low ES
(b) medium ES
(c) high ES
CIFAR- 10 with ResNet- 18
3400
17c
1222
120
1000
750
250
250
00
05
06
60
02
04
0. 10
0.4
Muntnication Scor
Panssisalion
(d) low ES
(c) medium ES
(f) high ES
CIFAR-100 100 with ResNet-50
Figure 8: Memorization distribution for low, medium, high ES forget / retain partitions. The mean memorization score for low, medium, high ES partitions are 0.084 0030, 0. 0.134:0.235, 0.390 for CIFAR-10, and 0. 0.159-10.283, 2.222-0..29 and 0.317 for CIFAR-100.
These findings demonstrate that embedding space entanglement and the memorization level of the forget set are distinct concepts, not merely different aspects of the same phenomenon.
A.6
Confidence-based proxy for memorization
To improve the efficiency of calculating memorization scores, we leverage insights from previous works 23 32 and introduce a model confidence-basee metric, referred as the C-proxy, which serves as proxy for the memorization score. The computation of the C-proxy proceeds as follows: For each data point (z,, 8:) E we track the softmax probability of the model's prediction ((x,) for the ground-truth label ys across all training epochs, as the model is trained on D using algorithm A. These probabilities are then averaged over epochs at the end of training to capture the training dynamics.
Table presents the fidelity (measured by Spearman correlation) and efficiency (measured by additional computational cost) of the C-proxy in relation to memorization. This evaluation is conducted both CIFAR-I0 and CIFAR-100 datasets. As shown, the C-proxy demonstrates a strong negative Spearman correlation with memorization scores while significantly reducing computational overhead compared to both computing memorization scores and retraining the model from scratch. This suggests that the C-proxy is an effective proxy for memorization while also offering substantial efficiency gains.
Table 4: Comparison of C-proxy based on Spearman correlation with memorization and relative computation time percentages in comparison memorization computation retraining the model from scratch, evaluated on CIFAR-I 10 and CIFAR- 100 using ResNet-18 18 and ResNet-50, respectively.
Dataset
Spearman corr. (mem)
Comp. time % (mem)
Comp. time % (retrain)
CIFAR-
-0.80
0.018%
17.123%
100
-0.91
0.002%
8.175%
18

---

1.0
1.0
TOW
0-4
0.4
Febrin
Orignal
Kimbune
Li-qpare
beglned
scnue
ephoeece
Fetrin
Onggnal
Himbune
Li-parse
wegfrade whoonce
Unvewrning method
Urvearning method
(a) ImageemageNet with ResNet-18
Tiny- ImageNet with VGG-16
Figure 9: Impact of C-proxy unlearning difficulty: C-proxy ToW on Tiny-ImageNer across low, medium, and high C-proxy levels, using ResNet-18 and VGG-16. Error bars represent 95% confidence intervals from 3 runs per algorithm.
wills
0.8
PAM
PUM
TO/
0.6-
0.6
Salnn
LI-500930
Fon
Sainn
Li-S00r00
Unboarning method
(a) Tmny-ImageNet with ResNet-18
(b) Tmny-Im with VGG-16
Figure 10: Impact of C-proyy on unlearning performance on Liny-ImageNer with ResNet- 18 and VGG-16. Each unlearning algorithm is applied in three ways: RUM as well vanilla and shuffle as comparison. Error bars represent 95% confidence intervals from 3 runs per algorithm.
Figure 4a and Figure present results across different datasets and model architectures, demonstrating that the pattern identified Section 4.2 remains consistent when using the C-proxy instead of the memorization score. Specifically, the lower the C-proxy value, the harder it is to unlearn (note that the C-proxy negatively correlated with memorization). Furthermore, Figure 4b and Figure 10 show the results of RUM across various datasets and model architectures, using the C-proxy in place of memorization for the refinement step described in Section [S] More detailed results, including accuracy and MIA, are available in Table 17 These results demonstrate that RUM still achieves significant unlearning performance gain when C-proxy is used, comparable to those obtained with the memorization score. A detailed analysis of various memorization proxies, their suitability, and their impact on RUM can found (37]
A.7 Per-example differences between the unlearned and retrained models
While accuracy is commonly used metric in the unlearning literature [29], it reflects average over a data distribution and may overlook finer details. To gain a more granular understanding of the similaritessddiferenees between the unlearned model and the retrained model, examine their behavior at the level of individual examples. Specifically, we collect predictions from both models for each example and count the examples where their predictions differ. Table Table[]ppreentt these results for a representative setting: CIFAR-10 10 dataset with ResNet-18 architecture.
We observe that, in all cases, the percentage of disagreements between the unlearned and retrained models remains low. For highly memorized data (which typically makes unlearning harder), the disagreement rate increases but stays modest, ranging between ca. [7%, 15%]. For memorizrd data, the disagreement rate is lower, within ca. [3%, 5%).
Interestingly, for the vast majority of unlearning algorithms, these per-example disagreement trends align with those observed using ToW: memmoriee examples are generally easier to unlearn, as indicated by both ToW and the lower disagreement rate, while megmommeeiized examples are harder to unlearn, with higher disagreement rates. similar trend is observed across different ES levels as well. This consistency between average accuracies (ToW) and per-example disagreements supports the robustness of our findings and underscores the validity of To W as measure of unlearning difficulty.
19

---

Table 5: ToW vs percentage of different predictions across ES or memorization levels on CIFAR- 10/ResNet- 18, averaged over 3 runs with 95% confidence intervals.
ToW
Percentage of different predictions (%)
Low ES
Medium ES
High ES
Low ES
Medium
High ES
Original
0.944 0.014
0.928
0.022
0.759
0.048
0.329
0.069
0.396 :: 0.048
1.514 0.212
Fine-tume
0.952 + 0.024
0.923
0.019
0.908 + 0.011
0.624 + 1.339
1.676 + 2.431
3.218 = 1.597
Ll-sparse
0.945 0.013
0.926
0.019
0.836
0.031
0.335 0.051
0.407 + 0.053
6.367 + 11.164
NegGrad
0.929 0.030
0.887
0,047
0.766
0.042
0.479
0.413
3,853 + 4.276
3.594 3.802
NegGrad+
0.941 = 0.029
0.920
0.038
0.800 = 0.029
0.370 + 0.022
1.7% + 1.510
1.752 = 0.174
Influence unlearning
0.928 0.023
0.890
0.012
0.744
0.029
0.362
0.208
0,986 +: 1.223
3,381 3,342
SulUn
0.783 0.031
0.656
0.046
0.618 0.056
8.816 1.756
1.752
+: 5.706
5.706 4.279
Random-labe
0.767 0.107
0.663
0.055
0.443 0.087 5.566 0.406
12.154 +: 6.952
0.579 0.579
(a) Per-example difference vs ES
ToW
Percentage of different predictions (%)
Low mem
Medium mem
High mem
Low mem
Medium mem
High mem
Original
0.988 0.007
0.723 + 0.053
0.231 + 0.058
3.611 + 0.856
5.081 +: 0.550
7.341 0.822
Fine-tune
0.933 0.052
0.884 = 0.019
0.760 1 0.065
5.962
1.525
6.359 1.135
7,099 2.370
Ll-spurse
0.914 0.061
0,816 = 0.011
0.629 + 0.087
6.668 + 1.676
10.163 +: 3.705
9,834 + 2,030
NegCirad
0.938 0,028
0.738 + 0.005
0.325 + 0.088
5.412 1.020
9.073 + 1.283
14.256 + 3.437
NegGrad+
0.965 0.020
0.831 0.032
0.661 + 0.082
4.288
0.452
6.357 1.023
12.190 = 2.720
Influence unlearning
0.986 0.031
0.738 + 0.051
0.381 + 0.037
3.964 + 0.803
9.668 + 2,469
18.484 + 1.491
SalUn
0.774 = 0.043
0.758 = 0.053
0.886 + 0.082
5,963
1.281
7.507 0.268
8.031 0.540
Random-lahy lahel
0.709 0.029
0.548 0.093
0.877 + 0.064
13.491
+ 5.363
12.973 + 1.635
12,905 +: 4.413
(h) Per-example difference vs memorization
A.8 Detailed results
In this section, we show the results that were used to construct all figures in the paper, as well as additional results and analyses.
Entanglement of forget and retain sets affects unlearning difficulty The first factor affecting unlearning difficulty, as discussed in Section 4.1 is the degree of entanglement between the forget and retain sets in the embedding space. Specifically, the more entangled these sets are, the harder it becomes to unlearn. We present detailed results on how this factor affects unlearning difficulty in 10 and Figure 11 6 the ToW results for different ES partitions across
Table 6 various settings, including CIFAR-10 10, CIFAR- 100 and Tiny-ImageNet datasets, as well as ResNet and VGG architectures. Additionally, comprehensive tata-covering forget, retain, and test accuracy along with MIA performance-ann are provided in Figure I and in Table 7 through Table 10 for these datasets and model architectures. All the results are averaged over 3 runs for each algorithm (6 runs for relabelling-basee based algorithms due to their higher variance), along with 95% confidence intervals.
Memorization of forget examples affects unlearning difficulty In Section 4.2 we discussed another critical factor affecting unlearning difficulty: the memorization level of the forget examples. Specifically, when a forget set consists of examples that are more highly memorized by the model, it becomes more difficult to unlearn (for most algorithms). The detailed results that support this observation are presented in Table Table 12 Table 13, and Figure 12
Table IT illustrates the ToW for forget sets with varying memorization levels. Additionally, Figure 12 and Table 12 to Table 13 provide extensive data on forget, retain, and test accuracy, as well as MTA performance. For all the experiments, the average results are reported with 95% confidence intervals over 6 runs for relabelling-based algorithms and 3 runs for others.
RUM experiments We present the details of our RUM experiment in this section. Table 15 presents the distribution of examples in the forget set for each class in the RUM experiments for both CIFAR- 10 and CIFAR-100 100 datasets. The size of the forget set is 3000 in all the experiments. For CIFAR-10. the table lists the number of forget set examples for each class. For CIFAR- 100, the table arranges the number of forget set examples in a 20x5 format for better readability. As shown in Table T5 the forget set covers examples from all classes in both CIFAR- 10 and CIFAR- 100 experiments.
20

---

Low E5
Moskum E5
High E5
Neve arouary
Nenue de arnuery
Negett
Tai
dearny
Tos ToLA
Parget
PS
(a) CIFAR-10 10 with ResNet-18
Low t5
Meckum
High E5
Betun Aoneery
Netuo de Arrweyy
(h) CIFAR-100 with ResNet-50
Figure 11: Forget, retain, test accuracy and MIA performance for forget retain partitions with varied ES, for CIFAR-I0 using ResNet- 18 and CIFAR-100 using ResNet-50. As the ES increases, there is notable decline forget accuracy and a significant rise in MIA performance. This trend indicates that the forget and retain sets become more entangled, more information from the forget set is effectively removed after unlearning.
Tables 16 provide detailed results when applying RUM to different unlearning algorithms for CIFAR- 10 using ResNet-18 18 and CIFAR-100 100 using ResNet-50. We selected the top-performing algorithms from previous experiments (i.e., Fine-tune, LI-sparse, NegGrad+, SalUn) and applied the refinement strategy to each algorithm in three ways: i) "vanilla", unlearming S in one go, ii)"shuffle" sequentially applying the algorithm on 3 equal-sized random subsets, serving as control experiment, RUMM" sequentially applying the algorithm on 3 subsets of S in low med - high memorization order. Each experiment was conducted 3 times, with average values and 95% confidence intervals reported. Our results indicate that RUM improves the performance of each unlearning algorithm, and the full RUM approach, which uses the best algorithm for each subset, achieves the overall best results.
Furthermore, to gain deeper understanding of the dynamics involved in applying RUM, we plotted the sequence dynamics for SalUn RUM on CIFAR- 10 (Figure and NegGrad+ RUM on CIFAR- 100 (Figure 13). These plots show the accuracy of the entire forget set, the retain set, the test set, and subsets of the forget set after each step. They demonstrate that while both orderings (low - med + high and high- -> med low memorization) yield similar ToW according to Table [6 their sequence dynamics during the unlearning phase are different. This phenomenon is discussed Section in the main paper, specifically in the Analysis of Sequence Dynamics paragraph.
21

---

Lowe memor cation
Medium memortbat.on on
High memorissiin
Baraa ArTAIIN
Bece Arnuary
Negest
dearny
PG
aa CIFAR-10 with ResNet-11
memorr cation Fetton
Medium memortaation
memortastion
Betun mumeer
Netuee Art Arrveyy
(b) CIFAR- 100 with ResNet-50
Figure 12: Forget, retain, test accuracy and MIA performance for forget / retain partitions with varying levels of memorization, for CIFAR-10 using ResNet- 18 and CIFAR-100 using ResNet-50. As the memorization level the forget sets increases, forget accuracy significantly decreases while MIA performance increases. This trend indicates that the forget examples become more memorized by the model, unlearning becomes more effective in removing the effect of these examples.
Adli come
Fuoel coley
Armyy dn
Aday
MAP
mant
surp tuerrrr
3
RUM sucuence
RUN sequeence
low med-
high (Tow: 0.92, MIA gap: 0.059) high med low (ToW: 0.93, MIA gap: 0.022)
Figure 13: Analysis of sequence dynamics for two different orderings. We apply NegGrad+ RUM on CIFAR-100 100 dataset using ResNet-50 and show the accuracy on overall forget set (and each of its subsets), the retain set and test set after each step in the RUM sequence.
22

---

Table 6: ToW for different unlearning algorithmms applied to forget retain sets with varying ES, evaluated on various datasets and model urchitectures. Across all algorithms, including the baseline without any unlearning performed (denoted as "Original"), we observe that ToW decreases from low to high ES partitions, indicating that unlearning becomes harder as the forget and retain sets become more entangled.
Low ES
Medium ES
High ES
Retrain
1,000 0.000
1.000 + 0.000
1.000 + 0.000
Original
0,944 0.014
0.928 + 0.022
0.759 0.048
Fine-tune
0.952 0.024
0.923 + 0.019
0.908 + 0.018
Ll-sparse
0.945 = 0.013
0.926 = 0.019
NegGrad
0.929 0.030
0.887 0,047
0.766
0.941 = 0.029
0.920 = 0.920 0.038
0.009
SCRUB
0,944 0.014
0.939 + 0.021
0.920 0.033
Influence Unlearning
0.928 = 0.023
0.890 + 0.012
0
SalUn
0.783 0.031
0.656 = 0.046
0.618
0.056
Random-labee
0.767 = 0.107
0.663 = 0.055
0.443 0.087
(a) CIFAR-10 with ResNet-18
Low ES
Medium ES
High ES
Retrain
1.000 0.000
1.000 + 0.000
1.000 0.000
Original
0.836 0.043
0.768 + 0.036
0.690 + 0.061
Fine-tune
0.878 0.014
0.852 + 0.006
0.773 0.029
Ll-sparse
0.867 0.039
0.803 = 0.025
0.709 +
NegGrad
0.755 = 0.027
0.707 = 0.039
0666
NegGrad+
0,922 0.055
0.844 0.014
0.766
SCRUB
0.842 = 0.089
0.831 = 0.068
4.811
Influence Unlearning
0.806 0.027
0.756 = 0.041
0.668 + 0.037
SalUn
0.835 0.037
0.704 + 0.028
0.678 0.042
Random-label
0.675 0.029
0.629 + 0.031
0.585 0.014
(b) CIFAR-100 with ResNet-50
Low ES
Medium ES
High ES
Retrain
1,000 0.000
1.000 + 0.000
1.000 + 0.000
Original
0.920 0.019
0.626 + 0.046
0.377 0.015
Fine-tune
0.919 0.015
0.770 = 0.039
0.765 0.022
LI-sparse
0.920 = 0.018
0.884 = 0.014
0.866:0.00
NegGrad
0,895 0.032
0.616 0.070
0 0.027
NegGrad+
0.921 = 0.021
0.914 0.041
0.021
Influence Unlearning
0.892 0.029
0.595 = 0.019 0.019
4442
SalUn
0.920 = 0.018
0.613 = 0.044
0.579 0.021
(c) Tiny- ImageNe with ResNet-18
Low ES
Medium ES
High ES
Retrain
1.000 0.000
1.000 + 0.000
1.000 0.000
Original
0.933 0.002
0.568 + 0.017
0.331 + 0.030
Fine-tune
0.936 0.003
0.749 + 0.055
0.677 0.087
Ll-sparse
0.936 0.006
0.763 = 0.043
0.737 + 0.026
NegGrad
0.919 = 0.019
0.569 + 0.037
0.379 0.016
NegGrad+
0,934 0.015
0.806 + 0.027
0.577
+ 0.181
Influence Unlearning
0.933 0.005
0.547 + 0.041
0.423
0.046
SalUn
0.934 0.001
0.554 = 0.010
0.411 + 0.118
(d) Tiny-ImageN with VGG-16
23

---

Table 7: Accuracy and MIA performance for different unlearning algorithms applied forget / retain sets with varying ES for CIFAR-I0 10 using BesNet-18.
Forget Ace
Retain Acc
Test Acc
MIA
Retrain
95.078 + 0.995
100.000 + 0.003
84.040 = 1.387
0.100 = 0.010
Original
100.000 + 0.000
100.000 + 0.000
84.353 = 3.455
0.000 0.000
Fine-tune
98.522 = 3.296
99.675 = 1.385
83.107 = 3.862
0.035 0.074
Li-sparse
100.000 = 0.000
99.994 = 0.022
= $
0.001 0.001
NegGrad
99.800 = 0.461
99.853 = 0.429
0.006 = 0.007
NegGrad+
99.500 = 0.647
99.988 0.051
0.007 0.007
SCRUB
99.978 + 0.096
100.000 0.000
84.280 + 3.505
0.002 0.001
Influence untearning
100.000 + 0.000
99.964 0.154
81.667 + 2.547
0.000 0.001
SalUn
88.400 + 6.843
91.442 + 2.438
75.833 + 0.725
0.696 0.172
Random-label
81.100 + 10.427
95.396 0.682
77.470 + 4.529
0.908 0.026
(a) Low
Forget Acc
Retain Acc
Test Acc
MIA
Retrain
94.067 + 0.722
100.000 + 0.000
84.167 + 1.243
0.147 + 0.009
Original
100.000 + 0.000
100.000 0.000
84.353 + 3.455
0,000 0.000
Fine-tune
96.678 + 3.525
98.655
2.575
80.210 3.004
0
L1-sparse
99.978 0.096
99.990
0.044
83.580 1:800:30 3.270
0.005
NegGrad
96.300 + 4.774
96.527 4.294
78.163 + 4.322
0.074
NegGrad+
93.200 4.241
98.907 1.839
78.580 4.404
0.087 +
SCRUB
98.756 + 1.273
100.000
0.000
84.180
+ 4.083
0.056
Influence unlearning
99.889 0.172
99.372
0.774
79.260
+ 0.774
0.007 0.005
SalUn
68.044 + 7.008
96.222 6.649
76.380
+ 6.843
0.739 + 0.270
Random-lad
73.922 + 3.527
92.244 7.643
74.157
= 10.427
0.607 + 0.316
(b) Medium ES
Forget Ace
Retain Acc
Test Acc
MIA
Retrain
77.300 + 3.130
100.000 + 0.003
83.653 + 2.280
0.562 + 0.026
Original
100.000 + 0.000
100.000 + 0.000
84.353 + 3.455
0.000 + 0.001
Fine-tune
81.289 + 2,298
98.371 1.395
79.763 + 1.587
0.375 + 0.066
LI-sparse
83.022 + 14.615
95.164 + 10.943
76.943 + 6.159
0.357 0.075
NegGrad
+ 9 3.705
97.925 3.977
80.927 = 6.576
0.066 + 0.052
NegGrad+
95.222 = 2.993
99,966 0.066
0.126 = 0.055
SCRUB
82.967 + 3.929
99.920 0.330
82.717 7.032
0.360 = 0.109
Influence unlearning
96.111 = 6.259
98.161 3.205
77.093 = 5.087
0.236 = 0.044
SalUn
40.722 + 11.043
99.967 = 0.080
81.250 = 3.335
0,951 0.077
Random-label
31.656 + 5.551
91.871 + 0.275
72.273 = 1.895
0.879 = 0.055
(c) High ES
24

---

Table Accuracy and MIA performance of different unlearning algorithms on forget / retain sets with varying ES for CIFAR- 100 using ResNet-50.
Forget Ace
Retain Acc
Test Acc
MIA
Retrain
84.222 + 3.734
99.968 + 0.012
75.493 + 0.756
0.296 + 0.041
Original
100.000 + 0.000
99.959 + 0.015
75.003 + 2.809
0.000 = 0.000
Fine-tune
90.044 = 8.075
99.148 1.639
69.577 5.366
0.231 = 0.067
Li-sparse
84.611 = 6.139
96.807 2.881
6 3.228
0.246 0.040
NegGrad
= #
66.793 3.820
0.118 = 0.022
NegGrad+
88.111 = 1.532
99.841 0.162
71.493 2.158
0.149 0.028
SCRUB
95.556 + 6.560
98.962 4.251
71.510 12.504
0.137 0.059
Influence untearning
99.722 + 0.981
99.202 + 2.531
71.610 + 5.558
0.007 + 0.020
SalUn
100.000 + 0.000
99.952 + 0.016
74.707 + 2.064
0.006 0.002
Random-label
89.267 + 1.159
85.992 + 0.557
58.190 + 1.640
0.130 0.023
(a) Low
Forget Ace
Retain Acc
Test Acc
MIA
Retrain
78. + 1.616
99.960 + 0.009
73.273 + 2.131
0.407 + 0.007
Original
100.000 + 0.000
99.959 + 0.015
75.003 + 2.809
0.001 = 0.001
Fine-tune
86.078 = 8.732
98.069 = 4.782
67.680 + 4.230
0.287 = 0.040
LI LI-sparse
89.511 + 4.761
96.656 + 2.561
66.983 + 3.893
0.195 + 0.025
NegGrad
88.289 + 1.800
87.242 + 2.187
63.380 3.313
0.189 0.018
NegGrad+
68.900 + 2.582
98.251 + 1.412
67.913 + 1.354
0.408 + 0.035
SCRUB
90.656 + 3.795
98.950 + 4.390
71.863
14.570
0.254 0.100
Influence untearning
98.978 + 1.962
98.613 2.268
70.093 + 0.981
0.035 0.042
SalUn
92.744 + 1.706
91.400 1.657
63.460 + 1.319
0.120
0.011
Random-label
80.622 + 0.751
76.779
+ 0.664
57.213 + 1.410
0.204 0.004
(b) Medium
Forget Acc
Retain Acc
Test Acc
MIA
Retrain
69.789 + 6.038
99.963 0.012
73.827 + 3.116
0.536 + 0.073
Original
99.989 0.048
99.960 0.018
75.003 2.809
0,009 0.007
Fine-tune
63.622 + 6.326
93.054
2.972
62.320
0.470
Ll-sparse
97.856 1.519
99.706 = 0.286
72.620 1.985
0.055
NegGrad
83.489 + 23.803
87.013 = 19.000
63.343
0.260 12.854
NegGrad+
52.944 5.967
98.491 0.732
67.257
0.649 0.091
SCRUB
91.000 + 0.865
99.962 0.010
74.703
0.341 + 0.038
Influence unlearning
84.133 8.032
88.398 5.947
62.170
0.242 0.046
SalUn
99.167 + 1.444
99.289 1.222
70.567 + 2.274
0.080 + 0.058
Random-label
76.756 4.633
78.256 4.378
54.160 1.832
0.250 0.041
(c) High ES
25

---

Table 9: Accuracy and MIA performance of different unlearning algorithms forget / retain sets with varying ES for Tiny-ImageNo using ResNet-18
Forget Acc
Retain Acc
Test Acc
MIA
Retrain
92.175 + 1.804
99,979 + 0.001
64.593 + 1.146
0.164 + 0,016
Original
100.000 = 0.000
99.979 0.000
64.773 + 1.027
0.001 = 0.001
Fine-tune
100.000 = 0.000
99.979 + 0.000
64.253
0.553
0.007 = 0.003
L1-sparse
100.000 + 0.000
99.979 + 0.000
64.393 0.596
0.001 + 0.002
NegGrad
99.600 0.166
98.858
0.319
62.386 + 1.957
0.011 + 0.004
NegGrad+
100.000
0.000
99.979
0.000
64.506 + 1.275
0.000 + 0.000
Influence unlearning
99.789
0.048
99.637 0.492
61.452 2.375
0.009 = 0.006
SalUn
100.000 = 0.000
99.979 + 0.000
64.513 + 1.446
0.000 + 0.000
(a) Low ES
Forget Acc
Retain Acc
Test Ace
MIA
Retrain
62.800 + 4.303
99.980 + 0.003
64.480 0.698
0.670 + 0.031
Original
99.978 = 0.048
99.980 + 0.001
64.773 1.027
0.009
Fine-tune
85.256 = 2.155
99.980 = 0.001
63.893 1.599
0.01
Ll-sparse
97.985 = 1.636
56.718
0.002
NegGrad
89.789 + 9.586
90.936 + 10.713
57.258 7.332
0.165 0.074
NegGrad+
68.878 + 2.883
99.977 + 0.007
61.792 + 0.516
0.495
+ 0.049
Influence unlearning
82.744 + 2.567
86.616 + 3.098
50.297 3.316
0.145 0.102
SalUn
87.344 3.792
86.758 4.405
58.045 0.977
0.689 + 0.166
(b) Medium ES
Forget Acc
Retain Acc
Test Ace
MIA
Retrain
37.822 + 1.112
99.986 + 0.006
64.313 + 0.553
0.891 + 0.006
Original
99.889 = 0.172
99.983 0.005
64.773 = 1.027
0.061 0.041
Fine-tune
60.889 = 1.574
99.983 + 0.003
63.799 = 1.499
0.934 + 0.008
LI-sparse
45.811 0.912
99.260 0.538
59.132 = 0.849
0.813 0.076
NegGrad
67.989 + 8.858
80.444 + 8.129
49.230 4.824
0.457 = 0.101
NegGrad+
49.933 + 2.191
99.982 0.006
61.432 = 0.928
0.843 0.025
Influence unlearning
78.678 + 13.145
86.881 + 11.277
50.703 = 5.029
0.640 = 0.091
SalUn
62.000 + 6.416
82.350 + 2.534
57.091 + 2.965
0.926 0.056
(c) High ES
26

---

Table 10: Accuracy and MIA performance of different unlearning algorithms on forget / retain sets with varying ES for Tiny-ImageNet using VGG-16.
Forget Acc
Retain Acc
Test Acc
MIA
Retrain
93,942 0.752
99.979 0.001
60.279 1.269
0.119 + 0.017
Original
100.000 = 0.000
99.979 + 0.001
60.479 = 0.938
0.001 = 0.003
Fine-tune
100.000 0.000
99.979 = 0.000
60.332 0.398
0.001
Ll-sparse
100.000 = 0.000
99.979 0.000
6
0.001 0,000
NegGraad
99.978 0.048
99.653 0.216
58.445 0.451
0.003 + 0.003
NegGrad+
100.000 + 0.000
99.979 + 0.001
59.899 0.748
0.001 + 0.002
Influence unlearning
99.989 + 0.048
99.979 + 0.001
60.412 1.172
0.001 + 0.001
SalUn
100.000 0.000
99.979 0.000
60.372 1.093
0.001 + 0.000
(a) Low ES
Forget Acc
Retain Acc
Test Acc
MIA
Retrain
56.978 + 2.110
99.980 0.003
60.139 + 0,445
0.662 + 0.043
Original
99.989 0.048
99.979 0.000
60.479 + 0.938
0.008 0.001
Fine-tune
57.133 + 5.043
87.325
4.327
47.283 1.898
0.464 0.033
Ll-sparse
61.633 3.340
89.064 0.687
49.983 0.875
0.436 0.049
NegGrad
88.200 = 20.298
90.202 = 20.598
52.604 12.745
0.167 0.189
NegGrad+
69.944 + 4.170
99.367 0.642
53.284 + 1.557
0.413 + 0.066
Influence unlearning
81.922 25.731
84.474 24.099
47.469 11.033
0.178 0.081
SalUn
78.122 + 6.099
78.968 4.343
49.116 + 0.508
0.251 + 0.089
(b) Medium ES
Forget Ace
Retain Ace
Test Acc
MIA
Retrain
33.289 2.942
99.982 0.004
59.859 + 0.959
0.880 0.039
Original
99.933 + 0.083
99.981 + 0.001
60.479 + 0.938
0.034 + 0.010
Fine-tune
30.089 + 5.122
81.706 5.794
45.336 + 2.339
0.709 0.055
LI-sparse
40.300 3.354
88.805 + 2.483
49.130 + 0.358
0.688 + 0.053
NegGrad
93.256 + 7.453
97.402 4.942
57.151 = 2.957
0.169 0.124
NegGrad+
73.489 + 23.035
99.802 + 0.736
56.711 = 3.155
0.513 = 0.406
Influence unlearning
82.933 + 4.683
92.497 1,794
50.690 + 0.650
0,233 0.249
SalUn
84.089 + 23.491
91.948 + 13.228
51.810 + 9.754
0.334 + 0.193
(c) High ES
27

---

Table 11:
ToW for different unlearning algorithms applied to forget sets with varying levels of memo- rization, for CIFAR-I0 using ResNet-18 and CIFAR- using ResNet-50. As the memorization level of the forget examples increases, the ToW significantly decreases for most algorithms, indicating that unlearning becomes harder when the forget examples are more memorized by the model.
Low memorization
Medium memorization
High memorization
Retrain
1.000 + 0.000
1.000 + 0.000
1,000 + 0.000
Original
0,988 0.007
0.723 + 0.053
0.231 0.058
Fine-tune
0.933 + 0.052
0.884 + 0.019
0.760 = 0.065
Ll-sparse
0,914 + 0.061
0.816 + 0.011
0.629 + 0.087
NegGrad
0.938 0.028
0.738 0.005
0.325 0.098
NegGrad+
0,965 + 0.020
0.831 + 0.032
0.661 + 0.082
SCRUB
0.988 0.010
0.923 0.033
0.780 0.101
Influence unlcarning
0,986 0.031
0.738 + 0.051
0.381 = 0.037
Salun
0.774 0.043
0.758 0.053
0.886 0.082
Random-labe
0.709 + 0.029
0.548 + 0.093
0.877 + 0.064
(a) CIFAR-IU 10 with ResNet-18
Low memorization
Medium memorization
High memorization
Retrain
1.000 0.000
1.000 0.000
1.000 0.000
Original
0,983 + 0.014
0.516 = 0.041
0.026 0.005
Fine-tune
0,974 0.064
0.830 = 0.052
0.768 = 0.025
L1-sparse
0.857 = 0.035
0.828 = 0.021
0.754 = 0.036
NegGrad
0.680 0.130
0.564 = 0.044
0.270 0.039
NegGrad+
0,986 0.024
0.917 + 0.040
0.889 0.012
SCRUB
0.982 0.017
0.761 + 0.051
0.594 0.118
Influence unlearning
0.960 + 0.112
0.556 + 0.044
0.154 0.081
SalUn
0.964 + 0.091
0,564 + 0.047
0.538 + 0.061
Random-label
0.770 0.105
0.548 + 0.022
0.409 0.044
(b) CIFAR- 100 with ResNet-50
28

---

Table 12: Accuracy and MIA results for various unlearning algorithms applied on forget retain sets with different memorization levels for CIFARIO using ResNet- 18.
Forget accuracy
Retain accuracy
Test accuracy
MIA
Retrain
99.711 + 0.253
100.000 + 0.000
84.280 + 1.184
0.010 + 0.006
Original 100.000 0.000
100.000 + 0.000
84.353 = 3,455
0.000 0.000
Fine-tune
99.156 + 1.634
98.390 = 1.731
79.587 = 3.173
0.020 = 0.024
LI-sparse
98.678 2.046
97.736 0.449
78.750 = 4.659
0.039 0.047
NegGrad
99.178 = 1.574
99.241 + 1.512
79.390 = 1.904
0.030 0.033
NegGrad+
98.300 = 1.490
99.944 0.142
82.257 3.363
0020=0000
SCRUB
99.911 = 0.126
100.000 + 0.000
= 3.794
1.000
Influence unlearning
100.000 0.000
100.000 0.000
83.133 4.128
0.002 0.004
SalUn
81.278 = 5.906
99.996 1 0.003
79.223 = 5.991
0,960 = 0.053
Random-label
86.667 + 6.902
90.604 = 7.740
74.350 = 1.287
0.611 0.089
(a) Low memorization
Forget accuracy
Retain accuracy
Test accuracy
MIA
Retrain
73.611 + 3.478
100.000 + 0.000
82.977 +: 2.060
0.654 + 0.067
Original
100.000 0.000
100.000 + 0.000
84.353 + 3.455
0.000 0.000
Fine-tune
82.856 + 1.706
99.422 + 0.721
80.957 + 2.376
0.413 = 0.117
LI-sparse
82.922 = 5.971
95.660 5.166
77.130 = 4.536
0.368 0.013
NegGrad
92.900 = 10.384
96.755 = 6.091
77.593 = 5.946
0.164 = 0.130
NegGrad+
88.122 = 5.826
99.962 0.072
80.303 5.202
0.193 0.060
SCRUB
69.411 = 18.532
99.963 0.127
1:887:40 81.787 4.574
0.582 = 0.140
Influence unlearning
91.800 + 13.743
96.267 6.256
7
0.217 0.099
SalUn
51.100 + 7.729
100.000 + 0.000
80.847 = 3.332
0.965 - 0.012
Random-labe
36.233 + 6.904
94.675 2.471
75.373 # 2.546
0.829 0.076
(b) Medium memorization
Forget accuracy
Retain accuracy
Test accuracy
MIA
Retrain
23.444 + 5.753
100.000 + 0.000)
82.967 + 1.910
0.961 + 0.018
Original
100.000 0.000
100.000 + 0.000
84.353 + 3.455
0.002 0.001
Fine-tune
45.822 + 10.494
98.987 + 4.112
82.003 + 4.700
0.820 + 0.135
LI-sparse
55.644 = 16.226
97.001 1.941
0.714 0.075
NegGrad
= + 15.675
93.289 + 10.807
74.827 = 7.390
0.266 = 0.179
NegGrad+
4
75.783 + 4.310
0.641 0.107
SCRUB
44.167 11.976
99.916
82.317 1 5.227
0.840 = 0.083
Influence unlearning
74.567 3.934
88.536 2,176
71.143 + 3.499
0.417 0.067
SalUn
32.311 5.235
99.560 + 0.597
80.647
+ 3.328
0.951 + 0.008
Random-label
22.700 + 3.360
94.728 4.453
76.323 + 2.830
0.936 0.025
(c) High memorization
29

---

Table 13: Accuracy and MIA performance for different unlearning algorithms applied on forget /
retain sets of varying levels of memorization for CIFARI00 using ResNet-50.
Forget accuracy
Retain accuracy
Test accuracy
MIA
Retrain
99.878 0.096
99.960 0.003
74.077 +: 2.112
0.016 + 0.005
Original
100.000 + 0.000
99.959
0.015
0.001 + 0.001
Fine-tune
99.878 = 0.266
99.806 1:6::000 0.273
7
0.018
LI-sparse
98,100 0.299
65.550
0.040
NegGrad
90.589 + 2.987
85.454 6.778
61.637
0.010
NegGrad+
99.556 0.345
99.951 0.018
73.823
0.014
SCRUB
99.933 = 0.143
+
75.127 3.093
0.006
Influence unlearning
99.867 0.430
09.077
0.006
Salun
99.922 + 0.191
99.225 1.600
71.227 + 6.693
0.008 + 0.008
Random-label
98. 144 1.559
88.833 5.531
62.207 4.559
0.030 0.011
(a) Low memorization
Forget accuracy
Retain accuracy
Test accuracy
MIA
Retrain
52.622 4.204
99.976 0.006
72.767 2.651
0.906 0.015
Original
99.800 + 0.000
99.973 0.015
75.003 + 2,809
0.020 + 0.007
Fine-tune
60.178 = 14.403
96.944
5.329
65.433 7.219
0.663
Ll-sparse
56.567 3.067
94.525 = 3.186
64.013 1.340
0.025
NegGrad
82.000 4.362
87.858 4.2005
63.673
0.062
NegGrad+
54.267 14.532
99.566 0.704
49.440+:..255
SCRUB
74.511 + 9.443
99.404 2.483
72.787 8.967
0.036
Influence unlearning
84.667 15.692
90.643
12.409
63.330
11.045
0.220
Salun
79.889 + 8:212
85,875 4.268
63.147 1.900
0.646
Random-labe
68.022 15.101
77.440 12.416
56.787 6.257
0.354
(b) Medium memorization
Forget accuracy
Retain accuracy
Test accuracy
MIA
Retrain
2.556 + 0.669
99.972 + 0.009
73.580 + 0.661
1.000 + 0.001
Original
99,900 0.166
99.966 0.003
75.003 2.809
0.045 + 0.020
Fine-tune
12.300 = 2.010
94.729 + 1.488
63.413 3.273
0.947 0,003
LI-sparse
15.244 + 2.825
94.814 4.718
64.707 + 2.808
- 0.940 0.023
NegGrad
62.578 + 14.734
80.621 9.071
58.027 9.041
0.493 + 0.096
NegGrad+
5.567 = 2.654
2.044
0.978 17780000 0.005
SCRUB
40.900 20.728
99.086
71.220
0.899 0,030
Influence unlearning
85.389 9.154
95.745
67.240
0.331 1 0.170
Salun
33.556 + 6.167
88.021 + 2.044
62.127 2.447
0.997 + 0.004
Random-label
39.778 13.921
78.972 8.172
56.333 + 3.225
0.645 + 0.101
(c) High memorization
30

---

Table 14: Unlearning performance evaluated by ToW-MIA, with ToW results included for comparison. Subtables (a, b): TOW-MIA for different unlearning algorithms applied to forget retain sets with varying ES or memorization levels on CIFAR- 10 using ResNet- 18, with ToW results from Tables 6 and for comparison. Subtable (c): RUM and RUM results evaluated by ToW-MIA on CIFAR- 10
using ResNet-18, 18, with ToW results from Table T6 for comparison.
ToW-MIA
ToW
Low ES
Medium ES
High ES
Low
Medium
High
Retrain
1.000 0.000
1.000 0.000
1.000 0,000
1,000 + 0.000
1,000
0.000
1.000 +: 0.000
Original
11,893 0.014
0.841 + 0.026
0.430 + 0.035
0.944 + 0.014
0.928 + 0.022
0,759 + 0.048
Fine-tune
0,921 0.061
0.882 0.020
0.768 0.055
0,952 - 0.024
0.923 = 0.019
0,908 + 0.018
L. L1-sparse
0.895 0.014
0.845 0.031
0.706 0.103
0.945 + 0.013
0.926 0.019
0.836 + 0.031
NegGrad
0,883 + 0.030
0,841 + 0.044
0.479 = 0.010
0.929 0.030
0.887 = 0.047
0,766 + 0.042
NegGrad+
0.893 0.026
0.877 0.057
0,550 0.055
0,941 0.029
0.920 0.038
0,800 + 0.029
SCRUB
0.895 0.014
0.884 0.045
0.778 0.075
0.944 0.014
0.999
0.021
0,920 + 0.033
Influence unlearinn
0.879 + 0.022
0,812 + 0.020
0.618 = 0.010
0.928 + 0.023
0.890
0.012
0.744 + 0.029
0.359
Random-label
0.171 0.037
0.446 + 0.219
0.557 0.086
0.767 + 0.107
0.663 0.055
0.443 + 0.087
(a) ToW-MIA ES
ToW-MIA
ToW
Low
Medium
High mem
Low
Medium
High
Retrain
1.000 + 0.000
1,000 + 0.000
1,000 = 0.000
1,000 + 0.000
1.000 = 0.000
1.000 +: 0.000
Original
0.973 0.037
0.340 + 0.072
0.041 0.017
0,988 + 0.007
0.723 0.053
0.231 + 0.058
Fine-tune
0.935 0.032
0,842 0.037
0.738 0.068
0,933 0.052
0.884 0.019
0,760 1: 0.065
Ll-sparse
0.904 + 0.098
0.700 0.018
0.642 = 0.025
0.914 0.061
0.816 = 0.011
0.629 + 0.087
NegGrad
0.933 0.054
0,466 + 0.024
0.258 0.105
0,938 1 0.028
0.738 = 0.005
0.325 + 0.098
NegGrad+
0.973 0.028
0.523 0.037
0.600 0.123
0.965 0.020
0,831 0.032
0.661 = 0.082
SCRUB
11.979 + 0.011
0.913 + 0.158
0.865 0.050
0.988 + 0.010
0.93 0.033
0,780 + 0.101
Influence unlearning
0,973 0.030
0,507 0.046
0.357 0.061
0,986 - 0.031
0.738 = 0.051
0,381 + 0.037
SalUn
0.054 0.030
0.674
0.963 0.024
0.774 0.043
0.758
0.053
0,886 0.082
Random-label
11328 0.394
0.722 + 0.119
0.862 0.084
0,709 + 0.029
0.548
0.093
0,877 + 0.064
(b) ToW-MIA memorization
ToW-MIA
ToW
Retrain
1.000 + 0.000
1.000 = 1.000
Fine-tune Fine-tune shuffle
0,820 + 0.042
0.849 0.030
0.848 1 0.125
0.712 0.040
Fine-tune RUM
0.888 0.022
0.937 0.052
SalUn
0.602 + 0.019
0.731 = 0.070
SalUn shuffle
0.677 + 0.023
0.727 = 0.030
SalUn RUM
0.878 0.049
0.887 0.069
NegGrad+
0.707 + 0.033
0,802 + 0.028
NegGrad+ shuffle
0.470 +: 0.027
0.632 0.022
NegGrad+ RUM
0.779 0.025
0.879 0.068
Ll-sparse
0.772 + 0.014
0.794 - 0.035
Ll-sparse shuffle
0.624 + 0.039
0.716 = 0.023
Ll-sparse RUM
0.855 + 0.023
0.900 0.020
Nothing->Fine-tune Saateettuee+Sallll
0.950 0.062
0.965 0.014
(c) ToW-MIA vs RUM
31

---

Table 15: Class distribution of the forget set in the RUM experiment for CIFAR-10 and CIFAR-100. with 3000 examples in the forget set for each dataset. The forget set includes examples from all classes in both datasets.
Class
3
4
7
Count 291
259
363
314
277
316
298
309
306
267
(a) CIFAR-10
Class
5
6
7
10
11
12
13
14
15
16
17
Count
23
24
34
29
34
25
25
40
17
43
34
30
27
21
25
30
25
32
37
26
Class
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
Count
31
22.
29
30
39
37
48
30
33
30
25
27
45
40
31
23
28
30
32
31
Class
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
Count
32
28
26
28
32
34
22
34
20
41
29
40
Class
60
61
62
63
64
65
66
67
68
69
71
72
73
74
75
76
77
79
Count
31
32
25
35
26
53
33
Class
85
86
93
94
95
Count
24
22
27
24
28
21
27
34
32
27
22
38
32
39
43
30
41
28
24
25
(b) CIFAR-100 100
32

---

Table 16: RUM results on CIFAR-10 and CIFAR-100 Results obtained by applying RUM with different algorithms, according to ToW (higher is better), its constituent ingredients, and MIA (lower better for MIA gap). The top section compares applying an unlearning algorithm // in three ways: i) in one-go, as usual (e.g. Fine-tune), ii) on three mundomly-deteriinet equal-sized subsets of S, sequentially (e.g. Fine-tune shuffle), and iii) on three equal-sized buckets obtained by refinement. F(S) according to memorization scores, in low -> med + high order (e.g. Fine-tune RUM The middle section of Table 66a further experiments with picking a different unlearning algorithm for each subset of F(S). Here A - B - C denotes applying algorithm A the first subset, B on the second subset, and on the third subset, where the subsets appear in low + medium + high order.
The bottom section shows different orderings.
ToW (f)
Forget accuracy
Retain accracy
Test accracy
MIA
MIA gap (J)
Retrain
1.00010.000
64.156:2..32
100.00010.000
83.917.120000
0.54911.020
0.000
Fine-tune Finc-tune shuffle
0.84910.030
73.067:14.064
97.923:14.664
79.277:7.487
0.42910.086
0,120
0.712=0.040
08.389=9.000 089
96.578+8.717
82.05748.500
0.451#0.037
0.098
Fine-tune RUM
0.937=0.052
69.13326.583
99.66410.175
83,230:2.221
0.45010.033
0,099
L.I-spare
0.794-0.005
79.300+-11 4467
98. 187-42.666
79.373-+6.872
0.374++0.097
0.175
Ll Ll-sparse shuffle
0.716100.23
38.244.15.217
97.152.11.462
81,047.1.3378
10.077
0.338
L.I-sparse RUM
0.900-000220
69:967-6.014
98.458-+ 1,399
81.047+3.377
0.4774-00039
0.072
NegGrad+
0.802+-0.028
76.867--7 7.382
98,415+007783
77.337++4.664
0.319+-0.093
0,230
NegGrad+ shuflle
0.632.0..02
99.600.10.933
99,979
83.700.15.336
0.009
0.520
NegGrade RUM
0.879+000068
61. 61.744-2.000 2.097
96.655+-2.274
77,063++ 1,991
0.415++00027
0,134
SaltIn
0.731
40 40.056+00. -0.39
99.796+004299
80.370-+4.159
0.923++0.067
0.374
SalUnn shufile
0.72710.030
81 88911
94.69312.866
77.270.1.3990
0.315110019
0,234
SaltIn RUM?
0.887+0.069
62.878+-3.1 8.777-837726
96.457 +-2482
77,857-33.
0.518++0.060
0.031
nothing -> Fine-lume + SulUn
0.965.10:014
66.011115.119 1319
99.205 1 1962
83.007.1 941
0.51510.019
0.034
nothing Sallin Fine-tune
19110-00010
00.3222--69115
29.0770+2.281
81,457++66 968
0.440+000047
0.109
mothing - SalUn + SalUn
0.91910.059
70.733.14.310
100.00010.000
84.190.33.328
0.644
0.095
Fine-tune RUM (low + med - high)
0.937-0.052
69. 10.133--6.500 583
99.6644-0.175
83.230--2.221
0.450+00003
0.099
Fine tune RUM (high med- low)
0.942.0.032
68.222.11.18
99.51512.009
83.587.12.22
0.47810.120
0.071
SalUn RUM (low - med + high)
0.88710.069
62.878:337726
96.457:12.482
27.857+33.280
0.518:10000
0.031
SalUn RUM (high med low)
0.881 08.120
14,64==1.615
99.921=0.124
82.393+3.018
0.328=00011
0,221
(a) RUM results on CIFAR- 10 using ResNet-18
ToW (?)
Forget accuracy
Retain aceracy
Test accracy
MIA
svg. MIA gap (1)
Retrain
1.000
52.0644:2.160
99.966.10.018
23,260:12.158
0.635_00005
0.000
NegGrad=
0.861=00069
63 64442440088
99.641 316
70 9.14
0.47==0.019
0.159
NegGrad- shuffle
0.61310.051
88.011144.228
97.99111.399
70.867.12.836
0.218-0.074
NegGrad- RUM*
0.921200.04
50.789=7.713
98-413-10.855
68. 3.6633333633
0,576=0.075
0.059
Ll-spurse
0.82410.011
54.078:13.275
93.367
63.287:42.388
0.546=0.028
0.089
1.1-sparse shuffile
0.604+00023
82. 11++222885
95.310+00877
63.880+2.3325
0.282-0.014
0.355
L1-spanse RUM
0.883:0.046
52.444 1.628
97.269 1,393
64.317:444.9
0.602=0.007
0.033
Finc-tunc
0.73420.025
76.4002-4.44
99.575::0.766
70. 70.7400::.422
0.496=0.156
Fine tune shufßle
0.58910.033
B1. $1.689.15.886 56860
93.616.22.316
2.3553333413
0.290
0.345
Finc-tune RUM-
0.78420.000
61.767=10.301
96. 3,399
33,090:13.575
0.512=00030
0.093
SalUIn
0.545=0.061
28.967=28.000
94.207 19:409
66.327=10.112
0:259-0.052
SalUin shuffle
0,538=0.019
63 389-
75 3.208
53.713:22.518
0.398=0.022
0.257
Saltin RUM"
0.614+00.03
58 1.5581
79.719+ 1.225
3.887
0.454-0.008
0.181
NegGrad- RUM (low med high)
0.901
50.789=7,713
98.413-20.855
68. 3.0:33.633
0.576=0.075
0.059
NegGrad- RUM fhigh - med low)
0.929100008
49.90
99.70311..340
31.08001:3144
0.657_0.040
1.1 11-sparse RUM (low > med > high)
0.883+00.046
52 1.44+1.628
97. 97.269++
64. 31. (4.317-+4.999
0.602-0.007
0.033
Ll L1-sparsc RUM (high med- kow)
0.908=00003
53.967:-2.587
98. 08.732:+1.376
66 2619663:22814
0.591=0.020
0.044
(b) RUM results on CIFAR- using ResNet-50
33

---

Table 17: RUM results using C-proxy for the refinement step, evaluated by ToW on CIFAR- 10 and Tiny-Iman ImageNet datasets. Each algorithm is applied in three ways: RUM as well as vanilla and shuffle comparison. Results averaged over 3 runs with 95% confidence intervals.
ToW (f)
Forget accuracy
Retain accracy
Test accracy
MIA
MIA gap ())
Retrain
1.000+0.000
50.433-+6.800
100.000=00000
84.167411.66
0.637++0.022
0.000
Fine-tume
0.701110.021
77 77.389.1..229 2.29
97 17,44.11..26.
81.847.11
0.286.10.066
0.351
Fine-tune shufile
0.829:10.022
62.800=16.311
98. 98.060:15.309
80.677:15.082
0,560.10.09
0,077
Fine-tune RUM
0.927:10.047
58 1.723
08.920:12.484
82.007:1:3 186
0.482:10.004
0.155
Ll-sparse
0.754=0.071
66.856+9.044
96.095:+4.677
78. 78.2000::5.17
0.492:40.044
0.145
LI-sparse shuffie
0.618+00.95
83.222++ 1,391
+.88900-44400
79. 79.963++-5117
0.256+000031
0.381
L.-sparse RUM
0.907+00.02
53.61 1-3.446
96.947+11. 1.991
80.783+2.540
0.002
0.071
NegGrad+
0.880.10.039
56.067.1.5.20
99.037.11.553
78.443.15.531
0,504 0.99
0.133
NegGrad+ shuffle
0.529:10.071
98.919:12.12 120
80.563-16.630
0.110:10.100 107
0.527
NegGrad+ RUM
0.893++0.026
58 18.000+-..725
98.920++2.484.
32.007++3.188
0.4024
0.155
SalUn
0.859110.022
58.11111.816
98.030.11.335
19..18014.300
0.593110.014
0.044
SalUn shuffle
0.638+00.07
44.069+5.009
97.897404449
82.387+3..331
0.381+00.015
0,256
SalUn RUM
0.737=00.000
75.578:14.606
99.996:10.009
82.597=144474
0.949:10.049
0,312
SCRUB vanilla
0.802++0.047
71.733+-2.630
99.671++0.008
82.077++4.319
0.462++0.031
0.175
SCRUB shuffle
0.732-+0.059
74.678 11677-82000 20.089
95.617+ 11 11.622
00.9833+8.00
0.3
0.312
SCRUB RUM
0.945:10.017
50.589:::.526
99.375:10.560
82.48713.2277
0.500+10.003
0.137
(a) ToW vs RUM on CIFAR-10 10 with ResNet-18
ToW (1)
Forget accuracy
Retain accracy
Test accracy
MIA
MIA gap (L)
Retrain
1.000=0.000
45.6334:2.521
99,999-100001
64.606:10.230
0.662410.005
0.000
Fine-tunc
0.777=0.020
52.356 9.99
91.768+2.399
55.438+t
0.530++-0.02
0.132
Fine-tune shuffle
0.566-0.021
80.622++0.981
195
66.9771-112999
0,276++-0.022
0.386
Fine-tune RUM
0,803=0.025
51.578:3.275
93.174:1..00
16.185511.393
0.554:10.032
0.108
LI-sparse
0.772=0.017
51.622+-2.34
89.556+: 1. 1,293
56.258+H0.96
0.534++0.025
0.128
L1-sparse shuffle
0.584-0.025
82.478+00. 0.112
0..426+0.6299
58 49.98 +- 1.567
0.244++00.08
0.418
L1-sparse RUM
0.883=0.032
55.722+00485
99.985+00058
62.833++0.255
0.660++0.007
0.002
NegGrad+
0.859-0.066
11122+111.999
97.359+ .060
57. 57.992++1.. 1.441
0.677++0.
0.015
NegGrad+ shuffle
0.482:00004
85.433-1:11995
93.073-117.340
$1.710 18.821
L.6664
0.094
NegGrad+ RUM
0.935=0.040
46.944+-4.332
99.6666+00056
59.952++ 194
0.621++0.019
0.041
SalUn
0.689-0.008
67.178+22.077
19.889+-08888
62.319++0.748
0.556++0.209
0.106
SalUn shuflle
0.553=0.032
65.333-155584
80.933.17.699 7,69
49.750 141
0.397-10.030
0.265
SalUn RUM
0.736=0.027
54.867+33.588
86.485+2.3009
+3.1513228853
0.560++0.024
0.102
(b) ToW RUM on Tiay-ImageNet with ResNet-18
ToW (1)
Forget accuracy
Retain accracy
Test accracy
MIA
MIA gap (L)
Retrain
1.000=0.000
19.10001..910
99.99510.003
60.699
0.637:10.013
0.000
NegGrad+
0.637=0.106
79.356=31.411
97. 97.146::11.966
55.6781 12.03 037
0.30111:0.15
0.336
NegGrad+ shuffic
0.514=0.009
78.078=#22.14
86.4044429.465
47. 47.103==25.0 866
0.376:+-0.59
0.261
NegGrad+ RUM
0.812=00053
57.233-19.692
97.515:19.221
51.430:16.92
0.477.10.062
0.160
Ll-sparse
0.771=0.030
53.167+-2.384
89.907#1.1114
50.117+:1.00 1.047
0.500-10.031
0.137
LI Ll-sparse shuffic
0.589=0.028
83.778=10.52
96.793=0.1
53.924 1024
0.217+00.025
0.420
Ll LI-sparse RUM
0.885-0.024
1.3178
99.932.100080
56.471:12.238
0.585110.04
0.052
Fine-tunc
0.750=0.013
1..
87.21442.133
1.
0.52810.016
0.109
Fine-tune shuffic
0.576=0.061
11.978=11.7400
81.317+8,350
46.023++4.367
0.360++0.1 160
0.277
Fine-tune RUM
0.767-0.044
49.989+2.632
90,01013.150
46.89662.862
0.5274:0.024
0.110
SalUn
0.602=0.041
85.522+2.569
98.336+0.254
56.945+1.357
0.32240.035
0.315
SalUn shuffle
0.566=0.011
72.622+43.082
84.977+36.200
15.871
0.313++0.380
0.324
SalUn RUM
0.679=0.025
54.62245.022
82.377+2.398
47.9 47.990+1..44
0.474:10.136
0.163
(c) ToW vs RUM on Tiny-ImageNet with VGG-16
34

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately outline the paper's s contributions, aligning well with the scope and findings presented in the study (we study interpretable factors affecting unlearning difficulty, uncover previously-unknowe behaviours of state-of-the ofrt algorithms and propose framework that leads to improvements of unlearning performance).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes a dedicated "Discussion and conclusion" section that discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
35

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
This paper does not include any theoretical results, hence no assumptions or proofs are required.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides detailed descriptions of the experimental setup, including the datasets used, model architectures, hyperparameters, and evaluation metries. This ensures that the main experimental results can be reproduced accurately, supporting the paper's claims and conclusions.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
36

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have included a GitHub repository containing our code and detailed instructions to enable other researchers to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guider wwt.//eumiisssioonooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
The experimental setting and details described in Section in the main paper and in Section A.2
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Docs the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports the results with 95% confidence intervals and provides error bars, which appropriately reflect the statistical significance and variability of the experimental findings. This ensures that the reliability and significance of the results are clearly communicated.
37

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper provides detailed information on the computational resources used for the experiments A.2 This ensures that other researchers accurately reproduce the experiments.
Justification:
Guidelines:
The answer means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research adheres the NeurIPS Code of Ethics, ensuring compliance with ethical standards in terms of data usage, experiment conduct, and the overall research process. Anonymity and confidentiality have been maintained where required, and all ethical considerations have been duly addressed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
38

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this is Section A.1
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deploymentn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no significant risks for misuse of data or models. Therefore, specific safeguards are not necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
39

---

The paper properly credits the creators and original owners of all assets used, including code, data, and models. The licenses and terms of use are explicitly mentioned and adhered to, ensuring compliance with the original creators' requirements.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papesssippseeeeeeee thcode. coodsattett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
40

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
41


Reviews:
[{"review_id": "elm2yC7VRV", "reviewer": "Reviewer_Bp7U", "summary": "This paper investigates the characteristics of forget data for machine unlearning algorithms and demonstrates their impact on the effectiveness of various unlearning methods. The paper further proposes refining the forget set into subsets based on data characteristics and selecting suitable unlearning algorithms for each subset. The experimental results show that this ensemble method outperforms single baselines.", "questions": "* What is the model accuracy before unlearning in experiments?\n* What's the criteria to group samples into low/medium/high partitions?", "limitations": "Authors discussed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The identification of properties in forgetting data is valuable for machine unlearning. The idea of adapting algorithms to the properties of specific data could benefit future algorithm design.\n* The proposed refine-based method is simple yet effective, as demonstrated by the experiments.", "weaknesses": "* Despite the significance of the insights about sample embeddings, they appear somewhat limited to classification models, which typically have an embedding space to group different samples. While this may be beyond the scope of this paper, it would be valuable to explore extending these investigations to generative models, such as diffusion models and transformer-based large language models (LLMs).\n* The experiments were conducted only on two CIFAR datasets, both involving image classification tasks. Including other tasks or datasets, such as text sentiment or topic classification, would make the paper more comprehensive.\n* Although the paper claims the RUM framework is a meta-algorithm framework, the selection of unlearning algorithms for each subset is now based on empirical investigation and human-designed heuristic, which limits the generalizability of this method.\n\nminor: I would recommend refining Figure 2, as some text is misaligned and mixed with the box boundaries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Re: limited to classification models.** We agree it is important to study different modalities, discriminative and generative models. However, these settings differ in three key ways.\n\n1) Unlearning definitions: The problem we study is removing the influence of a particular subset of training data (the “forget set”) from the model; success is defined as matching the distribution of retraining from scratch without the forget set. In generative models, unlearning can take different forms: i) remove a ‘concept’ (e.g. nudity, in diffusion models; Fan et al), or harmful content in LLMs, or ii) in LLMs, unlearn memorized text. In both i and ii, the goal is not matching retrain-from-scratch. For case i, the “forget set” may not even contain training images, it may be generated instantiations of unwanted content, and application-dependent metrics are used. Though ii is closer to our problem, computing retrain-from-scratch is not practical in large LLMs (none of the works use that as a reference point) and other definitions or metrics are used.\n\n2) Memorization definitions. According to the definition of label memorization (see equation 1), rare or atypical examples are more highly memorized, as we discussed. Instead, in LLMs, memorization is typically defined as “verbatim memorization” (e.g. see Definition 2 in [1]), capturing the model’s ability to regurgitate training data. Recent works (e.g. [1]) find that the more common a string is (i.e. less rare), the more likely it is to get memorized. This definition is what SOTA methods for memorized-data unlearning [2,3] use (case ii in the above). These notions are clearly distinct (and even seemingly at odds with one another) and an analysis of unlearning difficulty for verbatim memorization would thus require specialized treatment.\n\n3) SOTA methods. For example, the SOTA models for unlearning memorized data from LLMs are based on “simple” Gradient Ascent (GA) (see [2,3]). On the contrary, in computer vision / image classification tasks, (variations of) GA have been shown to be lacking badly. In particular, model utility for image classification suffers largely with GA (and variations thereof), whereas for text generation and reasoning, model utility can remain high with GA (as shown in for example [2,3].\n\nReferences: [1] Carlini et al, 2021 “extracting training data from large language models”, [2] Jang et al. 2023. “Knowledge unlearning for mitigating privacy risks [...]”, [3] Barbulescu et al. 2024. “To each (textual sequence) its own [...]”.\n\nIn summary, unlearning in generative models differs fundamentally. Hence, as the reviewer states, a comprehensive analysis is (well) beyond the scope of this paper. Our work is an important step in advancing knowledge and in-depth understanding of unlearning for image classification.\n\n**Re: only CIFAR and image classification.**\nWe have added a new dataset and architecture based on this feedback, please see the “common response” for the details. All conclusions remain consistent, supporting the generalizability of our findings. We hope this addresses the reviewer’s concern.\n\nPlease also note our answer to the above point for the comment on text generation and reasoning. We leave an investigation of tasks in other modalities and generative models for future work for those reasons.\n\n**Re: “human-designed heuristic [...] limits the generalizability of this method.”**\nThank you for raising this, we believe this is a very important point.\n\nFirst, our primary goal is a deeper understanding of behaviours of unlearning methods on different forget sets.  Crucially, our work shows the potential of such insights leading to better unlearning pipelines: each SOTA algorithm is shown to be improved upon using RUM. And, RUM using different algorithms for different partitions further improves performance. This is already a key contribution that can inform better generalizable approaches in the future.\n\nOf course, repeating our entire analysis on each new dataset or problem setting in order to plug in the “hand-picked” design decisions when using RUM is undesirable. Due to this, we designed our analysis to uncover key factors affecting algorithm behaviours (e.g. how algorithms interact with memorization) that transfer across datasets / architectures (e.g. as long as one knows the memorization scores). Since computing memorization scores for new training datasets is expensive, we show in our new experiments results with a cheap proxy (the “C-proxy”); see the “common response”. Hence, RUM is generalizable / practical: First, uncover key characteristics, for instance, memorization levels (or a cheaper proxy), and then plug them in to RUM to get big performance gains.\n\n**Re: minor**: thank you, we will fix these.\n \n**Re: “What is the model accuracy before unlearning in experiments”** – we denote this by “Original” in each experiment. The test accuracy values are 84.353±3.455% for CIFAR-10/ResNet-18 and 75.003±2.809% for CIFAR-100/ResNet-50, with 100% training accuracy for both settings. More detailed accuracy results can be found in the appendix: Table 5,6 for unlearning difficulty vs ES experiments, and Table 8,9 for unlearning difficulty vs memorization experiments.\n\n**Re: “What is the criteria to group samples into low/medium/high partitions”**. First, we sort in increasing order all examples in the dataset based on the metric of interest (memorization score or embedding distance). Then, say the forget-set size is N. We select the first (lowest value) N/3 examples to be the ‘low partition’, the last N/3 examples to be the ‘high partition’ and the middle N/3 to be the ‘medium partition’. \n\n**Overall**, we thank the reviewer for the thoughtful critical feedback. We believe the new results (with C-proxy and additional dataset / architecture) and discussion, address the reviewers central concerns. We would love to hear the reviewer’s thoughts and discuss further if there are remaining concerns. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qmgf8eBNZA", "reviewer": "Reviewer_3Cxb", "summary": "This paper studies how two properties of “forget sets” make machine unlearning hard: similarity/entanglement between the forget set and an accompanying retain set, and a memorization score for the forget set. The authors hypothesize and experimentally confirm that (1) high entanglement makes unlearning harder, likely due to it being hard to learn a model that forgets data that is so close to retain data, and that (2) more strongly memorized data is harder to forget, because models that memorize such data tend to be quite different from models than models that never see that data. Building on these observations, the authors propose a new algorithm for unlearning that first rank-orders forget data in terms of memorization scores, then selects an unlearning algorithm from a pool of algorithms that achieves the best ToW scores on data with the relevant level of memorization (as determined by previous analysis experiments). Results suggest that sequential forgetting based on forget set memorization achieves immediate improvements over whole-set forgetting, and that leveraging unlearning algorithms with good known ToW scores also improves results.", "questions": "- Since the retain set is like training data, do you think it would be better to exclude it from ToW? \n- Entanglement score: there might be variance in a direction that seems irrelevant to whether the two distributions overlap. I feel that some MMD or classifier-based score could be better here. What do you think?\n- In Fig1b, why is Original so low for medium and high memorization retain sets?\n- typo: ln171: will cause accidentally erasing", "limitations": "I think it is becoming increasingly important to evaluate methods on settings beyond CIFAR+ResNet. This setting is becoming a bit dated compared to current applications of CV or other ML methods. Of course you can’t evaluate on everything, but if the experimental setting is this specific, it could be noted. \n\nOn broader impacts, a current (negative) dual use scenario for unlearning techniques is to remove the influence of important safety data from finetuned models, e.g. https://arxiv.org/pdf/2310.03693. I think unlearning work should generally be aware of this scenario.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- Very important: The paper discovers a relationship between two retain set properties and unlearning difficulty, and leverages this relationship to improve unlearning performance on some metrics in a setting where these properties can be measured for forget sets. These are interesting findings in their own right, and provide a good starting point for follow up work.\n- Important: Experiments are straightforward and easy to understand, with only a few exceptions.\n- Important: The paper is very well written and enjoyable to read.\n- Important: The paper is well situated in the literature by the introduction, preliminaries, and related work sections. The connection to work on forgetting during training is interesting.\n- Of some importance: The paper contains some individual observations that are interesting, though not well supported by the remainder of the analysis. For example, that “operating sequentially on homogenized subsets according to memorization can boost the performance of unlearning algorithms” and that “a key observation from our results is that we can do well by actually doing nothing on a subset of the forget set”.", "weaknesses": "- Important: The ToW metric seems very reasonable at first glance but has some worrying issues on further thought. Mainly, the fact that this metric compares average accuracies across a data distribution is problematic. Consider that two models could disagree on many individual data predictions while achieving the same overall accuracy. Thus, an unlearning algorithm could yield a very different model from $\\theta^r$ while still achieving good ToW. I like that the metric balances three different desirable quantities, similar to a harmonic mean, but it seems better to balance existing, accepted metrics from the literature, like empirical measurements of divergence between unlearning/original model predictions, MIA, extraction attack success, and accuracy on retain and held-out test sets. Analysis in the paper is only conducted in terms of ToW (Sec 4).\n- Important: Results on MIA could be stronger. It appears that the refinement step accounts for most/all of the gains here, which is basically unexpected given the previous analysis in the paper. Improvements on ToW are positive, which would be good for the paper, if I didn’t have some skepticism around the ToW metric described previously.\n- Important: Conducting all experiments with ResNet models on CIFAR datasets limits the generalizability of the results. While a breadth of unlearning methods are considered, I think evaluations should be conducted with other datasets and architectures too to ensure generalizability of the results.\n- Of some importance: This is a personal opinion: I find the focus on unlearning in the differential privacy vein to be the wrong approach for many unlearning settings. I think it makes sense for users requesting their data not be used in training, and for model developers assuring approximate compliance with this request. I think it doesn’t make sense for unlearning private or dangerous information from models. The paper mentions extraction attacks once in the related work section, but extractability seems like it should be the core metric for unlearning to me. We either want a model to get a datapoint correct or get the datapoint wrong. Whether or not it was used for training seems inconsequential much of the time (one exception is when model developers owe people money for using their data). Reading this paper, this tension appears often. For instance, lines 242-246 reveal that unlearning algorithms can get worse ToW scores when they lower accuracy on a forget set too much, echoed in L.252-253 (cf. “overforgetting”). How can you overforget a forget set? I don’t like this problem formulation. I prefer defining forget sets as data where we want extraction attacks to fail, with retain sets being data where we want high model accuracy, and whether MIAs work on retain data is a matter of user data *access* (not privacy per se, since low-mem data might be relabeled accurately by models with high confidence, even when users do not grant access for training). All this said, I recognize that DP-based definitions for privacy remain popular in unlearning research, so I factor this point into my final score with less weight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Re: Weaknesses of ToW**, we discern two interesting issues raised: 1) using average accuracies, 2) that it might be preferable to base ToW on another metric like MIA. \n\nFor 1), we computed for each example the output of the unlearned and retrained models, and we count the number of different predictions made by the two models. We find that in all cases there is only a very small percentage of disagreements and that for highly memorized data (which generally makes unlearning harder) the number of disagreements increases, but remains small. For low-memorized data, across all algorithms tested, the percentage of disagreements is within ca. [3%, 5%], while for highly-memorized data, it ranges between ca. [7%, 15%]. We will add these results to the paper.\n\nFor 2) we note:\n- There is an ongoing debate for the most appropriate metric for unlearning. An advantage of ToW is the possibility to compare unlearning algorithms “holistically” based on a single scalar.\n\n- ToW does use existing metrics: the “accuracy gap” (between the unlearned and retrained models), is commonly used in the literature!\n\n- While we used ToW for our analyses (for simplicity and efficiency), we did evaluate RUM with MIA too. The fact that RUM improves w.r.t. MIA as well can be seen as an indication that ToW is actually a useful proxy for analyzing factors affecting difficulty that generalizes to other metrics.\n\n- But to address this more concretely, we tried a variant coined “ToW-MIA” that replaces the accuracy gap on the forget set term with the MIA gap (see Section 6). The results shown in Figures 7-9 in the PDF clearly show the same trends with ToW-MIA as for ToW, both for analyses and RUM. We will add these to the paper.\n\n**Re: “Results on MIA could be stronger” and “ refinement step accounts for most/all gains”**\n- Our results in general show that both the refinement and per-subset algorithm choice of RUM play a big role. We do not see why it is “unexpected” to see that, for MIA, the refinement step plays a bigger role. This is actually a nice finding, which may lead to further insights.\n\n- Table 1 in the paper clearly shows that RUM introduces strong MIA gains! For any algorithm there, the RUM version has much better MIA (and ToW). We are thrilled with this result (and frankly surprised to read this criticism).\n\n**Re: limited to ResNet and CIFAR**. We address this point by adding an additional dataset and architecture. Please see the “common response” to all reviewers. We hope this considerably helps to put at ease doubts about the generalizability of our results.\n\n**Re: personal opinion on unlearning, privacy and “whether or not it was used for training seems inconsequential”**: We believe different applications require different treatment. Here we are addressing unlearning defined as removing a subset of training data (the forget set). We agree some problems (e.g. preventing harmful generations) are not necessarily formulated as this type of “unlearning” because, as the reviewer said, all we care about there is to prevent those “bad” outputs (regardless of what data was used for training). In that case, retraining from scratch isn’t even the right reference point. However, our formulation is useful in other cases, e.g. enabling data deletion from a vision classifier. In that case, it is reasonable (and common) to adopt this DP-like formulation: ideal “deletion” of a data point leads to a model similar to one that was trained without that point. And in this case, MIAs are a common and convenient tool for measuring “leakage” and success of this type of unlearning. Overall, these are highly debated topics, with subjective opinions, and without consensus. While work is needed to settle these debates, we hope the reviewer agrees beyond doubt that our problem setting is widely considered a valuable one.\n\n**Re: “how can you overforget a forget set?”** Following from above, our goal is to produce an unlearned model “similar” to that of retrain-from-scratch. So its accuracy on the forget set should be just as low as that of retraining, but no lower. We use the term “overforgetting” to describe the situation where the unlearned model causes a forget set accuracy lower than that of retrain. This is problematic as it can lead to vulnerability to MIA (due to noticeable differences from retraining).\n\n**Re: should we exclude retain set from ToW?** The retain set is important. For instance, if a user has contributed some data towards training a model, and that user is not in the forget set, the user wants to still benefit from the model performing well for their data. At the same time, we want to preserve the ability to generalize to new users (test performance).\n\n**Re: using MMD instead of ES**: Interesting point! We experimented with MMD and find there exists a (negative) correlation between ES and MMD scores. For CIFAR10/ResNet18 and CIFAR100/ResNet50, there exists a negative correlation where low(high) ES scores lead to high (low) MMD values.\n\n**Re: “why is original so low for medium and high memorization retain sets”**. We assume the reviewer means “forget sets” rather than “retain sets” here (please correct us if not). An example that is “low mem” is one that has not influenced the model too much by definition (see Equation 1). So the predictions of the “original model” already look like those of the “retrained” model for this example, making the original model represent “good unlearning” according to the metrics of interest (based on similarity to the retrained model). On the other hand, for higher-mem examples, the original model is more influenced by those examples and therefore it “looks less” like a model that was trained without those examples. In these cases, the original model is a poor solution. Please let us know if this clarification helps.\n\n**Re: limitations**, we will note these, thank you.\n\nWe hope our rebuttal satisfies all key concerns of Reviewer 3Cxb. We look forward to discussing any remaining concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gbLWanyOWg", "reviewer": "Reviewer_HxTF", "summary": "This paper looks at measurable qualities of forget/retain sets in machine unlearning contexts, which can make it difficult to 1) unlearn the forget set at all 2) unlearn the forget set without also unlearning the retain set. The paper puts forward well-reasoned metrics that get at entanglement/homogeneity in the embedding space for the forget and retain set, and also examine memorization as a possible factor for complicating machine unlearning. Based on the lessons from this, the paper suggests a meta-learning algorithm to figure out how to use existing unlearning algorithms to better unlearn the forget set.", "questions": "See weaknesses -- It would be great to understand more about efficiency and if there are interesting trade-offs here with respect to the proposed method", "limitations": "See above re: efficiency analysis", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "This paper studies a very sensible set of questions with respect to qualities of the forget set that could make machine unlearning hard. The results (on the motivation side) are arguably expected, but someone needed to actually do the work to show this, and I'm really glad this paper did so (and did so very comprehensively, with respect to entanglement and memorization, and the ToW metric). Doing this / using these (perhaps expected) lessons to motivate the meta-learning RUM algorithm is also very principled, and also makes for a really tight paper. Without doing the first half fully, it would be hard to understand why the second half/ proposed algorithm is so well-motivated.", "weaknesses": "Overall, I think this paper makes excellent contributions to unlearning science. There is one thing I would have liked to see addressed in  more detail in the main paper, and I have some nit comments (that do not affect my score) that I think would improve readability. I organize my response here according to these distinctions, to (hopefully) make it clear which things could possibly lead to improvements in my score. \n\n**Affects my score, engaging with this could lead to me increasing my score**\n\nGiven how expensive unlearning methods tend to be still, it would be great to see more discussion about efficiency. It is not clear to me entirely from the main paper if there are significant associated costs of using this meta-learning approach. These costs might be entirely worth the benefits presented (depending on the context of use, of course), but it would be really useful to include more on this in the main paper.\n\nHowever, I also note that this paper packs in so much already in the page count, that I understand why some things had to be cut.  \nBut this also led to other problems (which I understand are in tension with what I'm saying about including more details about efficiency): the quantity of presented results for a paper of this length has demanded really small (often difficult to read and process) tables and figures. The reason this is important is not just about readability. It takes a lot more time as a reader to unpack and verify the results in relation to the claims being made. \n\n**Nit comments, do not have an impact on my score**\n\nMath readability: Using different symbol classes for things of different types could help readability of the definitions a lot in this paper. For example, curly braces are used here for both algorithms and sets of datapoints. A simple improvement would be to use visual cues in the format to show they're different (not just with the chosen letter). For example, using bolded capitalization for a matrix of examples. Doing this can also make editing/finding typos more noticeable (e.g., line 60 has an $S$ for the forget set that, with the current style/format, should be $\\mathcal{S}$).\n\nPlease explicitly define $e$ in the follow-on to Definition 2.1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Re: More discussion about efficiency / costs of RUM**. This is a great point! Many researchers in this area do not bother with addressing issues of efficiency. Due to space constraints, we may have also been guilty of not paying as much attention as needed on this topic. This is unfortunate as indeed with our contributions we do address many issues about unlearning efficiency issues and those that arise as a result of our RUM framework. \n\nPlease note the below, which we will add more thoroughly in the paper too:\n\n1) Our contributions show the importance of memorization, how it affects different SOTA algorithms, and how it can be leveraged with our RUM framework. But computing memorization scores, a la Feldman, is a very computation-heavy process, requiring a large number (100s) of models. We only just alluded in the paper to using proxies for the Feldman memorization score. But we agree this requires more concrete discussion.\n\nIn preliminary experiments, we experimented with various memorization score proxies from the literature. We have identified a promising one (C-proxy) and experimented with it in detail during the rebuttal (please see the “common response”). Our results show that: (i) our analyses and trends in Section 4 regarding the impact of memorization on unlearning algorithms continue to hold when using C-proxy instead, with no noticeable differences. (ii) leveraging the c-proxy values instead of the actual (a la Feldman) memorization scores still  leads to very large performance gains with our RUM framework, consistent with the ones in the original paper. In the attached PDF, please see Figure 1 and 2, showing the performance of algorithms according to low/medium/high c-proxy values and the performance of RUM when using C-proxy, respectively. As before the gains are large and clear with C-proxy as well!\n\nPlease note that computing c-proxy instead of the Feldman’s memorization score enjoys orders of magnitude performance improvement  (due to avoiding training a large number of models that are needed to compute Feldman’s memorization scores). This makes our RUM framework practical and efficient to deploy.\n\n2) Our contributions show when it makes sense to apply any of the SOTA algorithms, based on the memorization scores of examples in the forget set. So, a forget set consisting of only low-memorized examples, can be excluded from the unlearning process, saving all unlearning costs: as discussed in the paper, unlearning in this case is at best not needed (and at worse harmful), because even a model retrained without such examples can classify them correctly, so there is no need to modify the model to make it unable to classify those examples. In fact, attempts at doing so may lead to worse unlearning performance, as we discussed in the paper for Random Label and SalUn.\n\nThis finding leads directly to efficiency improvements during unlearning. The exact gains are hard to quantify, but one should reasonably expect the efficiency gains to be analogous to the ratio of the size of the low-memorized examples over the overall forget set size.\n\n3) Our results show that often much more efficient unlearning algorithms (like GA and other similar baselines that mainly need access to the examples in the drastically smaller forget set and do not depend heavily on the drastically larger retain set) can be competitive in terms of ToW/MIA performance for at least some partition (low/medium/high) of the forget set. Thus, for these partitions of the forget set, with RUM one could enjoy much higher efficiency without sacrificing ToW/MIA performance. This will obviously reduce the overall time for unlearning. \n\n4) In addition, RUM is also shown to be able to significantly improve the performance of all algorithms studied in terms of ToW/MIA. Hence, consider a practitioner who chooses any algorithm (which e.g., for efficiency reasons may not be the best performing in terms of ToW/MIA). By “RUMifying” it, it also boosts its ToW/MIA performance, having a more acceptable and efficient unlearning solution. \n\nWe will gladly reflect these discussions about efficiency issues with respect to our contributions in the paper, since, as HxTF points out these are very important (and strengthen our paper). \n\n**Re: the tables and figure are small**: We agree! This is actually an important point. We will address this, making figures and table more readable. \n\n**Re: nit comments on math readability**: great point, we will make this adjustment (and thanks for catching the typo re: the style of S).\n\n**Overall**, we thank the reviewer for the thoughtful feedback and encouraging remarks. We believe that  the discussion above addresses all key elements of efficiency which pertain to our contributions and that our contributions can lead to an efficient and practical deployment (in addition to their contributions to unlearning science). We would love to hear the reviewer’s thoughts on this and discuss further if there are remaining questions. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j43xys0NHp", "reviewer": "Reviewer_7Yis", "summary": "This paper studies the difficulty of unlearning with respect to different unlearning sets. Specifically, the authors examine two factors, namely entanglement between forget and retain sets, and memorization. The authors further propose a framework called RUM which first divides the forget set into different subsets according to the memorization score and performs different unlearning algorithms to unlearn each subset.", "questions": "See above.", "limitations": "Limitations are addressed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I agree that studying the difficulty of unlearning different data (subset) is a very important problem and would help us evaluate current approximate unlearning algorithms. I generally enjoyed reading the paper, especificallly Section 4, where the 2 factors are nicely chosen and the experiments (figure 1) well demonstrate their effects. Specifically, the memorization part aligns with the study of data attribution and matches my intuition for approaching this problem.", "weaknesses": "I will merge the weaknesses and my questions in this part such that the authors can easily address each point during the rebuttal:\n\n**[Memorization]**: according to equation (1) and Section 4.2, it appears to me the authors perform \"leave-one-out\" experiments, where they train a model without a target data sample and calculate the memorization score in (1). This is repeated for every training data point (50000 times for CIFAR-10) and it seems quite expensive (if training until convergence). My questions here are: \n- Did you use any approximation in your experiments? How many epochs did you train? Can you show the computational cost?\n- In Section 6, the refinement step is also performed based on memorization scores. Thus I believe the calculation of memorization score should be also considered for \"efficiency\" in Section 7? Given a large model (e.g., LLMs), and a large forgetting set, this process would be extremely expensive and may decrease the practicality of RUM.\n\n**[MIA and ToW]**: The authors use ToW as the only evaluation in Section 4 and apply both MIA and ToW for Section 5. It is unclear to me why they omit MIA for evaluating unlearning difficulty. Is it because MIA may be a biased evaluation? Sequentially, for step 3 in RUM, will the selection of optimal unlearning algorithm be affected by considering MIA as well?\n\n**[Transferability]**: RUM relies on the observation in Section 4 to select optimal unlearning algorithms. This comes handy as Section 4 and 6 both consider the same datasets. However, in practical scenarios, this is not possible and one needs to rely on the assumption that one unlearning algorithm is optimal across different datasets and model architectures. Is this assumption true? \n\n**[Title and Section 4]**: The title of this paper and name of Section 4 \"what makes unlearning hard\" is misleading to me. This presentation suggests that the authors are studying: (1) unlearning is a hard problem; (2) why it is hard. However this is not true and they actually studied \"the difficulty of unlearning regarding choice of forgetting sets\". Specifically, the memorization subsection actually suggests different behaviors for different unlearning methods. I would suggest a more accurate delivery such as \"on the difficulty of unlearning...\".\n\n**Overall**, I like the problem of interest in this paper and I appreciate the takeaways from Section 4. I think the observations help the community understand unlearning better. However, I don't believe RUM is a super practical framework based on the tremendous effort needed to perform refinement and the unknown behavior of transferability. If any of my speculations are wrong I would love to hear back from the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Re: “leave-one-out experiments” and “did you use any approximation?”**. Let us clarify the setup. \nWhile the reviewer is correct that Feldman's definition of memorisation implies a 'leave-one-out' training, in practice this evaluation is prohibitively expensive, as pointed out. As a more efficient empirical approximation, (Feldman and Zhang, 2020 “What neural networks memorize and why: discovering the long tail via influence estimation”) suggests performing 'leave-k-out' training and repeating it many times (e.g. 4k times). This way, for each data point, one can find a sufficiently large number of models that include that point in the training and a sufficiently large number of models that don’t (in the pool of 4k models). In Feldman's experiments, k is 30% of the training dataset size. We have followed the same recipe to calculate the memorisation scores and our results agree with Feldman's findings. Thank you for pointing this out. We will add these clarifications to a section in the appendix for clarity (and refer to it from the main paper). \n\nFurther, in our new experiments in the rebuttal, we have considered using a proxy based on confidences that we find correlates well with memorization scores, leading to an efficient and practical recipe using RUM. Using this proxy is actually key as its computation requires orders of magnitude fewer models than Feldman’s memorization score, making our framework practical while still highly performant! Please see the “common response” for details. We hope that this line of experimentation and our thorough results also alleviate the concern of the reviewer that “the refinement step is also performed based on memorization scores” and “this process may be extremely expensive and may decrease the practicality of RUM”, since our findings using this cheap proxy align with those obtained using the true memorization scores.\n\n**Re: “how many epochs did you train? Can you show the computation cost”**. We trained for 30 epochs for cifar10. For cifar100, we used the public memorization scores (computed by Feldman et al) and used the same model architecture (resnet50) as Feldman’s for the experiments. Please also see section A.2 in the appendix for additional details.\n\n**Re: MIA and ToW**. The reviewer is correct that we could have repeated our analysis in Section 4 using MIAs, and accordingly we could have also based RUM’s selection of the unlearning algorithm per subset based on those findings. In our original experiments, we made the design decision of using ToW there due to its simplicity and compute-efficiency. We want to emphasize that, ultimately, we evaluate the algorithms we produce (RUM variants) using both ToW and MIA, and we find improvement on both fronts! We view this general improvement as evidence that ToW is a useful proxy for gauging how behaviours of different algorithms vary according to key characteristics.\n\nHowever, during the rebuttal, we have expanded our analysis to address this feedback. Specifically, we tried a variant coined “ToW-MIA” that replaces the accuracy gap on the forget set term with the MIA gap (see Section 6). The results shown in Figures 7-9 in the PDF clearly show the same trends with ToW-MIA as for ToW, both for analyses and RUM. We will add these to the paper.\n\n**Re: transferability and “one needs to rely on the assumption that one unlearning algorithm is optimal across different datasets and model architectures”**. It is true that our findings in this paper don’t inform us about the behaviour of any algorithm on any dataset (a limitation shared by any empirical paper). Understanding behaviours of algorithms across different unlearning problems is actually a major challenge that the community is faced with still, perhaps owing to unlearning being a young area of research. We view our work as a first step in uncovering key factors that practitioners can focus on when attempting to investigate the best algorithms to use in new contexts, which were not previously known. \n\nWe also emphasize that, in the context of a single chosen algorithm, even applying only the refinement step of RUM (applying that given algorithm in a sequence on refined subsets of the forget set, according to characteristics we discover) can boost performance significantly. This is very important as it offers a direct way to leverage our findings to get a boost in performance even when in scenarios where we don’t know the optimal algorithm to choose per subset (and rely on using a single pre-chosen algorithm). \n\nFinally, we note that during the rebuttal we have investigated an additional dataset and architecture, showing that our main conclusions hold there too, as evidence for transferability. Please see the “common response” to all reviewers for more details.\n\n**Re: title and section 4**. When we say “what makes unlearning hard”, the word “what” refers to “what properties of the forget set, or the retain / forget partition”, but that is a little bit of a mouthful, so we simplified it. We believe our paper studies what makes the problem hard, through showing under which circumstances (in terms of ES and memorization scores) different algorithms struggle to perform well; and we view this as largely conveying the same message as “on the difficulty of unlearning”. Does this clarification help, or does the reviewer still feel that the title is a little misleading? We are happy to brainstorm more about this.\n\n**Overall**, we thank the reviewer for the thoughtful feedback. We actually believe that our new results (with C-proxy; see the “common response”, and with new a new dataset and architecture) as well as the discussion above addresses the reviewer’s central concerns about the practicality and transferability of RUM. We would love to hear the reviewer’s thoughts on this and discuss further if there are remaining concerns. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for the valuable feedback. We are pleased to hear that the reviewers found our analysis “valuable”, that we study “very sensible questions”, “very comprehensively”, and our paper “makes excellent contributions to the unlearning science”; that our findings are good both “in their own right” as well as “for follow-up work”, the paper “well-written” and “well-situated in the literature”.

In this general response, we address concerns shared by more than one reviewer. We respond to individual concerns in the per-reviewer responses separately.

**Re: practicality and efficiency.** To address reviewers concerns on the efficiency and practicality of the RUM framework, in particular as it relates to the cost of computing memorization scores, we experimented with a proxy for these scores, that we coin as the C-proxy, based on the confidences, as proposed in [1] (see reference below). Specifically, for a data point (x,y) and a model M, we measure the softmax confidence of M(x) corresponding to the correct class y at each epoch during the training of the original model. The C-proxy metric is then computed by averaging these confidences across all training epochs for each data point at the end of the training process. The C-proxy exhibits a strong negative correlation with memorization, with a Spearman correlation coefficient of approximately -0.9.

We begin by showing (in Figures 1 and 2 of our uploaded PDF) that C-proxy behaves the same as using memorization levels themselves (by contrasting these Figures with the corresponding figures in our paper). Based on this observation, we conduct other rebuttal experiments (e.g. see the below point for other datasets and architectures) using this C-proxy, since it’s more efficient.

Please note that computing C-proxy instead of Feldman’s memorization score enjoys orders of magnitude performance improvement (due to avoiding training a large number of models that are needed to compute Feldman’s memorization scores). This makes our RUM framework practical and efficient to deploy.

**Re: we only used two dataset-architecture pairs (CIFAR-10 with ResNet-18 and CIFAR-100 with ResNet-50)**

First, we would like to emphasize that producing the results and deriving the insights in the paper was enormous amount of effort. Specifically, training different size models, several times (in order to e.g. compute memorization scores) across different datasets, using many state of the art algorithms and analyzing them with respect to ToW and MIA as they are affected by both memorization scores and embedding space distances of forget-set examples takes enormous effort, as we are the first to explore this space. And in addition to the above effort we further showed the effectiveness of RUM, for different combinations of SOTA and baseline algorithms and across datasets and architectures. Please note as well that, thanks to our analyses (and also the discovery that C-proxy yields similar findings as computing memorization scores), future efforts can be much more computationally-efficient thanks to this effort. We will now present evidence in support of RUM’s generalizability to new datasets/architectures as well.

We agree that using only two CIFAR datasets is a valid concern. To respond to it, we provide results both: a) with an additional dataset (Tiny-imagenet) using Resnet18, and b) with Tiny-imagenet using an additional architecture (VGG16), and we do this both for the analyses in Section 4 (investigating behaviours of algorithms w.r.t ES and memorization) and in the context of RUM.

These results Figures 3-6 in the PDF, agree completely with the previous results in the paper. Specifically, all trends identified with the previous experiments (e.g., regarding the impact of memorization and ES on unlearning difficulty) and the impact of RUM in improving each algorithm in isolation and overall are manifested with the new dataset and with the architecture. We believe this is important evidence for the generalizability of our findings and we hope alleviates this concern. 

**Overall**, we thank the reviewers for the insightful comments and believe that we have addressed the concerns of all reviewers during the rebuttal, which has improved our paper. We are eagerly awaiting the reviewers’ responses (on these general points and per-reviewer rebuttals) and we look forward to continue these discussions.

References:
[1] Jiang, Ziheng, et al. "Characterizing Structural Regularities of Labeled Data in Overparameterized Models." International Conference on Machine Learning. PMLR, 2021.

Meta-review: This paper looks at the machine unlearning problem — roughly, given a model m and a subset of the data (the “forget set”) used to train that model, efficiently create a new model m’ that does not include information from the forget set but is representative of the family of models that would’ve been trained on the initial data sans the forget set (the “retain set”).  This is a known tricky problem, and a well-motivated one due to so-called “right to forget” capabilities enforced in the EU’s GDPR and other evolving regulatory movements.  The paper finds that if the forget and retain sets are close, in some sense, then the unlearning problem becomes harder.  It also finds that, if elements of the forget set are fundamentally part of the model itself via memorization, then the unlearning problem becomes harder.  Neither of these results are particularly surprising, but the analysis is strong and this is great to have written down.  The paper then provides a new method based on these two insights that arguably improves unlearning over various SotA and SotA-adjacent methods.  Reviewers found issue with the experimental analysis, as well as some nitpicks around the identified sources of hardness, but these were largely assuaged in the rebuttal and post-rebuttal discussion.
Decision: Accept (poster)