Towards Unbounded Machine Unlearning
Authors: Meghdad Kurmanji
Peter Triantafillou
Jamie Hayes
Eleni Triantafillou
Venue: NeurIPS 2023 poster (2023)
Keywords: machine unlearning
deep learning
TL;DR: We propose a scalable unlearning algorithm that is by far the most consistent in performing strongly among all SOTA methods we evaluated, across different sets of evaluation metrics and application scenarios

Abstract: Deep machine unlearning is the problem of 'removing' from a trained neural network a subset of its training set. This problem is very timely and has many applications, including the key tasks of removing biases (RB), resolving confusion (RC) (caused by mislabelled data in trained models), as well as allowing users to exercise their 'right to be forgotten' to protect User Privacy (UP). This paper is the first, to our knowledge, to study unlearning for different applications (RB, RC, UP), with the view that each has its own desiderata, definitions for 'forgetting' and associated metrics for forget quality. For UP, we propose a novel adaptation of a strong Membership Inference Attack for unlearning. We also propose SCRUB, a novel unlearning algorithm, which is the only method that is consistently a top performer for forget quality across the different application-dependent metrics for RB, RC, and UP. At the same time, SCRUB is also consistently a top performer on metrics that measure model utility (i.e. accuracy on retained data and generalization), and is more efficient than previous work. The above are substantiated through a comprehensive empirical evaluation against previous state-of-the-art.

Towards Unbounded Machine Unlearning
Meghdad Kurmanji'
Peter Triantafilouu University of Warwick
Jamie Hayes Google DeepMind
Eleni Triantafillou Google DeepMind
University of Warwick
Abstract
Deep machine unlearning is the problem of 'removing" from trained neural network subset of training set. This problem very timely and has many applications, including the key tasks of removing biases (RB), resolving confusion (RC) (caused by mislabelled data in trained models), as well as allowing users to exercise their 'right to be forgotten' the first, to our knowledge, to study unlearning for different applications (RB, RC, UP), with the view that each has its desiderata, definitions for 'forgetting' and associated metrics for forget quality. For UP, we propose a novel adaptation strong Membership Inference Attack for unlearning. We also propose SCRUB, a novel unlearning algorithm, which is the only method that is consistently a top performer for forget quality across the different RB, RC, and UP. At the same time, SCRUB is also consistently a top performer on metrics that measure model utility (i.e. accuracy on retained data and gener- alization), and is more efficient than previous work. The above are substantiated through comprehensive empirical evaluation against previous kfte-of-the-art
protect User Privacy (UP). This paper is
opplication-deenddern metrics for
Introduction
Can we make a deep learning model 'forget' a subset of its training data? Aside from being scientifically interesting, achieving this goal of deep machine unlearning is increasingly important and relevant from a practical perspective and this research direction is enjoying significant attention recently (Triantafilouu al., 2023]. Regulations such as EU's General Data Protection Regulation [Mantelero]] 2013] stipulate that individuals can request to have their data 'deleted' (the 'right to be forgotten') ). Nowadays, given the ubiquity of deep learning systems wide range of applications, including computer vision, natural language processing, speech recognition and healthcare, allowing individuals exercise this right necessitates new technology. Unlearning aims to fill this need.
However, unlearning has several additional important applications where privacy with respect to subset of individual training examples isn't necessarily consideration: removing out-of-date examples, outliers, poisoned samples (Jagielski et al., 2018), noisy labels [Northcutt et al., 2021]. or data that may carry harmful biases [Fabbrizzi al. [022]] In fact, are the first to argue that the definition of 'forgetting' is application--deeeddent. and unlearning comes with different desiderata and priorities in different scenarios. For instance, for a privacy application, 'forgetting' the data of users who requested deletion is the primary objective and we may be willing to sacrifice model performance some degree, exchange for reliably defending Membership Inference Attacks (MIAs). In other cases, for instance, when deleting outdated data to keep a model current, maintaining the model's performance throughout such 'cleanup' operations may be the primary consideration. and guaranteeing that no trace of the old data is left behind may be irrelevant. While previous work does not establish such distinctions, we study three scenarios and propose a method that the most consistent top- performer across applications and metrics.
*Correspondence Meghdad. Kurmanj www.wwiiccc....m..
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Unfortunately, removing the influence of a subset of the training set from the weights of a trained model is hard, since deep models memorize information about specific instances [Zhang et al. 2020 2021 Arpit et al., 2017] and their highly non-convex nature makes it difficult to trace the effect of each example on the model's weights. Of course, we can apply the naive solution of retraining the model from scratch without the cohort of data to be forgotten. This procedure indeed guarantees that the weights of the resulting model aren't influenced by the instances to forget (it performs "exact unlearning") ), but the obvious drawback is computational inefficiency: retraining a deep learning model to accommodate each new forgetting request isn't 't viable in practice. To mitigate that cost, recent work turned to approximate unlearning (Izzo et al.) 2021 Golatkar et al., 2020a b) where the goal is to modify the weights of the trained model to approximate those that would have been obtained by exact unlearning. That is, they strive to achieve 'indistinuuishaiiiity between the approximate and exact solutions, typically accompanied by theoretical guarantees for the quality of that approximation. This goal is also mirrored in the metrics used by these works: the model's error on the deleted data is desired to be just as high as the error of the 'exact' retrained model (which truly never saw that data). The argument is that error higher than that reference point may cause susceptibility to membership inference attacks: a noticeably poor performance on an example can reveal that it was unlearned.
In this work, we show empirically that such approximate unlearning methods often perform poorly in practice (in one or more of RB, RC and UP). We hypothesize that this is due to potential violations of assumptions made by these methods, e.g. that the "optimal" solution of unlearning is close in weight space to the original model [Golatkar et al. 2020alb (due to assumptions of stability of SGD and 'small enough' forget sets). Further, we find these previous methods scale poorly to the size of the training set and the model, often making them impractical. To address these issues, we propose a new unlearning model, SCalable Remembering and Unlearning unBound (SCRUB) that departs from that methodology. SCRUB bears a novel teacher-student formulation, where the student model selectively disobeys an all-knowing teacher, to inherit from it only knowledge that does not pertain to the data to be deleted. This method is 'unbound' from limiting assumptions (e.g.. to facilitate formal guarantees), poor scalability limits, and is not tailored can yield a very high error on the deleted data, which is desired in some scenarios but may cause susceptibility to membership inference attacks. To mitigate this when relevant, we extend SCRUB with novel 'rewinding' procedure that pinpoints the 'checkpoint' of the unlcarning process to use such that the error on the deleted data approximates the reference point of being 'just high enough'.
single unlearning application. We find that SCRUB
We also propose the first, to our knowledge, adaptation of the state-of-the-art LiRA MIA CCrllini et al. 2022 to the framework of unlearning, as metric for UP. Overall, we perform a comprehensive evaluation using different baselines, datasets, architectures and three application scenarios. We find that SCRUB is by far the most consistent top performer out of all state-of-the-ar methods considered. SCRUB is able to ensure high forget quality, satisfying different forget-quality metrics for different applications while also being a top performer with respect to model utility (accuracy on retained data, and generalization). Our findings highlight the need to consider a broad spectrum of applications when evaluating unlearning algorithms, to better understand their properties and trade-ofts.
2 Problem Definition for Unlearning Unbound
Notation and preliminaries Let D = {Z;, (ek.::1' B denote a training dataset containing N examples, where the i'th example is represented by vector of input features z; and a corresponding class label yi- Let (c; w) denote a function parameterized by trainable weights w. In this work, we study the case where / is represented by deep neural network, trained in a supervised manner via empirical risk minimization. Specifically, we define the loss function as L(w) = NN EN ((f( w), y:) where I denotes the cross entropy loss.
Deep Machine Unlearning We now formalize the problem of deep machine unlearning. We assume that we are given a model f(: w²) that has been trained on D. where ff denotes neural network and w" its trained parameters, obtained by minimizing the above loss function. We will refer to this as the 'original model' (i.e. before any unlearning intervention is performed). We are also given a 'forget set' DD- (I;, Vi , C D. comprised of N, examples from D and 'retain set' D, of N, training examples that the model is still allowed to retain information for. For simplicity, we consider the standard scenario where D, is the complement of D, or, in other words, Dj U UD, D.

---

Given f(; ((:w"), Dj and D,.. the goal of deep machine unlearning is to produce a new set of weights w" such that the 'unlearned model' f(; w") has 'forgotten" D, without hurting the 'utility' of the original model (its performance on the retained data D, and generalization to data outside of D). We argue that defining (and measuring) 'forgetting" best of our knowledge, to perform thorough empirical evaluation of unlearning algorithms on three different application scenarios, each with its own definition of forgetting and associated metrics.
application-dependentt Our work is the first, to the
Defining and measuring 'forgetting' We consider three different application scenarios in this work, each with their own definition of forgetting and associated metrics. 1) Removing biases. We want maximal error on Dy: the model should never predict the assigned labels of forget examples, since they reflect unintended behaviour. 2) Resolving confusion. A model may be 'confused' due to incorrect labels in its dataset. successful unlearning algorithm resolves this if mislabeled examples are placed in the forget set (see metrics in Section [5. 3) User privacy. Success from a privacy perspective is associated with a forget error only as high as that of forget error could lead to vulnerability to Membership Inference Attacks (MIAs). In all of the above cases, in addition to achieving forgetting. we seek unlearning algorithms that don't 't damage model utility, i.c. accuracy on retained data and generalization ability, and that unlearning is efficient enough to be preferable over the simple and expensive solution of retraining the model.
a higher
Fundamental trade-offs A top-performing unlearning algorithm must simultaneously achieve three feats: (i) obtain high 'forget quality', (ii) ensure continued high accuracy on retained and unseen data, and (iii) efficiency / scalability. In practice, there are fundamental trade-offs in two different dimensions. The first dimension involves trade-off's between (i), (ii) and (iii,, for example achieving good 'orgett quality' at the expense of efficiency. Retraining from scratch is such an example, but, as we will show later, some methods suffer from inefficiencies even worse than retraining. Another common issue we will observe is achieving good 'forget quality' at the expense of utility. The second dimension involves trade-offs within the context of achieving good 'forget quality' across applications, since the metric for this quality is application-speciie strongly one such metric but poorly others. SCRUB is designed to handle these trade-offs and is shown empirically to consistently outperform previous methods across the board.
and an algorithm may perform
3 SCalable Remembering and Unlearning unBound (SCRUB)
We present SCRUB, a new unlearning algorithm based on a novel casting of the unlearning problem into a teacher-student framework. We design SCRUB to meet the desiderata of unlearning: efficiently 'forgetting" without hurting model utility. Because we argue that the definition of forgetting is application-dependent, we propose a recipe that works across applications: We first design SCRUB to strive for maximal forget error, which is desirable in some scenarios (e.g. RB and RC) but not in others (c.g. UP). To address the latter case, we complement SCRUB with a novel 'rewinding' procedure that reduce the forget set eттor appropriately when required. SCRUB is 'unbound' from limiting assumptions (c.g.. to facilitate formal guarantees), poor scalability limits and is by far the most consistent method in forgetting across applications while preserving model utility.
3.1 The SCRUB
We consider the original model w as the 'teacher' and we formulate our goal training a 'student' w" that selectively obeys that teacher. Intuitively, our goal for w is twofold: forget Dy while still remembering D,. To that effect, w" should be obeyed when teaching about D,., but disobeyed when teaching about Dy. Our code is available for reproducibility
En route to deriving our training objective, let us first define:
d(æ; w") (pl((( w (I)("
In words, d(æ;; w") is the KL-divergence between the student and teacher output distributions (softmax probabilities) for the example æ. We make the dependence of d on w" explicit, since we will optimize the student weights nh while keeping the teacher weights frozen wº, treating them constant.
3

---

Concretely, we begin by initializing the student wn to the weights of the teacher w". Since the teacher weights wº were trained on all of D, the teacher performs well on both constituents D, namely both of DD and Dj- Therefore, the student has good performance on D, at initialization, already fulfilling one of our desiderata. Can we then start from this solution and modify it to unlearn Dy? In principle, one could do this by optimizing:
Σ (ezrw" ByEP,
min QU1
NJ
However, in practice, when performing this maximization of the distance between the student and teacher on the forget set, we noticed that, while indeed the performance on D degrades, as desired, it unfortunately also results in degrading performance to simultaneously encourage the student to 'stay close' to the teacher on retain examples, while encouraging to 'move away' from the teacher on forget examples, adding contrastive flavour. Formally, the optimization objective becomes:
D,. To amend that, we propose
min LOS
Σ ETDD,
d(s,; w") - N₂
Σ d(y;;w") TYEDs
Furthermore, SCRUB also simultaneously optimizes for the task loss on the retain set, to further strengthen the incentive to perform well there. Our final training objective is then the following:
min W
a N
Σ d(x;:=))+ EDD
Y N
Σ (ff(x;:""::) - N₁ (Zr:
Σ d(zy:w") ByED,
(3)
where stands for the cross-entropy loss and the and y are scalars that treat as hyperparameters.
In practice, we found that optimizing the objective in Equation 3 is challenging, due to oscillations in the loss. Intuitively, this is due to trying to simultaneously satisfy two objectives, which may interfere with each other, namely moving close to the teacher on some data points while moving away from others. To address this, SCRUB provides practical recipe for optimization, reminiscent of 'tricks' used in other min- max-mm problems like Generative Adversarial Networks [Goodfellow et al. 2014] where the discriminator is trained for several steps before each update to the generator.
Specifically, SCRUB iterates between performing an epoch of updates on the forget set (the max- step) followed by an epoch of updates on the retain set (the min-step), in an alternating fashion. Guarding against hurting retain performance due to this alteration of min- mie-steps and max-steps, SCRUB also performs sequence of additional min-steps at the end of the sequence restore the retain performance in the event that it was harmed. SCRUB training stops when the forget error has increased without harming the retain set error. We find in practice this point can be reached with small number of epochs. We provide pseudocode, training plots and ablations the Appendix.
3.2 SCRUB and Rewind (SCRUB+R)
By construction, SCRUB encourages obtaining high error the forget set. This is desired for some application scenarios. However, an uncharacteristially high error on deleted examples can make them identifiable, causing vulnerability to membership inference attacks. SCRUB+R addresses this.
To this end, we are after a procedure for selecting the checkpoint of SCRUB where the forget set error is just high enough'. One could obtain reference point for that by retraining from scratch without the forget set and recording the error of that model on the forget set. However, that would defeat the purpose of unlearning, we want to avoid that computation. Therefore, we propose a different way of establishing reference point. First, we create validation set of the same distribution as the forget set. For instance, if the forget set has only examples of class O, we keep only examples of class O in the validation set too. Next, we train SCRUB as usual, storing a model checkpoint every epoch. At the end of its training, where the forget error is typically the highest, we measure the error on the constructed validation set. This will serve as the reference point for the desired forget set error. Finally, we 'rewind" to the checkpoint where the forget error is closest to that reference point.

---

acouTccy
Traturing
Rao
SOYUB
25
Scale
Ratan
Soae 10
fatan
Fikee
Porget
Rataln
eng
c10
(a) CIFAR-I0 10 class unlearning.
(b) 10 selective.
(c) CIFAR-5 selective.
Figure 1: RB results: SCRUB the only consistent top-performer in terms of forgetting and preserving utility. It is also highly efficient: second only to Bad-T which, however, fails at forgetting and damages utility. In all subfigures, each point represents the average across ResNet and AlI-CNN variants. For large-scale results, also compute the scale-up factor: the fraction of the runtime of retraining from scratch over the runtime of the given unlearning algorithm (multiplied by 5 here, for visualization purposes). We find that selective unlearning is harder for all methods, especially for FU-k and Fisher (see Figure Ta vs Tbp. CF-k and Finetuning perform similarly (they perform poorly) in all cases, their lines occlude one another. Complete tables are in the Appendix.
The intuition is that the last step of unlearning approximates 'maximally forgetting' Consequently, the error on the deenically-distribuute validation set approximates the error of a model that never learned about that distribution from the forget set: any correct predictions on the held-out examples are due only to the generalization power of the model that was trained only on the retain set. Therefore, we choose that validation set error to serve as the reference point for how high we would like the forget set error to be for UP applications. We show empirically that this rewinding procedure greatly increases the defense against membership inference attacks, making SCRUB a strong performer for applications where privacy is consideration.
Related Work
Unlearning definitions Despite prior work on unlearning, this research problem is still in its infancy and we lack a well-established formal definition. The term 'unlearming' was coined in Cao and Yangl [2015), where, in the context of certain structured problems, unlearning defined to be successful if deleting an example yields the same outputs on dataset as if that example had never been inserted. Ginart et al. [2099] introduce more relaxed probabilistic definition for machine unlearning inspired by Differential Privacy (Dwork et al., (2014) that requires the unlearned model to yield outputs that are similar to those that would have been produced by a model retrained from scratch without the forget set. This led to the development of "approximate learning" methods that aren't identical to retrain- retrain-from-seee from-scratch ("exact unlearning') but 'similar' [Guo et al. 2019 Golatkar et al. (2200a] Sekhari et al. [021]] Neel et al. [222 adopt similar DP-inspired definitions of (approximate) unlearning based on the goal of indistingussaabiity from retrain- fcom-scratch.
However, there is recent line of criticism against the idea of defining or measuring unlearning success based on indistinguishabiiiyy from titriin-from-crattee Thudi et al. (0226) theoretically show that we can obtain arbitrarily similar model weights from training on two non-overlapping datasets. This implies that reaching a particular point in parameters space isn' 't sufficient condition for having unlearned. Furthermore, [Goel et al.] [2022] argue that it's not necessary condition either: retraining from scratch may arrive at different model distributions due to changes in e.g. perpaarameeeess (Yang and Shami 2020). Therefore, a unlearning method may be unnecessarily penalized for not matching particular retrained-fromm-craaee viewpoint and define unlearning as 'forgetting" in efficient manner and without hurting model utility, for three different application-spette metrics for forgetting that we discuss in the next section.
model. our work, we take a practical
Unlearning methods Cao and Yang 2015 introduce unlearning and provide an exact forgetting algorithm for statistical query learning. [Bourtoule et al. 2021 proposed training framework by sharding data and creating multiple models, enabling exact unlearning of certain data partitions. Ginart et al. 2019 introduce probabilistic definition for machine unlearning inspired by Differential
5

---

Privacy (Dwork et al. 2014], and formed the origin of the idea of the model indistinguishabiity goal. More recently. Guo et al. 2019). Izzo et al. (202T) Sekhari et al.] [202T] built upon this framework and introduced unlearning methods that can provide theoretical guarantees under certain assumptions. Further, Huang and Canonne [023] provide bounds for 'deletion capacity' in the DP-based unlearning methods using the group property of DP. [Graves et al.) 2021] proposed two methods applicable to deep networks. The first randomly re-labels the forget set and retrains for a small number of updates, while the second stores the index of examples that participated in each batch and the per-batch parameter updates, and unlearns by 'undoing' the affected updates. This requires a lot of storage and can also lead to poor proxy of the model that would have been obtained had those parameter updates not happened in the first place, especially for larger sequences. [Golatkar et al. [2200a] proposed an information -teeoortton- procedure for removing information about D, from the weights of a neural network and [Golatkar et al. 2020b 2021 propose methods to approximate the weights that would have been obtained by unlearning via linearization inspired by NTK theory [Jacot et al. 2018) in the first case, and based on Taylor expansions in the latter. However, [Golatkarr eet al.] 2020a scale poorly with the size of the training dataset, as computing the forgetting step scales quadratically with the number of training samples. [Golatkar et al. [022]] addresses this issue, albeit under different restrictive assumption: that large dataset is available for pre-training that will remain 'static', in that no forgetting operations will be applied on it; an assumption that is not met in many important applications (e.g. healthcare, medical images). Some more recent works propose modifications to the original model training, to make the resulting model more amenable to unlearning. Thudd et al. [002a]] propose regularizer to reduce the 'verification error', which is an approximation to the distance between the unlearned model and a retrained-frome from-scratch model. The goal is to make unlearning easier in the future. Additionally, they propose a single gradient update that reverses the effect of D This regularizer, however, may have a negative effect on the model's s performance, and the single gradient unlearning step only removes the effect that DD has had in the first iteration of training. [Zhang et al. 2022] present training process that quantizes gradients and applies randomized smoothing. This process is designed to make unlearning unnecessary in the future and comes with certifications under some conditions. However, a deletion request that causes too large a change to the data distribution (e.g. if doing class unlearning), may exceed the 'delction budget' and invalidate their assumptions.
Related research areas Differential Privacy (DP) (Dwork et al. 2014] and Life-Long Learning (LLL) PParisi et al., 2019] have shared goals with unlearning. DP seeks to ensure that the trained model doesn't i store information about any training instance; strong goal that is difficult to achieve for deep neural networks [Abadi et al.) 2016). In contrast, in unlearning we only desire to remove the influence of the training instances in the forget set, which can be seen as a relaxation of the same problem. LLL seeks to continuously update a model to solve new sets of tasks, without previously-learnee tasks. The notion of forgetting task, though, is
"catastrophically forgetting' distinct from that of forgetting training example: the performance on task can degrade, without having removed the influence of particular examples from the model weights.
Teacher-student and contrastive methods The teacher-student methodology has been used in a different contexts, e.g. self-supervised (Grill et al. 2020 Chen and He 202T] and cemi-supervised learning [Xie et al. 2020). Knowledge Distillation Hinton et al.) 2013 can also be thought of as a teacher-student approach with the additional desire for compression, where the student chosen to have a smaller architecture than the teacher. In addition, our training objective has a contrastive flavour [Chen et al. 2020 He et al. 2020]. due both pulling the student close to the teacher, and pushing it away from it, for different parts of the training set. For unlearning, [Kim and Woo| 2022 Wu et al., 2022] propose methods with a two-stage pipeline: 'neutralization' / forgetting followed by a 'retraining' / restoring, that restores performance that may be lost in the first phase. Both use distillation only in the second phase on only the retain set. In contrast, SCRUB employs a single phase where we carefully design a teacher-student formulation to both encourage forgetting the requested data and retaining the other knowledge, simultaneously. Most similar to our work in that regard is the method of [Chundawat et al.] 2022 that employs related but different teacher-student objective that encourages moving close to a 'Bad Teacher' for the forget set (we thus refer to this as 'Bad-T"). Moving close to different, 'bad', teacher for forget examples is fundamentally different from our goal of moving away from the (single) teacher as in SCRUB. We find that Bad-T forgets poorly compared to SCRUB in all applications, and degrades model quality.

---

ResNet
Medel
IC test (1)
IC ICraiiin oneer ())
<id
Pyt test crnar
Fig retain crne (1)
IC (3)
Ki ())
Figt tos cror (
sid
For rotain crnor
sil
Retrain
24.0
1.5
0.0
0.0
15.3
4.2
0.0
0.0
19,0
1.3
0.0
0.0
11.3
4,6
0.0
0.0
Orignnl
56.0
3,0
0.0
0.0
91.0
7.9
0.0
0.0
49.0
4.8
6.0
10.39
80.2
12.6
5.0
10.4
Finetun
52.0
9.1
0.0
0.0
79.3
10.1
0.0
0.0
43.7
7.3
0.0
0.0
62.3
16.0
0.0
0.0
NegCinall
47.5
5.3
00.
0.0
80.0
13.5
0.0
0.0
42.3
11.3
0.0
0.0
62.0
22.6
0.0
0.0
CT*
5488
2,0
0.0
0.0
85.3
7.0
0.0
0.0
47.8
5.8
0.0
0.0
76.3
14.
0.0
0.0
BU-k
47.0
4.8
8.3
4.7
63.3
9.7
3.7
255
59.5
5.2
18.3
6.7
66.7
15.6
19.7
10.4
Ficher
51.5
7.
96.3
9.5
79.0
3.6
900
79
56.8
8.7
51.7
14.0
76.1
15.5
17.7
11.5
NTK
87.5
40
0.0
0.0
52.0
10.6
0.0
000
16.7
4.1
3.0
533
54-1
9.0
3.0
52
Rul-T
47.3
9.7
6.0
6.2
27.0
8.7
0.3
0.6
39.3
8.2
3.0
100
25.7
5.0
1.
1.0
SCRUB
19.0
39
0.0
0.0
19.7
7.5
0.0
0.0
26.8
4.4
0.0
0.0
18.0
11.1
0.0
0.0
Table 1: RC-application results: SCRUB is the strongest performer in terms of resolving class
confusion. These results on CIFAR-5 with ResNet and AlI-CNN, where confused 50 samples of class 0 by mislabeling them as class 1, and 50 of class 1 by mislabeling them as class
Experiments
5.1 Experimental setup
Overview of applications and metries We consider three sets of forget-quality metries, for dif- ferent applications. First, we consider the scenario of Removing Biases (RB) where we desire the highest possible forget set error (reflecting that we want the model to never predict the associated labels of the forget set, since those capture an unintended behaviour / bias). Next, inspired by Goel al. [022], we consider unlearning for Resolving Confusion (RC) between classes, caused by portion of the model's 's original training set being mislabelled. Finally, we consider a scenario of data deletion for User Privacy (UP). In this case, we use Membership Inference Attacks (MIAs) to measure success (including the first, to our knowledge, adaptation of the state-of-thh-aar LiRA MIA (Carlini al., 2022 to unlearning), where unlearning succeeds if the attacker unable to tell apart forgotten examples from truly unseen ones. In all cases, aside from achieving the application-spectie forgetting goal, we also measure utility (performance on D, and a test set D₂) as an additional metric.
Experimental details We utilize the same two datasets from previous work:
CIFAR-10 Krizhevsky et al.) 2009) and
&
Lacuna-10 [Golatkar al.] 2020a]. which is derived from VGG-Faces [Cao and Yang 2015). and the same All-CNN (Springenberg et al. 2014) and
architectures:
Original
Pevetanting
Kolber
AYT
Hegporet
one
Redt
scrup
Oegiival
Rovetniing)
meppicedd
One
sctuu
ResNet-18 18 [He et al.) [2016]. For consis- tency with previous work, we pre-train the original model on CIFAR-I00 / Lacuna- 100 for the CIFAR-I0 Lacuna-10 ex periments, respectively [Golatkar al. 2020a b). We run each experiment with random seeds and report the mean and standard deviation.
(a) Small-scale.
(b) Large-scale.
Figure 2: The number of times each method was top performer in forgetting in RB-application. A model is top-torfooeeeee pepp-erformrr if its 95% confidence interval overlaps with that of the best mean. Small-scale counts are over (ResNet, All-CNN) (CIFAR, Lacuna) (4 total), large- scale additionally x (class, selective) (8 total). SCRUB is the most consistent pop-performer.
We consider two settings throughout the pa- per: small-scale and large-scale. The for- mer allows comparisons with NTK, which doesn't 't scale beyond it. For the small-scale, we exactly follow the setup in [Golatkar et al. 2020b that uses only 5 classes from each of CIFAR and Lacuna ('CIFAR-5' / 'Lacuna-5"), ) with 100 train, 25 validation and 100 test examples per class. The forget set contains 25 examples from class 0 (5%). For the large-scale we exactly follow the setup in [Golatkar al., 2020a] that uses all 10 classes of each of CIFAR and Lacuna, and considers both a class unlearning scenario where the forget set is the entire training set for class (10%), as well as a selective unlearning one where 100 examples of class 5 are forgotten (0.25% in CIFAR and 2% in Lacuna). We note that the Appendix contains complete results for all scenarios (several are ommitted from the main paper due to space constraints) and also contains the complete experimental details and hyperparameters.
7

---

Unlearning algorithms We compare against state-of-the-an approaches as well as various base- lines: Retrain: Retraining from scratch without the forget set. This assumed to not be viable in practice, but we include it as reference. Original: The 'original' model trained on all of 'D without any unlearning. Finetuning: Finetunes the original model on data from the retain set D,-. +egGrad+: Similar to Finetuning, starts from the original model and finetunes it both on data from the retain and forget sets, negating the gradient for the latter. Previous work considered a weaker baseline that only trains the forget set, with negative gradient (NegGrad). We tune the stronger NegGrad+ to achieve a good balance between the two objectives in the same way as SCRUB. Fisher Forgetting [Golatkar al. 2020a]. NTK Forgetting [Golatkar et al. 2020b]. Catastrophic Forgetting-k k (CF-k) and Exact Unlearning-K k (EU-k) [Goel et al. 2022]: they freeze the first k layers of the original model and either finetune the remaining layers on D, (CF-k) or train them from scratch on D, (EU-k). Bad-T: the concurrent teacher-student method of [Chundawat et al.) 2022 We utilize the public code for the NTK and Fisher for correctness. We implement Bad-T ourselves (no code was available).
5.2 Unlearning for Removing Biases (RB)
In RB, we desire to achieve the highest possible error on D,, without hurting the error of D,- and D Some example scenarios are removing biases from trained models, outdated or incorrect information. maximal forget error on Dy this case desirable as it reflects that we want the model to never make the predictions that were deemed harmful or no longer valid or accurate.
We report the results for this scenario in Figures and 2 Appendix for all architectures, datasets and baselines. As shown in Figure SCRUB is by far the most consistent method in terms of achieving high forget quality (highest forget error), while doesn't 't hurt retain and test errors.
We also report complete results in the
5.3 Unlearning for Resolving Confusion (RC)
In this section, following Goel et al. (2022), study unlearning for resolving confusion between two classes that the original model suffers from due to a part of its training set being mislabelled. Specifically, place all and only the mislabeled training examples in the forget set. A successful unlearning algorithm would thus entirely resolve the model's confusion. In more detail, the setup is: 1) Mislabel some portion of the training dataset (we mislabeled examples between classes O and 1 of each of CIFAR-5 and Lacuna-5), 2) Train the 'original model' on the (partly mislabeled) training dataset, and 3) Unlearn with the confused examples as the forget set.
We consider the following metrics inspired from Goel et al. [2022] that are variants of the model's error on examples from the confused classes (lower is better for both). We outline them below and define them formally in the Appendix.
Interclass Confusion IC-ERR (e.g. IC test error, IC retain error). This counts mistakes where an example from either of the two confused classes was incorrectly predicted to belong to any other class.
FGT-ERR (e.g. Fgt test error, Fgt retain error). This metric counts only misclassification between the confused classes, i.e. an example of class O being predicted as belonging to class 1, or vice versa.
We observe from Table that, across the board, SCRUB is by far the most consistent method in terms of eliminating class confusion without damaging the quality of the model (on retain and test sets).
5.4 Unlearning for User Privacy (UP)
Next, we consider a privacy-critical application: the deletion of the data of a user exercising their right-to-be-forgolie We use Membership Inference Attacks (MIAs) as the metric where the goal is that, after applying unlearning, an attacker can't 't tell apart examples that were unlearned from those that were truly never seen, thus protecting the privacy of the user requesting deletion.
While previous unlearning papers have reported MIA results [Golatkar et al., 2020a], those MIAs are simple and far from state-of-the-art MIAs used in privacy and security papers. Indeed, it's not trivial to adapt such MIAs to the unlearning protocol. To initiate bridging this gap, we report results on two MIAs: 1) simple one that is closer to the typical MIAs used in the unlcarning literature, and
8

---

40
40
N06
50
50
(a) Forget 100 examples.
(b) Forget 150 examples.
(c) Forget 175 examples.
(d) Forget 200 examples.
Figure 3: MIA results for different sizes of forget set on CIFAR-10 with ResNet. Error bars show 95% confidence intervals. Rewinding is most useful for smaller forget sizes (see the discussion in Section 5.4 SCRUB+R successfully defends MIAs, comparably to the Retrain oracle.
2) the first, to our knowledge, adaptation of the state-of-the-art LiRA attack (Carlini et al. 2022 to the unlearning protocol. We describe both high-level here and in detail in the Appendix.
Basic attack We train a binary classifier (the 'attacker') on the unlearned model's losses on forget and test examples for the objective of classifying 'in' (forget) versus 'out" (test). The attacker then makes predictions for held-out losses (losses the attacker wasn't 't trained on) that are balanced between forget and test losses. For perfect defense, the attacker's 's accuracy is 50% indicating that it is unable to separate the two sets, marking a success for the unlearning method.
We present Basic MIA results in Table 2 for CIFAR-I0. We observe that SCRUB+R is a strong performer. EU-k can also perform strongly in this metric, though not as consis- tently. We also observe that rewinding was only triggered once, for selective unlearning. Recall that, in both selective and class unlearning, the forget set contains examples coming from sin- gle class, but in the former, it doesn't t contain all examples of that class; the rest are in the re- tain set. Thus, we indeed expect that rewinding would be more useful for selective unlearning: the smaller the portion of class in the forget set, the larger the remainder of that class in the retain set and consequently, the better expect the model to generalize on held-out examples of that class (lower error), making it in turn more likely to need to rewind, order to also lower the forget set error commensurately. In Figure we plot results for different variants of selective unlearning, with different number of examples in the forget set (from the same class) and indeed find that for smaller numbers, rewinding is triggered. Our take-away is that, when needed, rewinding can substantially improve SCRUB's MIA results.
ResNet
AIL-CNN
Selective
Selective
Model
muan
sid
mecm
sd
merm
sld
mean
sid
Retrain
49.33
1.67
$4.00
1.63
55.00
4.00
18.73
0.21
Original
71.10
0.67
65.33
0:47
66.50
0.50
71.40
0.70
75.57
0.82
1.27
NegGrad
1.70
0.00
1.92
CF-k
75.73
0.34
65.00
0.00
69.00
2.00
72.93
1.06
BL1-K
54.20
2.27
53.00
3.27
66.
3.5g
51.60
1.22
Bad-T
54.00
1.10
59.67
4,19
63.4
1.2
77.67
4.11
SCRUB
52.20
1.71
78.00
2.45
$2,00
0.00
54.30
2.24
SCRUB-R
52.20
58.67
1.89
0.00
2.24
Table 2: Basic MIA results (for UP): SCRUB successfully defends MIA. Note the rewinding prucedure SCRUB+R was triggered only once; discussion in Section and Figure 3 In the Appendix, we also show that the forget error of SCRUB(+R) close to that of Retrain, desired.
LiRA-for-umleariing attack In the standard privacy setting, the LiRA attacker trains a large number of "shadow models' [Carlini et al., [022], for which it controls which examples are in the training set each time (by construction). To then predict the membership status of 'target example', estimates two Gaussian distributions: the distribution of confidences of that example under shadow models that trained on it, and the distribution of its confidences under shadow models that didn't 't It predicts that the target example is 'in' if the likelihood of the former is larger than that of the latter.
We propose the first, to our knowledge, adaptation of LiRA for unlcarning. This is a strong attack where we allow the attacker knowledge of the unlearning algorithm. Concretely, for each shadow model, the attacker also produces 'shadow unlearned' model by applying the given unlearning algorithm several times, using a large number of forget sets (similar Chen al.] [202T). Now, for each 'target example', this setup allows the attacker to estimate a different pair of Gaussians: the distribution of (confidences of) that target example under models where was forgotten, and as before, under models where it was not seen. The attacker predicts the example was forgotten if its likelihood under the former is larger than under the latter. We present all details and findings in the Appendix and observe that SCRUB+R outperforms the other methods in defending this attack.

---

5,5 Overview of results and take-aways
Finetuning is a poor method: it may retain model utility but fails to forget. The previous state-the- art NTK and Fisher models aren't among the top-performers either. The former performs poorly across the board in terms of forgetting and doesn't I scale beyond small datasets. The latter sometimes performs well in RB in terms of forgetting, though not always and it very slow (Figure IT. exceeding the runtime of Retrain; as also observed in [Goel et al. 2022]. NegGrad+, our proposed enhancement over the NegGrad baseline, is strong bascline in terms of achieving balance between forgetting and utility (it performs well in several settings in terms of RB, albeit not as consistently as SCRUB) and we encourage future work to report this baseline, though SCRUB outperforms it, especially for RC. CF-k inherits the performance profile of Finetune: it maintains utility but forgets poorly. On the other hand, EU-k more reliably forgets (performs strongly in UP, and often in RB), which is expected relative to CF-k due to retraining part of the network, but SCRUB outperforms it significantly in terms of RC and is more consistently top performer in RB. A notable failure case that we discovered for EU-K in RB is selective unlearning (notice the contrast between EU-k's ' forget error between e.g. [ This finding may speak where class instance information is stored in neural networks and warrants further investigation. Bad-T is very efficient, but performs poorly in terms of forgetting, across all applications and also damages model utility. Overall, SCRUB by far the most consistent in successfully forgetting under different metrics without hurting utility and being efficient.
Figures Ia and
In the Appendix, discuss limitations of our work, broader impact, future work and present full experimental results and all details including hyperparameters and pseudocode.
6 Discussion and Conclusion
Despite substantial recent attention Triantafillou et al. 2023 unlearning is a young area of research.
In this work, propose SCRUB, unlearning method that is unbound from limiting assumptions and poor scalability limits. SCRUB(+R) is found empirically to perform very strongly: it is by far the most consistent method in achieving good forgetting quality with respect to different application- dependent metrics, while incurring only minimal utility degradation. As such, we believe that SCRUB fills the important need for scalable unlearning methods that perform well in practice on several metrics of interest. However, important questions remain open for future work.
Crucially, despite progress in this direction, a well-established formal definition of the problem of unlcarning remains clusive, and consequently we also lack well-established metrics for measuring (theoretically or empirically) the quality of unlearning algorithms. In this work, focus on empirical evaluation with respect to different application-dependent metrics for forgetting quality that we believe relevant in real- wallworld applications. However, as the unlearning community matures, we hope that success criteria for (different applications of) unlearning will be formalized and standardized.
Further, important limitation of SCRUB is the absence of theoretical guarantees, making it perhaps ill-suited for certain application scenarios. While methods that come with guarantees exist, they either aren't 't applicable to deep neural networks, or find through our empirical analysis that they perform poorly and don't 't scale beyond small datasets. Therefore, future work that theoretically studies scalable methods like SCRUB would be very valuable. In absence of that, continuing to study and better understand the trade-offs offered by different methods is an important direction.
On the practical side, we hope that future work continues to push the limits of scalability and studies unlearning in larger models, different architectures, different training objectives like self supervueeee learning and different domains and modalities, including foundation language models.
Broader Impact While recent advances in deep learning represent exciting opportunities for our community, they also come with great responsibility. As researchers, we are responsible for under- standing and mitigating the issues associated with the widespread use of deep learning technology. Machine learning models may carry harmful biases, unintended behaviours, or compromise user privacy. Our work is intended to take a step in addressing these issues via a post-processing 'unleam- ing' phase that makes progress over previous solutions in practice, as we show through an extensive empirical investigation. However, SCRUB does not come with theoretical guarantees and we can not prove that applying SCRUB perfectly mitigates those issues, so caution must be taken in practice and proper auditing of machine learning models is critical.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Fabian Pedregosa for countless insightful discussions, especially on the topic of membership inference attacks.
References
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-3 018-,18,77 2016.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks. In International conference on machine learning, pages 233-242.
PMIR, 2017.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Chloquette-Chod Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lic, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159, IEEE, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlcarning. In 2015 IEEE Symposium Security and Privacy, pages 463-480. IEEE, 2015.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium Security and Privacy (SP), pages 1897-1914. IEEE, 2022.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang, When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 896-911, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2021.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using incompetent teacher. arXiv preprint arXiv:220S.08096. 2022.
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211 11:2:1--407, 1-407, 2014.
Simone Fabbrizzi, Symeon Papadopoulos, Eirini Ntoutsi, and Ioannis Kompatsiaris. A survey bias in visual datasets. Computer Vision and Image Understanding, page 103552, 2022.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Shashwat Goel, Ameya Prabhu, and Ponnurangam Kumaraguru. Evaluating inexact unlearning requires revisiting forgetting. arXiv preprint arXiv:2201.06640 2022.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 0304-9312. 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In European Conference on Computer Vision, pages 383-398. Springer, 2020b.

---

Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 792-801, 2021.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516-11524 2021.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv: 1,03030, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and patlern recognition, pages 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0253/1 2(7), 2015.
Yiyang Huang and Clément L. Canonne. Tight bounds for machine unlearning via differential privacy. arXiv preprint arXiv:2309.00 .00866, 2023.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008-2016. PMLR, 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium Security and Privacy (SP), pages 19-35. IEEE, 2018.
Junyaup Kim and Simon S Woo. Efficient two-stage model retraining for machine unlearning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4361-4369, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alessandro Mantelero. The eu proposal for general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 29(3):229-235, 2013.
Seth Neel, Aaron Roth, and Saeed Malvjji-aa Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pages 931-962. PMLR, 2021.
Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103. 14749, 2021.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances Neural Information Processing Systems. 34:18075-18000 2021.
12

---

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1412.6866, 2014.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In 2022 IEEE 7th European Symposium Security and Privacy (EuroS&P), P), pages 303-319. IEEE, 2022a.
Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22), pages 4007-4022, 2022b.
Eleni Triantafillou, Fabian Pedregosa, Jamie Hayes, Peter Kairouz, Isabelle Guyon, Meghdad Kur- manji, Gintare Karolina Dziugaite, Peter Triantafillou, Kairan Zhao, Lisheng Sun Hosoya, Julio C. S. Jacques Junior, Vincent Dumoulin, Ioannis Mitliagkas, Sergio Escalera, Jun Wan, Sohier Dane, Maggie Demkin, and Walter Reade. Neurips 2023 machine unlearning, 2023. URL
https://kagg http:///cooppeiiiiiooeeeeeriiiia 202p machine-unlearrnnngg unlearning
Chen Wu, Sencun Zhu, and Prasenjit Mitra. Federated unlearning with knowledge distillation. arXiv preprint arXiv:2201.09444 2022.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves ImageNet classification. In Conference on Computer Vision and Pattern Recognition. pages 10687-10698, 2020.
Li Yang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415:295-316, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 253-261, 2020.
Zijie Zhang, Yang Zhou, Xin Zhao, Tianshi Che, and Lingjuan Lyu. Prompt certified machine unlearning with randomized gradient smoothing and quantization. Advances in Neural Information Processing Systems, 35:13433 -3455,, 2022.
13

---

7
Appendix
This Appendix contains the following sections:
Section g
Limitations and Future Work
Section
Experimental Details and Pseudocode
Section 10
Formal Description of Metrics
Section
Membership Inference Attacks: Description and Additional Findings
Section 12
Ablations and Sensitivity Analysis
Section
Larger-scale settings
Section 14
Additional Results for Removing Biases (RB)
Section
15
Additional Results for Resolving Confusion (RC)
Section
16
Additional Results for User Privacy (UP)
8 Limitations and Future Work
SCRUB has shown impressive results in terms of being consistently a top-performer in terms of unlearning, with minimal drop in performance compared to previous works. However, SCRUB has limitations that we hope future work will address.
A significant step for future work is to develop thcoretical guarantees for the gains provided by our methods. We opted to focus on an empirical approach for the following reasons: First, while theoretical guarantees abound for linear models, deep networks pose additional significant challenges. Second, methods accompanied with theoretical guarantees suffer from practical limitations with respect to accuracy and/or scalability. For these reasons we opted to approach the problem from a practical standpoint, pushing the envelope by developing unlearning algorithms which are top performers across many important different application scenarios, different evaluation metrics, different architectures and datasets. We look forward to future work that strives to strike compromise between effective unlearning, good performance, scalability, and theoretical insights.
Another limitation of SCRUB is the difficulty and instability associated with tuning the min-max objective, shown in the literature e.g. for GANs. For instance, this can lead to oscillating behaviour, as we show in Figure algorithm that works well, showing consistently improved results over prior work, but there is room for improvement on this front for future work.
We remedy this to a large extent in practice by providing a practical
SCRUB's rewinding procedure also has limitations. We find in practice throughout all of our experiments that it can help to substantially increase the success SCRUB's s defense on MIA in scenarios where the forget error obtained by SCRUB the end of unlearning 'too high'. However, a different failure case which can also appear is that SCRUB's s forget error at the end of training is "too low'. This can happen due to the way in which we tune hyperparameters, which is designed to not harm the retain and validation performance too much and thus can in some cases lead to 'premature stopping' before the forget error reaches the same level reference point for how high it would be the model had truly never seen those examples. We highlight that addressing all possible issues that arise in all scenarios and provide an unlearning algorithm that performs strongly across the board is extremely challenging. The fact that we have observed failure cases for each algorithm, be it SCRUB or other baselines, indicative of the extensiveness of the experimentation we conducted. Our work has made important strides in designing consistently strong-performing unlearning methods and we look forward to future contributions in this direction.
We hope that future work also continues to push the limits scalability. We believe that our work constitutes important step in this direction. However, the datasets and models consider aren't 't too large, in order to allow comparisons to previous works that would not be feasible to run for larger scale experiments. An interesting topic of future work is investigating the interplay between SCRUB and other scalable algorithms like NegGrad+ with increasing amounts of scale.
Another really interesting future work direction is to investigate how different unlearning algorithms interact with different architectures, like Transformers, and loss functions, like self-supervised learning.
14

---

Experimental Details and Pseudocode
Datasets. We have used CIFAR- and Lacuna- 10 datasets for evaluation purposes. CIFAR-10 10 consists of 10 classes with 60000 color images of size 32 32. In our experiments, the train, test, and validation sizes are 40000, 10000, and 10000 respectively. Lacuna-10 10 is dataset de- rived from VGG-Faces [Cao and Yang 2015]. We have followed the same procedure described in [Golatkar et al. 2020a to build Lacuna. We randomly select 10 celebrities (classes) with at least 500 samples. We use 100 samples each class to form the test-set, and the make the train-set. All the images are resized to 32 x 32. We also CIFAR- 100 and Lacuna- 100 to pre- train the models. Lacuna- 100 is built in a similar way Lacuna- 100 and there is overlap be- tween the two datasets. We have not applied any data augmentation throughout the experiments.
Algorithm SCRUB
Require: Teacher weights wo
Require: Total max steps MAX-STEPS
Require: Total steps STEPS
Require: Learning rate
Th w
i i 0
repeat
i < MAX-STEPS then
"" DO-MAAX-EOOH(rr
end if'
"h DO-MIN-EPOCH((""
until << STEPS
Small-Scale datasets. We followed the same procedure as decribed in [Golatkar et al. 2020b] to create the small versions of CIFAR-10 and Lacuna- 10, namely CIFAR-5 and Lacuna-5. To this end, we take the first 5 classes of each dataset and randomly sample 100 images for each class. We make the train and test sets by sampling from the respective train and test sets of CIFAR-10 10 and Lacuna- 10. We also make 25 samples from each class from the train set to create the validation sets.
Models. We use the same models with the same architectural modifications [Golatkar al.) 2020a/b). For AlI-CNN, the number of layers is reduced and batch normalization is added before each non-lincarity. For Resnet, ResNet- 18 ar- chitecture is used. For small scale experiments, the number of filters is reduced by 60% in each block. For the large-scale experiments, the exact architecture has been used.
Algorithm 2 DO-MAX-EPOCH-
Require: Student weights "
Require: Learning rate
Require: Batch size B
Require: Forget set Dy
Require: Procedure NEXT-BATCH
NEXT- BEXT-BATCH(DJ.B B) repeat
Pretraining. Following the previous work for consistency, we apply pretraining. Specifically, for CIFAR datasets, we have pretrained the mod- els on CIFAR-100. For Lacuna, trained the models on Lacuna- 100. We pretrain the models for 30 epochs using SGD with fixed learning rate 1. Cross- Pross-Entropy loss function, weight decay 0.0005, momentum 0.9, and batch
" w" +eV w" Σ d(zy;=") Ey
have pre-
b- NEXT-BATCH(D,, until b
size 128.
Baselines. 'Original' is the model trained on the entire dataset D. For 'Retrain', we train the same architecture on D,, with the same hyperparameters used during training of the original model. For 'Finetune', we fine-tune the 'original' model on D, for 10 epochs, with fixed learning rate of 0.01 and deight-decay 0.0005. For 'NegGrad+', we fine-tune the 'original' model using the following loss:
Del Σ
p1 Σ Ø=1
((w) = B
(1-
(f(o,;
(4)
D
Where 3 E [0, 1]. We have tuned / get a high forget-error while destroying retain-erro error and validation-error. For small-scale experiments, 8 = 0.95 and we have trained for 10 epochs, with SGD, 0.01 and 0.1 weight-decay. For large-scale experiments 5 epochs, with SGD, 0.01 Ir, and 0.0005 weight-deay. Please note that small B result in explosion quickly. For 'CF-k', we freeze the first layers of the network and finetune the rest layers with D,.. We use the same setting as 'Finetune' bascline. For 'EUkk' freeze the first k layers, and
0.9999 and we have trained for
15

---

model
dataset
unlearning-tye type
forget-set bs
retain-set bs
max steps
min steps
CIFAR-I0
class
512
128
3
CIFAR-10
selective
16
64
5
5
Lacuna-Il
class
128
128
5
ResNet
Lacuna-10
selective
32
32
4
4
CIFAR-5
selective
32
32
10
10
Lacuna-5
selective
32
32
5
10
CIFAR-10
class
512
256
3
4
CIFAR-10
selective
16
64
5
5
Lacuna-10
class
32
32
4
4
All-CNN
Lacuna- 10
selective
8
32
2
4
CIFAR-5
selective
16
32
5
10
Lacuna-5
selective
32
32
5
10
Table 3: SCRUB's hyperparameters for each experiment
re-initialize the weights of the remaining layers and retrain them with D,. As all the models pretrained on larger datasets, for re-initializing we use the weights of the pretrained models. For "EU-K' the same settings the 'Retrain' baseline. In both 'EU-k' and 'CF-k' baselines, for both ResNet and All-CNN we freeze all the layers except for the last block of the network. For Resnet the last block is block4 and for All-CNN, the last block of layers the 9th sequential block. For Bad-T, we follow the specifications given in Chundawat et al. [0222] with possible tuning of the parameters in different settings to get the highest forget-error without damaging retain-ers error. More specifically, for all models we perform one epoch of unlearning using Adam optimizer, and temperature scalar of 4. Also, we use the whole retain-set compared 30% reported in their paper as we empirically observed that using only 30% of retain-set for Bad-T yields high test errors.
SCRUB pseudocode and parameters. We train SCRUB using Algorithm[] [ Throughout the experiments, we the parameters get high forget-error while retaining the retain and validation error of the original model. We use the same optimizer for both min and max steps. We observed that for small- mcall-ccale settings 'Adam' optimizer works better, while in large-scale set- tings both 'Adam' and 'SGD' could used. For all experiments, we initialize the learning rate at 0.0005 and decay by 0.1 after number of min and max steps. Decaying the learning rate is cru- cial to control the oscillating behaviour of our min and max optimization. We apply weight decay 0. for small-scale setting and 0.0005 for large scale experiments, with momentum of 0.9. Finally, we use different batch sizes for the forget-set and the retain-set to control the number of iteration in each direction, i.e the max and the min respectively. We report these in Table
Algorithm 3 DO-MIN-EPOCH
Require: Student weights "h
Require: Learning rate
Require: Batch size B
Require: Retain set D,
Require: Procedure NEXT-BATCH
+ BEXT-BATCH(D,,,
repeat
" w"
eVe
1
Σ
ad(z);"")
[b]
("): b- NEXT-BATCH(D,,,) until
System specification. For scale-up experiments, the code is executed in Python 3.8, on an Ubuntu 20 machine with 40 CPU cores, a Nvidia GTX 2080 GPU and 256GB memory.
10 Formal Description of Metries
In this section, we give more details and mathematical definitions of the metrics that we use throughout the paper. We first mathematically define the forget, retain and test errors, and then other application- dependent metries, for Resolving Confusion (RC) and User Privacy (UP).
16

---

Forget, retain and test errors Here, we define the retain error, forget error and test error. Let Dr., Dj and D, denote the retain and forget portions of the training dataset, and a test dataset of heldout examples, respectively. We define error (Err) as follows:
Err(D) =
Σ (.DD
L(arg max(f(z;; == yil
(5)
D
where f. parameterized by w the neural network model (comprised of a feature extractor followed by a softmax classifier layer), arg max(f w)) is the label that the model thinks is most likely for example 24. and [[x]] the indicator function that returns 1 if is True and 0 otherwise.
Based on the above, the retain error, forget error and test error are computed as Err(D,), Err(D,) and (D,), respectively.
Metries for Unlearning for Resolving Confusion (RC) We now define the class confusion metrics inspired by Goel et al. [022]. Specifically, we explore scenario where the forget set has confused labels (e.g. for two classes A and B, examples of A are labelled as B, and vice versa). The idea here is that, because mislabelled examples only present in the forget set, successful unlearning (removing the influence of the forget set) would lead to a model that is not at all confused between classes A and B.
In more detail, the setup we follow is: 1) We first mislabel some portion of the training dataset (we mislabelled examples between classes 0 and 1 of each of CIFAR-5 and Lacuna-5 in our experiments), 2) train the 'original model' on the (partly mislabelled) training dataset (it has mislabelled examples for classes 0 and 1 but correct labels for the remaining classes), 3) perform unlearning where the forget set contains all and only the confused examples. Given this, the goal for the unlearning algorithm is to resolve the confusion of the original model.
We consider the following metries (using terminology consistent with Goel al. 2022 ) They are presented in order of decreasing generality, and increasing focus on measuring degrees of confusion between the two classes considered.
Error (e.g. test error, retain error, forget error). This counts all mistakes, so anytime that an example of some class predicted to be in any other class, will be counted. These are the same metrics that we use for the rest of the paper (see Equation 5). For test and retain error, lower is better, whereas for forget error, higher is better.
Interclass Confusion IC-ERR (e.g. IC test error, IC retain error). This counts only mistakes that involve examples from the confused classes A and B. Specifically, it counts instances of any example of class A being predicted to be in any other class, and similarly for class B. Compared to Error, this metric is more focused towards understanding the result of the introduced confusion, since it only considers cases that relate the confused classes. A successful unlearning method would make no such errors, so lower is better for each of IC test error and IC retain error.
FGT-ERR (e.g. Fgt test error, Fgt retain error). This metric counts only misclassification between the confused classes and B. Here, mistake of an example of class A (or B) being predicted to be in class other than A or B will not be counted. Only mistakes of an example of class A being predicted to be in class B, and vice versa, are counted. This is the most focused metric that directly measures the amount of remaining confusion between the two classes in question. A successful unlearning method would make no such errors, so lower is better for each of Fgt test and Fgt retain.
More formally, Error is the same as defined in Equation and FGT-ERR: We denote by C the confusion matrix for model parameterized by the dataset D. and let A denote the part of the dataset D that belongs to class A. So, for example Dr denotes the part of the retain set D, that belongs to class A, and the entry C of the confusion matrix stores the number of times that a sample belonging to class A was (mis)classified as belonging to class B in the dataset D by the model parameterized by w. Then, we have:
5 Let us now mathematically define IC-ERR
IC- IC-ERR (D, W; = ΣB CnD + +E D
B.K
(6)
|DAl + +lDl
17

---

where k + A, B.
So, for example, the 'IC test error' column in our tables is computed via IC-ERR(D,, 1;1;;),,; w), where DD denotes the test set, and and are the two classes confused in our experiments. Analogously, 'IC retain error' is computed as IC-ERR(D,, 0, 1; w)
Finally:
Cu.. FGT-ERR A, B; u) = C..D + B,A
That is, FGT-ERR only measures the misclassification between the two confused classes and So, for example, the 'Fgt test error' in our tables is computed as FGT-ERR(D,, 0, w) and analogously 'Egt retain error' is computed as FGT-ERR(D,,, 0, 1; w).
User Privacy (UP) Metries Please see the next section for full details for each of the two Membership Inference Attacks (MIAs) that we use and experimental results.
11 Membership Inference Attacks: Description and Additional Findings
As mentioned in our paper, we utilize two different MIAs: 1) 'Basic MIA' that is similar to the ones typically used in unlearning papers (but far from the state-of-the-an of MIAs used by privacy and security colleagues), and 2) the first, to our knowledge, adaptation of the state-of-the-art LiRA attack [Carlini et al. 2022] to the framework of unlearning ("LiRA-for-uniee --rn----nnn MIA).
In this section, we use the term 'target model' to refer to the model that is being attacked and the term 'target example' to refer to an example whose membership status ('in' or 'out') the attacker tries to predict, based on the 'behaviour' (e.g loss value) of that example under the 'target model'. In both attacks that we consider, the target model is the unlearned model, and target examples are either forget set ('in') or test set (" ('out") examples. The unlearning algorithm successfully defends an MIA if the attacker can't 't tell apart examples that were unlearned (forget set examples) from examples that were truly never seen.
11.1 Basic MIA
Returning previous notation, let (J( w) denote the cross-entropy loss of the unlearned model (a deep network f with weights w") on example with label y. We abbreviate this as (x,, y) from now on; dropping the dependence on f and w".
The attacker a binary classifier that takes as input loss values, coming from either the forget set Dj or held-out test set Dr, and predicts whether the example whose loss value was presented was in the training set of the original model. We train this attacker via supervised learning on a class-balanced labelled training set for this binary problem: Dr Diras {((I,, Va), sf')} where each x, is an example coming either from D, or D., and its binary label y is defined as being O, if x, De and 1 x2 E Dy. Once the binary classifier attacker is trained, we use it to make predictions for a held-out evaluation set of the binary problem: De = {1(x1,, u/)) that also balanced between examples coming from Dy and De, but is disjoint from Dirain-
The attacker succeeds if it achieves high accuracy on Db De meaning that it can tell apart examples that were part of the original training set from those that weren't, t, which marks a defeat for the unlearning model in terms of this metric, since it has 'left traces behind' (in this case, in terms of loss values) and leaks information about membership in the forget set. We consider that an optimal defense against this MIA corresponds to a 50% attack accuracy; that is, no better than randomly guessing whether an example had been trained on. In principle, the Retrain oracle should defend optimally: fact did not train on the forget set, so the forget and test sets are simply two different held-out sets for this model whose loss values should generally be indistinguishable from each other these sets are identically identically-diitriuuueed distributed. We find in our experiments, presented in the main paper, that SCRUB+R is able to defend this MIA comparably to Retrain, and outperforms the other baselines in its ability to consistently do so.
Experimental details In practice, if the distribution of the forget set and the test set are very different from each other, their loss values will be very distinguishable. This means that the binary classifier can tell them apart casily, but without having truly learned to infer membership in the
18

---

29m
18
10-1
Figure 4: ROC curves for the strong LiRA-for-unlesne for-unlearning attack (Area Under the Curve (AUC) is also reported in the legend, for each unlcarning method). Different subplots correspond to different target models (we trained the target unlearned model times for each unlearning method, using different random seeds, and different forget sets). Positives are examples in the forget set, and negatives the test set. true positive means that the attacker correctly identified that an example was forgotten, whereas false positive means that it incorrectly predicted that test example was forgotten. We are primarily interested the area small False Positive Rate [Carlini et al. 2022] and good unlearning method is associated with smaller True Positive Rate, i.e. fewer successes for attacker, especially in the region of interest. We observe that SCRUB(+R) defends the strong HIRA-for-unleanne attack more successfully than the other baselines.
training dataset. This makes the attacker's evaluation unreliable. To circumvent this problem, we ought to pick the held-out test set from the same distribution. More specifically, the forget set is examples from the 'cat' class of CIFARI0 dataset, we use the same class for our held-out test set.
In our experiments, we clip the loss values to range between [-400, +400] to remove anomalies. Also, we use the default Logistic attack model, and perform the evaluation part of each of the 5 folds as the MIA score. Ideally (for perfect defense), this score closest to 50%, indicating that the attacker fails tell apart the forget set from the test set.
classifier of the Python's 's scikit-learn library as our vols-vaaii with 5 random splits. We report the average accuracy of
We present additional results for the Basic MIA attack in Section 16
11.2 LiRA-for-unlearrning attack
In the standard privacy setting, the LiRA attacker (Carlini al. 2022 trains large number of "shadow models' (Shokri et al. 2017]. for which it controls which examples are in the training set each time (by construction). To then predict the membership status 'target example", it estimates two Gaussian distributions, using the shadow models: the distribution of confidences of that example under shadow models that trained on and the distribution of its confidences under shadow models that didn't. t. Then, it computes the confidence of the target example under the target model and it predicts that the target example was 'in' the likelihood the target confidence under the former Gaussian is larger than that under the latter Gaussian.
Adapting LiRA to the framework unlearning is not trivial, and we are not aware this done in any previous work. We propose the first, to our knowledge, adaptation of LiRA for unlearning. This a strong attack where we allow the attacker knowledge the unlearning algorithm. Coneretely, for each shadow model, the attacker also produces a "shadow unlearned model by applying the given unlearning algorithm several times, using large number of forget sets (similar to Chen et al.) (202T). Now, for each 'target example', this setup allows the attacker to estimate a different pair of Gaussians: distribution of (confidences of) that target example under models where was forgotten, and as before, under models where it was not seen. The attacker then computes the confidence of the target example under the target model, and predicts the example was forgotten its likelihood under the former is larger than under the latter.
19

---

Experimental setup: overview We run our LiRA-for-unleamee attack on selective unlearn- ing on CIFAR-10, 10, for the scenario where the forget set has size 200 and comes from class
Errar Plots with Confidence intervals (over 3 runs)
M
63-
For the attacker, we first train 256 "shadow original' models on random splits of half of the CIFAR-I0 training set. Let D de- note the original dataset. To train each shadow model, we split D in half, and use one half to as the 'training set' and the other half the "test set' of that particular shadow model. Then, for each of these "shadow original' models, we run unlearning on 10K different forget sets (for each unlearning method). Specifically, the for-
Attacker:
remor nerror
20-
iamma, aphaa
get set is random subset of 200 examples of class 5, sampled from the training set of the cor- responding 'shadow original' model. After this procedure, for every example in class 5, we se- lect 256 associated shadow models when it was not included in training, and 256 shadow models when it unlearned (i.e. it was in the forget set). After this, the LiRA attack proceeds as normal, where take each in out distribution and apply a likelihood ratio test on an unknown example infer membership.
Figure 5: Sensitivity of SCRUB to y and a. To create this plot, ran SCRUB many times for dif- ferent values of y ((0.1, 0.5, 1, 2, 3)) and a ([O.1, 0.5, 1, 3)). The x-axis is represents combinations of these values. t-error, f-error and r-error refer to test, forget and retain enror, respectively. We find that SCRUB is not very sensitive to these hyper- parameters: the retain values, and there are several different settings to these hyperparameters for which we can obtain the desired results for test and forget errors too.
remains low across
Defender:
We next train the target model that finnllanunng will attack. For this, we begin by training the 'original' model, on (a random split of) half of D. Then, forget set, which is subset of the original model's training set of size 200, coming from class 5, in the same way as was done by the shadow models.
apply the given unlearning algorithm mandomly-sampler
Implementation note We run this attack at a much larger scale than the remaining experiments of the paper (we run 10K unlearning runs, on different forget sets, for each of the 256 'shadow original' models). We do this because the strength of the attacker is heavily dependent on the number of shadow original/unlearned models, and we wanted to benchmark our baselines against very strong attacker. Therefore, to allow better scaling, instead of implementing SCRUB+R rewinding, we implement it through filtering runs of SCRUB that don't 't satisfy the condition of SCRUB+R that the forget set error should be close to the validation error (where, explained in the main paper, this refers to the validation set that is constructed to have the same distribution as the forget set; containing examples of only the same class as the one in the forget set). We used 0.1 as threshold.
Computing the 'confidence' Consistent with [Carlini et al. 2022 the confidence of an example (x, y) (where y denotes the ground-truth class label) is defined as softmax(f(e (z))) words, the confidence is the softmax probability of the correct class. Following Carlini et al. [2022]. we apply logit-scaling to each confidence, to make their distributions Gaussian.
Conclusions and findings Figure 4 plots the ROC curve, showing the False Positive Rate and True Positive Rate of the attacker, in log-log scale. Different subplots correspond to different target unlearned models, each of which was trained with a different random seed, and different retain/forget set split. A successful defense is associated with a smaller Area Under the Curve (AUC); meaning fewer True Positives for the attacker. Carlini et al. [2022] however advocate that the AUC is not good indicator of the attacker's 's strength and, instead, they argue that we should primarily consider the region of the ROC curve associated with very small False Positive Rates. We observe that, especially in that region, SCRUB(+R) is the strongest method in terms of defending our A-for-unlearring attack (and also we observe that SCRUB(+R) has the best AUC too). The improved NegGrad+ baseline that we also proposed in this paper is also strong model in terms of defending this attack.
20

---

100
retain-set fargat
set
emor
retain set
target
walidation sot
spoch
epoch
(a) Only max steps (Equation
(b) SCRUB with insufficient max steps.
70
retain set
relein-set
60
fargat
forget-set
volidation sot
velidstion-sel
20
40
10
0.
25
5.0
3.5
10.0
17.5
15.0
17.5
20.0
10
epoch
opach
(c) SCRUB with too many max steps.
(d) SCRUB.
Figure 6: Illustration of training dynamics of SCRUB variants, on CIFAR-5 with a ResNet model. Performing the right interleaving of min-steps and max-steps is important for achieving a good balance between high forget error and low retain and validation errors.
We found that CF-k is not able improve the privacy of the original model in most cases, while EU-k can sometimes improve but only slightly, and not reliably.
Limitations To stay consistent with previous work on unlearning. as mentioned previously, we turn off data augmentations. Consequently, the 'original model' (before unlearning applied) has overfitted more than a state-of-the-aar CIFAR model would. Indeed, as can be seen from Figure 4 the 'Original' model has poor privacy (the attacker has high True Positive Rate). We note that this the first, to our knowledge, investigation of privacy of unlearning algorithms using strong MIAs, and we hope that future work continues investigate increasingly more realistic scenarios with models closer to the state-of-the-ard and considers unlearning on original models of varying degrees of privacy and generalization ability.
12 Ablations and Sensitivity Analysis
In this section, we illustrate the training dynamics of SCRUB and the importance of different design choices. As reminder, the student is initialized from the teacher and subsequently undergoes an alternating sequence of max-steps and min-steps; the former encouraging the student move far from the teacher on the forget set, and the latter encouraging it to stay close to the teacher on the retain set. We also found it useful perform sequence of additional min-steps after the alternating sequence. We now explore the effect of these decisions.
First, show that performing only max-sieps, by optimizing Equation T is not good solution. Simply pushing the student away from the teacher on the forget set achieves forgetting but unfortu- nately also hurts the retain and validation set performance FFp. Therefore, alternating between max-steps and min-steps is necessary. However, it is important to find the right balance. For instance, as seen Figure 6b) performing too few max-steps leads the unwanted consequence of the forget error dropping. On the other hand, removing the final sequence of only min-steps is also harmful, as shown Figure 6c that trains for a larger number of epochs of an equal number of (alternating) max-stepp ste-s and min-steps without achieving a good balance at any point throughout the trajectory. the other hand, SCRUB (Fi achieves a good balance of high forget error and low retain and validation error simultaneously. We also ablate the cross-entropy term in Equation 3 which provides small but consistent added protection against degrading performance Figure We show additional examples of training dynamics (Figures
21

---

retain-set
effer
eSTOr
forget-set
validation-set
retain-set
validation set
epoch
epoch
(a) Performing only max steps (Equation
(b) SCRUB with insufficient max steps.
forget set
forget-set
validation-set
validation-set
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
10
epoch
epoch
SCRUB with too many steps.
(d) SCRUB
Figure 7: Illustration of training dynamics of SCRUB variants, on CIFAR-5 with AlI-CNN. Per- forming the right interleaving of min-steps and max-sleps is important for achieving good balance between high forget error and low retain and validation errors.
Table 4: Results for class-unlearning. Class-O (400 examples) of Cifar-100 (40k examples). The numbers averaged over runs.
Test error ())
Retain error ({) mean
Forget error (t) mean
Basic MIA
Model
mean
std
std
std
mean
std
Original
35.84
0.76
0.07
0.02
0.00
0.00
61.67
16.20
Retrain
36.92
1.02
0.09
0.06
100.00
0.00
55.67
34.09
NegGrad+
43.16
3.13
6.86
5.16
49.50
12.79
64.67
18.07
Bad-T @e@Teppoooo @eppch1
44.14
1.84
8.78
2.08
63.83
24.64
61.67
20.57
Bad-T epoch10
43.33
1.27
7.74
3.21
85.33
4.18
63.33
28.15
SCRUB+R
37.60
1.18
0.12
0.10
100.00
0.00
57.33
28.99
SCRUB 38.27
3.08
0.78
2.47
100.00
0.00
48.33
12.25
Finally, we also investigate the sensitivity of SCRUB's 's results on the y and hyperparameters, in F We find that SCRUB is not very sensitive to these hyperparameters: the retain error remains low across values, and there are several different settings to these hyperparameters for which we can obtain the desired results for test and forget errors too.
13 Larger-scale Settings
In this section, scale up to an even larger setting, by choosing a larger model and performing larger classification task. The experiments here are conducted on VGGI6+BN with almost 138M parameters, and CIFAR-100, with varying forget set sizes. The results are reported in Table 4 and Table 5 Furthermore, for these experiments we train Bad-T for different number of iterations (recall that Bad-T originally proposes epoch of unlearning) to study if its performance would improve with longer iterations. The results suggest that, again, SCRUB is consistently among the top performers across all the metrics. Furthermore, the results show that longer iterations does not improve Bad-T in any of the metrics.
22

---

100
retain-set
forget-set
80
validation-set
CSTOF
retain-set
forget-set
validation-set
epoch
epoch
(a) Performing only max steps (Equation
(b) SCRUB with insufficient max steps.
forget-set
forget-set
validation-set
validation-set
6FTer
4-
2-
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
10
epoch
epoch
(c) SCRUB with many steps.
(d) SCRUB.
Figure 8: Illustration of training dynamics of SCRUB variants, on Lacuna-5 with ResNet. Performing the right interleaving of min-steps and max-steps is important for achieving good balance between high forget error and low retain and validation errors.
retain-set
forget-set
validation-set
retain-set
cTDr
forget-set
validation-set
epoch
epoch
(a) Performing only max steps (Equation 1
(b) SCRUB with insufficient max steps.
forget-set
validation-set set
retain-set
forget-set
validation-set
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
10
epoch
opoch
(c) SCRUB with too many steps.
(d) SCRUB
Figure 9: Illustration of training dynamics of SCRUB variants, on Lacuna-5 with AII-CNN. Per- forming the right interleaving of min-steps and max-steps is important for achieving good balance between high forget error and low retain and validation errors.
23

---

retain set
retain-set
forget-set
forget-set
validation set
validation-set
validation-set
validation-set
eroor
esTor
epoch
epoch
(a) CIFAR-5 with ResNet.
(b) CIFAR-5 with AII-CNN.
retain-set
retain-set
forget-set
error
forget-set
validation-set
validation-set
validation-set
validation-set
20
10
10
epach
epoch
(c) Lacuna-5 with ResNet.
(d) Lacuna-5 with AlI-CNN.
Figure 10: Effect of adding the cross-entropy loss in Equation 3 Dashed lines omit cross-entropy
while solid lines use it. We find that the addition of cross-entropy offers additional protection to maintaining the model's performance during the unlearning procedure. This sometimes comes at the cost of smaller forget set error, compared to the forget set that would have been achieved cross-entropy was omitted from the loss.
Table 5: Results for selective unlearning. 10 examples from class-0 of Cifar-100 dataset. The numbers are averaged over 10 runs.
Test error (L) mean
Retain error mean
Forget error (t) mean
Basic MIA
Forget Size
Model
std
std
mean
std
Original
37.46
4.99
0.54
1.74
0.00
0.00
58.00
9.10
Retrain
37.54
5.24
0.57
1.84
19.00
15.81
45.00
19.04
NegGrad+
45.98
6.79
11.64
12.02
44.00
11.15
55.00
23.88
10
Bad-T @e@ooch
47.23
5.76
13.79
10.28
66.00
18.22
62.00
30.20
Bad-TQ Bad-T@epocle epoch10
47.09
5.65
13.80
9.79
96.00
8.07
77.00
17.25
SCRUB+R
38.40
4.70
0,69
2.07
40.00
13.33
61.00
17.59
SCRUB
38.18
4.78
0.48
1.44
45.00
13.60
56.00
19.76
Original
37.46
4.99
0.54
1.74
0.00
0.00
63.00
10.96
Retrain
37.74
5.31
0.56
1.77
19.50
12.00
49,00
11.47
NegGrad+
45.56
7.00
11.14
12.57
47.00
9.89
67.50
13.94
20
Bad-T Teepchl
46.46
6.28
12.95
11.71
55.50
16.20
55.00
21.59
Bad-Te @pepoc110
45.88
5.70
12.03
9.44
92.00
10.25
70.00
10.87
SCRUB+R
38.39
4.63
0.68
2.00
23.00
8.70
49.50
11.68
SCRUB
38.29
4.61
0.49
1.42
29.00
16.28
54.50
18.54

---

CIFAR-I0
Lacuna-10 10
ResNet
AlI-CNN
ResNet
All-CNN
Model
class
selective
class
selective
class
selective
class
selective
Finetune
3.8
3.09
3.33
3.03
1.7
2.03
2.16
2.00
Fisher
0.08
0.07
0.16
0.14
0.08
0.07
0.16
0.15
+
3.4
2.96
2.30
2.97
1.66
1.5
2.41
2.27
CF-k
3.55
3.17
3.37
2.91
3,42
3,20
3.27
3.11
EU-k
1,41
1.26
1.34
1.20
1.39
1.28
1.32
1.26
Bad-T
19.07
20.44
17.91
17.03
20.05
20.27.
16.32
16.02
SCRUB 7.84
7.41
6.36
5.33
2.17
1.95
2.81
2.48
Table 6: Scale-up factor: the fraction of the runtime of retrain from scratch over the runtime each
given unlearning algorithm. That is, scale-up value of X for an unlearning algorithm means that that algorithm runs X times faster than retrain from scratch.
CIFAR-S
Lacuna-5 Retain error (L)
Test error (L)
Retain error ({)
Forget error (f) mean
Test error (L)
Forget error (1)
Model
mean
std
mean
std
std
mean
std
mean
std
mean
std
Retrain
24.9
2.5
0.0
0.0
28.8
5.9
5.8
0.4
0.0
0.0
4.8
3.4
Original
24.2
0.0
0.0
0.0
5.7
0.0
0.0
0.0
Finetune
24.3
0.0
0.0
0.0
5.6
0.3
0.0
0.0
0.0
0.0
Fisher
31.6
3,4
14,0
6.0
4.8
5.2
14.0
3.6
6.7
3.3
6.4
8.3
NIK
24.4
2.6
0.0
0.0
22.4
9.2
5.6
0.5
0.0
0.0
0.0
0.0
NegGrad+
25.5
1.1
0.0
0.0
41.3
6.1
6.1
0.7
0.0
0.0
1.3
2.3
CF-k
22.6
1.9
0.0
0.0
0.0
0.0
5.8
0.4
0.0
0.0
0.0
0.0
BU-k
23.5
1.1
0.0
0.0
10.7
2.3
5.9
0.6
0.0
0.0
0.0
0.0
Bad-T
27.73
1.89
5.12
1.56
8.00
8.64
5.00
0.33
0.14
0.10
0.00
0.00
SCRUB
24.2
0.0
0.0
40.8
6.2
0.73
0.0
0.0
24.8
Table 7: Small-scale results with ResNet for the Removing Biases (RB) application. SCRUB is the
top-performer terms of forgetting with minimal performance degradation.
14 Additional Results for Removing Biases (RB)
In this section, we provide the results for all scenarios we studied for the Removing Biases (RB)
application for ResNet and All- ANI, on both CIFAR and Lacuna, for both small-scale and large-scale, for completeness, in Tables
CIFAR-5
Lacuna-5 Retain error (L)
Test error ())
Retain error (L)
Forget error (f)
Test error (L)
Forget error (1)
Model
mean
mean
mean
mean
mean
std
mean
Retrain
24.36
1.61
0.13
0.28
28.8
9.12
4.6
0.38
0.0
4.67
6.41
Original
24.08
1.86
0.17
0.38
0.0
0.0
4.53
0.47
0.0
0.0
0.0
0.0
Finetune
23.48
1.91
0.04
0.09
0.0
0.0
9.77
10.76
13.22
19.33
40.03
Fisher
42.64
6.56
31.83
10.47
15.2
16.83
52.53
13.87
51.09
14.54
39.33
NTK
24.16
13.6
8.29
3.33
NegGrad+
26.07
1.21
0.56
0.49
36.00
10.58
5.27
0.76
0.14
0.12
12.00
13.86
CF-k
22.67
1.55
0.00
0.00
0.00
0.00
4.67
0.70
0.00
0.00
0.00
0.00
EU-k
25.87
0.64
3.23
1.69
8.00
6.93
5.20
0.20
0.00
0.00
0.00
0.00
Bad-T 25.87
1.80
9.68
0.45
10.67
4.99
8.87
0.66
2.32
0.79
0.00
0.00
SCRUB
23.88
1.78
0.08
0.12
40.8
8.2
3.87
0.72
0.0
0.0
25.33
4.13
Table 8: Small-scale results with AlI-CNN for the Removing Biases (RB) application. SCRUB is
the top-performer in terms of forgetting with minimal performance degradation.
25

---

CIFAR-I0
Lacuna- 10
Test error (4)
Retain error (
Forget error (1)
Test error (U)
Retain error (4)
Forget error (f)
Model
mean
std
mean
std
mean
std
mean
std
mean
std
mean
std
Retrain
14.72
0.16
0.0
0.0
100.0
0.0
2.87
0.34
0.0
0.0
99.75
0.56
Original
16.56
0.1
0.0
0.0
0.0
0.0
3.07
0.26
0.0
0.0
0.0
0.0
Finetune
16.41
0.09
0.0
0.0
0.0
0.0
3.02
0.37
0.0
0.0
0.0
0.0
Fisher
26.42
1.41
2.45
0.84
100.0
0.0
3.33
0.54
0.0
0.0
100.0
0.0
NegGrad+
17.84
1,46
1.74
2.55
91.26
7.73
3.41
0.17
0.00
0.00
14.90
1.78
CF-k
15.31
0.12
0.00
0.00
0.03
0.01
2.89
0.22
0.00
0.00
0.00
0.00
EU-k
18.73
0.42
0.00
0,00
98.79
0.18
3.19
0.17
0.01
0.02
4.06
Bad-T
19.56
1.44
11.34
1.82
6.12
3.37
0.50
1.06
0.47
67.60
24.26
SCRUB
15.73
0.17
0.51
0.02
100.0
0.0
3.69
0.36
0.28
0.23
100.0
Table 9: Large-scale, class unlearning results with ResNet for the Removing Biases (RB) applica-
tion. SCRUB and EU-k are the cop-performers in this setting in terms of forgetting with minimal performance degradation. Note. however, that EU-k doesn't i perform strongly across the board and in particular performs very poorly in selective unlearning (notice the contrast between EU-k's forget eroor between Figures la and 1b Fisher is also a po-performmr in terms of forget error in this setting too, but on CIFAR causes a large degradation in test error, as often observed for this method.
CIFAR-10
Lacuna-
Test error (1)
Retain error (L)
Forget error (f) mean
Test error (L)
Retain error (L)
Forget error (f)
Model
mean
mean
std
std
mean
mean
std
mean
Retrain
13.97
0.19
0.0
100.0
0.0
1.59
0.36
0.0
0.0
100.0
Original
15.56
0.25
0.0
0.0
0.0
0.0
1.56
0.33
0.0
0.0
0.0
0.0
Finetune
15.39
0.22
0.0
0.0
0.0
0.0
1.67
0.0
0.0
0.0
Fisher
27.4
3.66
1.03
99.0
0.0
1.78
0.0
0.0
89.0
NegGrad+
17.87
0.58
0.13
87.22
1.67
1.63
0.00
0.00
6.56
CF-k
14.99
0.23
0.00
0.00
0.00
0.00
1.48
0.36
0.00
0.00
0.00
0.00
BU-k
15.30
0.69
0.13
0.14
100.00
0.00
1.74
0.45
0.00
0.00
77.19
39.51
Bad-T
16.98
0.40
5.84
0.43
81.93
3.50
2.56
0.09
0.37
0.18
38.65
36.80
SCRUB
15.06
0.14
0.12
0.03
100.0
0.0
2.0
0.4
0.0
0.0
100.0
0.0
Table 10:
Large-scale, class unlearning results with AlI-CNN for the Removing Biases (RB) ap-
plication. SCRUB is the top-performer in terms of forgetting with minimal performance degradation.
CIFAR- 10
Lacuna- 10 Retain error (})
Test error (L)
Retain error ({)
Forget crror (f) mean
Test crror (L)
Forget error (f)
Model
mean
std
mean
std
std
mean
std
mean
std
mean
std
Retrain
17.4
0.14
0.0
0.0
29.67
3.21
2.7
0.2
0.0
0.0
1.0
1.0
Original
17.36
0.14
0.0
0.0
0.0
0.0
2,73
0.15
0.0
0.0
0.0
Finetune
17.37
0.11
0.0
0.0
0.0
0.0
2.63
0.12
0.0
0.0
0.0
0.0
Fisher
21.23
0.27
2.88
0.54
3.0
2.65
3.1
0.35
0.0
0.0
0.0
0.0
NegGrad+
22.7
0.6
4.1
0.5
53.7
6.8
4.7
0.2
0.9
0.1
13.0
1.0
CF-k
17.4
0.1
0.0
0.0
0.0
0.0
2.7
0.2
0.0
0.0
0.0
0.0
EU-k
21.8
0.2
0.4
0.6
23.7
3.5
2.9
0.1
0.0
0.0
0.0
0.0
Bad-T
23.47
1.57
14.53
1.65
34.67
1.70
7.30
2.20
3.26
1.83
0.33
0.47
SCRUB
18.04
0.0
0.0
70.33
4.16
3.0
0.0
0.0
0.0
4.67
3.06
Table 11: Large-scale, selective unlearning results with ResNet for the Removing Biases (RB) application. SCRUB and NegGrad+ are the ter-performers in terms of forgetting, though NegGrad+ has worse test performance than SCRUB in both cases. Note also that NegGrad+ isn't consistent at forgetting across settings as SCRUB.
26

---

CIFAR-10
Lacuna- 10
Test enror (L)
Retain error (L)
Forget error (1)
Test error (L)
Retain error ())
Forget error (f)
Model
mean
std
mean
std
mean
std
mean
std
mean
std
mean
std
Retrain
16.47
0.21
0.0
0.0
25.67
2.31
1.6
0.44
0.0
0.0
0.67
0.58
Original
16.43
0.08
0.0
0.0
0.0
0.0
1,53
0.31
0.0
0.0
0.0
0.0
Finetune
16.5
0.18
0.0
0.0
0.0
0.0
1.43
0.21
0.0
0.0
0.0
0.0
Fisher
21.39
1.22
4.0
1.44
13.0
11.27
1.87
0.21
0.01
0.02
0.0
0.0
NegGrad+
21.36
0.34
3.23
0.37
45.33
2,89
2.77
0.25
0.40
0.07
8.67
0,58
CF-k
16.29
0.07
0.00
0,00
0.00
0.00
1.53
0.31
0.00
0.00
0.00
0.00
EU-k
17.62
0.61
0.11
0.11
0.33
0.58
1.83
0.47
0.00
0.00
0.00
0.00
Bad-T
22.43
0.37
10.13
0.15
1.67
1.25
4,90
2.10
1.34
1.20
0.67
0.94
SCRUB
16.55
0.11
0.0
0.0
29.33
3.21
2.07
0.31
0.0
0.0
1.67
0.58
Table 12: Large-scale, selective unlearning results with AlI-CNN for the Removing Biases (RB)
application. SCRUB and NegGrad+ are the top-performers in terms of forgetting, though NegGrad+ has worse test performance than SCRUB in both cases. Note also that NegGrad+ isn't as consistent forgetting across settings as SCRUB, as can be seen in Figure
model
Test emmr (.)
Retain emror (U)
Forget error (f)
IC test mean
()) sid
IC retain error ({)
Fgt error
Fgt retain
(J)
mcan
std
mecn
std
maan
sid
macn
sid
miaa
mcan
std
Retrain
26.67
2.87
0.0
0.0
90.33
1,53
24.0
1.8
0.0
0.0
18.33
4.16
0.0
0.0
Original
92.0
Finetune
38.13
1.42
0.0
0.0
0.0
0.0
52.0
3.12
0.0
0.0
79.33
10.07
0.0
0.0
NegGrad=
36.27
0.42
0.0
0.0
12.67
21.94
47.5
5.27
0.0
0.0
69.0
13.53
0.0
0.0
CF-
0.0
54.83
2.02
0.0
0.0
85.33
0.0
EU-K
37:47
1.62
7.53
1.26
43.67
2.08
47.0
4.77
8,35
4,73
63.33
9.71
3.67
2.52
Fisher
2.36
21.33
3:45
32.0
51.5
7,47
26.33
9.5
79.0
20.0
7.94
NTK
32.6
2.51
0.0
0.0
60.33
0.58
37,5
4.0
0.0
0.0
52.0
10.58
0.0
0.0
SCRUB
25.93
3.13
L.8
0.52
96.0
1.73
19.0
3.91
0.0
0.0
19.67
7.51
0.0
0.0
Table 13: Results on CIFAR-5 with ResNet for the Resolving Confusion (RC) application. (Con-
fused class 0,1; 50-50 samples). SCRUB is the hest-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first 3 groups of columns).
15 Additional Results for Resolving Confusion (RC)
We show the full results are in Tables 13 across the board, SCRUB is a top-performer on this metric too (see the captions of the individual tables for more details about performance profile).
14
15
16
8
19
20 for all settings. We observe that
16 Additional results for User Privacy (UP)
We present Basic MIA results for all settings in Tables 21 22 23 SCRUB, especially equipped with its rewinding procedure, able consistently have strong defense against MIAs.
24
26
27 28
We find that
Test error (.)
Retain erver (1)
Forget error (f)
IC test error (1)
IC retain error (1)
Fgt test error (1)
Fgt retain error (1)
model
mean
std
mean
std
mean
std
meon
std
mean
std
mean
std
mean
sid
Retrain
24.4
2.75
0.0
0.0
90.67
4,04
19.0
1.32
0.0
0.0
11.33
4.62
0.0
0.0
Original
37.07
4.67
1.5
2.6
5.67
9.81
49.0
4.77
6.0
10.39
80.67
12.58
6.0
10.38
Finetune
34,33
3.35
0.0
0.
3.0
5.2
43.67
7.29
0.0
0.0
67.33
16.04
0.0
0.0
NegGrad-
21.36
42.33
0.0
CT-k
36.13
4.21
0.0
0.0
0.33
0.58
47.83
5.8
0.0
0.0
76.33
14.43
0.0
0.0
EU-k
51.6
1.0
27.67
3.5
52.67
6.03
59,5
5.22
38.33
6.66
68.67
15.57
19.67
10.41
Fisher
35.17
8.69
17.67
NTK
32.2
2.84
0.75
1.3
43.33
14:15
36.67
4.07
3.0
5.2
$4.33
9.02
3.0
5.2
SCRUB
25.0
3.14
0.0
93.33
2.52
26.0
4.44
0.0
0.0
18.0
11.14
0.0
0.0
Table 14: Results on CIFAR-5 with All-CNN for the Resolving Confusion (RC) application. (Con-
fused class 0,1; 50-50 samples). SCRUB is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first 3 groups of columns).
27

---

Test ervee (.)
Retain erver (1)
Forget error (1)
IC test error (1)
IC retain error (1)
Fgt test error (f)
Figt retain error (3)
model
mcan
maan
mican
mcan
std
mican
mean
Retrain
6.0
0.2
0.0
0.0
99.67
0.58
7.17
2.57
0.0
0.0
0.0
0.0
0.0
0.0
Original
27.07
3.33
1.67
0.88
4.33
1.53
57.5
6,26
6.67
3.51
108.0
14.18
6.67
3.51
Finetune
6.03
37.67
0.0
0.0
63.67
0.0
0.0
NegGrad=
17.8
2.95
1.67
0.72
55.33
2,08
33.17
5.25
5.33
1.53
56.67
12.9
4.33
1.53
CF-K
22.27
4.31
0.08
0.14
10.67
5,03
46.33
10.97
0.33
0.58
81.67
23.01
0.33
0.58
EU-K
15.27
3.19
0.83
0.38
62.0
12:49
29.33
9.0
2.33
1.53
43.67
16.29
0.33
0.58
Fisher
35.87
3.33
17.75
3.78
27.33
3.79
60.0
5.27
31,0
7:94
109.0
14.53
30,0
7.0
NTK
51.67
23.18
27.17
0.0
0.0
SCRUB
8.47
1.17
0.33
0.14
96.0
1.0
11.33
3.82
1.33
0.58
9.33
1.53
1.33
0.58
Table 15: Results on 5acuna-5 with ResNet for the Resolving Confusion (RC) application. (Con-
fused class 0,1; 50-50 samples). SCRUB is the hest-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e-g. the usual Error metries in the first 3 groups of columns). NTK in some cases is able to resolve confusion, but not consistently, and it also suffers from higher Test Error.
model
error (.)
errer (U)
Forget error (f)
test errur (L)
IC retuin error (L)
Fgt test error
Fgg retain error (1)
mean
std
mean
std
mean
sid
mean
sid
mcan
sid
maan
std
mcaan
std
Retrain
4.2
0.87
0.0
0.0
100.0
0.0
5.33
2.25
0.0
0.0
0.0
0.0
0.0
0.0
Original
25.47
2.32
5.75
5.63
20.33
25.74
56.17
4.93
23.0
22.54
105.67
8.08
23.0
22.54
Finetune
12.8
2.8
0.0
0.0
23.0
7.94
25.83
7,75
0.0
0.0
39.67
12.74
0.0
0.0
NegGrad-
20.33
17.04
5.0
6.24
12.67
11.68
2.67
3.79
CF-k
21.27
1.63
0.58
0.8
9.33
0.58
47.0
4.58
2.33
3.21
82.67
10.12
2,33
3.21
EU-K
17.0
8.91
3.92
3.99
92.33
4.93
35.0
18.26
13.0
11,36
3.67
4.73
0.0
0.0
Fisher
4.73
39.25
7.45
57.67
10.79
11.68
29.67
NTK
12.87
6.63
2,83
4,91
12.
12.06
25.5
17.88
11.33
19.63
35.67
24:03
10.0
17.32
SCRUB
3.87
0.7
0.0
100.0
0.0
4.33
1,26
0.0
0.0
0.0
0.0
0.0
Table 16: Results on 5 with All-CNN for the Resolving Confusion (RC) application. (Con-
fused class 0,1; 50-50 samples). SCRUB is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first 3 groups of columns).
model
mean
error (.)
std
mean
emrer (1)
std
Forget error (f)
mean
sid
mean
test errur (1)
std
mean
retuin error (1)
std
Fgt error (1)
mean
std
Fgg retain emoor (1)
mean
std
retrain
18.7
0.07
0.0
0.0
98.57
0.28
14.78
0.18
0.0
0.0
31,33
2.08
0.0
0.0
original
21.86
0.37
0.0
0.0
0.0
0.0
31.23
0.45
0.0
0.0
356.0
11.53
0.0
0.0
linetune
20.85
0.37
0.0
0.0
0.0
0.0
26.75
0.48
0.0
0.0
255.0
10.58
0.0
0.0
NegGrad-
23.41
3.87
0.31
6.77
0.6
20.29
8.72
0.67
CF-k
20.93
0.38
0.0
0.0
0.0
0.0
27.27
0.76
0.0
0.0
267.33
16.17
0.0
0.0
BU-k
20.03
0.19
0.25
0.08
95:55
0.54
17.85
0.67
0.18
0.03
53.0
7.94
3.33
2.31
SCRUB
18.01
0.18
0.02
0.01
95.45
0.26
15.07
0.99
0.04
0.03
30.33
3.79
0.33
0.58
Table 17: Results on CIFAR-10 with ResNet for the Resolving Confusion (RC) application. (Con-
fused class 0,1; 2000-2000 samples).
model
Test error (.)
Retain errer (L)
Forget error (f)
IC uest error (1)
retain error (1)
Fgt test error (1)
Fgt retain error (1)
mean
std
mean
std
mean
std
mean
std
mean
std
mean
std
mican
std
retrain
98.42
0.15
14.37
0.24
0.0
0.0
2.31
0.0
original
19.95
0.23
0.0
0.0
0.0
0.0
30.18
0.66
0.0
0.0
348.67
13.58
0.0
0.0
finetune
18.72
0.11
0.0
0.0
1.08
0.61
24.33
0.2
0.0
0.0
223.67
6.66
0.0
0.0
NegGrad-
21.74
0.44
4.48
0.34
87.65
2.98
40.05
0.44
21.8
0.66
44.0
5.2
2.33
3.21
CF-
19.31
0.23
0.0
0.0
0.0
0.0
27.45
0.61
0.0
0.0
294.0
4.36
0.0
0.0
EU-k
0.19
0.79
2.89
0.46
8.62
91.67
SCRUB
15.92
0.17
0.2
0.06
87.47
1.46
14.98
0.13
0.39
0.15
54.0
3.61
9.67
2.52
Table 18:
Results on CIFAR-10 with AlI-CNN for the Resolving Confusion (RC) application.
(Confused class 0,1; 2000-2000 samples).
28

---

Test eroor (.)
Retain error (L)
Forget error (f)
IC lest error (l)
IC retain errot ())
Fgt test error (1)
retain error (
model
mean
sid
meun
sdd
mean
sid
meun
sid
meun
std
meun
std
meun
sid
2.43
0.32
0.0
0.0
99.83
0.29
3.33
0.58
0.0
0.0
0.0
0.0
0.0
original
7.37
0.31
1.21
0.12
15.83
4.19
27.67
1.26
8.26
0.8
46.0
4.36
36.33
3,51
linetune
4.17
0.5
0.0
0.0
56.83
9.44
11.17
1.76
0.0
0.0
15.67
3.51
0.0
0.0
NegGrad=
5.63
0.38
0.31
0.22
71.33
9.88
19.33
1,04
2.12
1.51
8.33
2.31
0.0
0.0
CF-8
5.4
0.4
0.07
0.06
33.83
3.35
17.33
1.76
0.45
0.39
27.67
3,51
2.0
1.73
BU.k
3.0
0.26
0.0
0.0
90.17
4,65
6.0
1.8
0.0
0.0
2.0
2.65
0.0
0.0
SCRUB
3.07
0.59
0.0
0.0
98.5
0.5
6.83
1,26
0.0
0.0
0.67
0.58
0.0
0.0
Table 19: Results on Lacuna- 10 with ResNet for the Resolving Confusion (RC) application. (Con-
fused class 0,1; 200-200 samples).
Test emme (.)
Retain emrrr (t)
Forget error (1)
IC test error (1)
IC retain error (})
Fgr test ertor ())
Fgt retain emorr (
model
mcaa
maan
mcan
mcan
miaan
mcaa
retrain
2.13
0.25
0.0
0.0
99.83
0.29
2,5
1.32
0.0
0.0
0.0
0.0
0.0
0.0
original
7,83
0.55
1,21
0.52
16.0
4,92
31.33
1,76
8.26
3.53
56.33
3.79
36.33
15.53
lincuune
3.0
0.7
0.0
0.0
74.5
6.08
6.5
2.0
0.0
0.0
9.0
3.0
0.0
0.0
NegGrad=
4.3
0.52
0.4
0.06
89.67
4.25
15,33
2.47
2.73
0.39
4.67
3.21
0.0
0.0
CF-k
5:27
0.47
0.11
0.07
33.33
1.61
18:5
2.29
0.76
047
31.33
3.51
3.33
2.08
EU-K
2.53
0.67
0.09
0.02
97.83
2.08
5.17
1.04
0.38
0.35
0.33
0.58
0.67
0.58
SCRUB
2.1
0.4
0.0
0.0
97.5
1,73
4.17
0,58
0.0
0.0
0.33
0.58
0.0
0.0
Table 20:
Results on Lacuna-Il 10 with AlI-CNN for the Resolving Confusion (RC) application.
(Confused class 0,1; 200-200 samples).
Test error mean
Forget error
Retain error
MIA
method
std
mean
std
mean
std
mean
std
Retrain
16.71
0.05
26.67
3.09
0.00
0.00
51.33
6.13
Original
16.71
0.07
0.00
0.00
0.00
0.00
68.67
3.09
Finetune
16.86
0.13
0.00
0.00
0.00
0.00
69.33
2.05
NegGrad+
21.65
0.40
47.00
3.74
4.54
0.70
73.00
1.41
CF-k
16.82
0.03
0.00
0.00
0.00
0.00
69.67
1.89
EU-k
18.44
0.21
0.33
0.47
0.32
0.02
66.00
2.94
Bad-T
22.43
0.37
1.67
1.25
10.13
0.15
77.67
4.11
SCRUB
17.01
0.20
33.00
5.89
0.00
0.00
51.00
1.41
SCRUB+R 16.88
0.19
26.33
4.50
0.00
0.00
49.33
2.49
Table 21: Basic MIA for All-CNN architecture on CIFAR-10 for selective unlearning, for the User
Privacy (UP) application.
Test error
Forget error
Retain error
MIA
method
mean
std
mean
std
mean
std
mean
std
Retrain
13.98
0.07
100.00
0.00
0.00
0.00
48.73
0.24
Original
15.70
0.09
0.00
0.00
0.00
0.00
71.40
0.70
Finetune
14.53
0.13
1.31
0.54
0.00
0.00
74.97
1.27
NegGrad+
17.04
0.11
59.91
1.53
0.43
0.09
70.03
1.92
CF-k
15.72
0.06
0.00
0.00
0.00
0.00
72.93
1.06
BU-k
15.76
0.28
100.00
0.00
0.24
0.02
51.60
1.22
Bad-T
16.98
0.40
81.93
3.50
5.84
0.43
58.07
1.76
SCRUB
14.93
0.17
100.00
0.00
0.09
0.02
54.30
2.24
SCRUB+R 14.93
0.17
100.00
0.00
0.09
0.02
54.30
2.24
Table 22: Basic MIA for All-CNN architecture CIFAR-10 for class unlearning, for the User
Privacy (UP) application.
29

---

method
Test error
Forget
error
Retain
error
MIA
mean
std
mean
std
mean
std
mean
std
Retrain
17.38
0.15
29.33
2.49
0.00
0.00
54.00
1.63
Original
17.41
0.15
0.00
0.00
0.00
0.00
65.33
0.47
Finetune
17.48
0.16
0.00
0.00
0.00
0.00
64.00
0.82
N
21.69
0.07
45.33
2.62
3.94
0.43
66.67
1.70
CF-k
17.53
0.19
0.00
0.00
0.00
0.00
65.00
0.00
EU-k
19.77
0.04
13.67
0.47
0.06
0.01
53.00
3.27
Bad-T
23.47
1.57
34.67
1.70
14.53
1.65
59.67
4.19
SCRUB
17.01
0.03
71.67
0.94
0.01
0.01
78.00
2.45
SCRUB+R 17.54
0.28
19.33
14.64
0.01
0.01
58.67
1.89
Table 23: Basic MIA for ResNet architecture on CIFAR-10 for selective unlearning, for the User
Privacy (UP) application.
Test error
Forget error
Retain error
MIA
method
mean
std
mean
std
mean
std
mean
std
Retrain
14.69
0.10
100.00
0.00
0.00
0.00
49.33
1.67
Original
16.33
0.14
0.00
0.00
0.00
0.00
71.10
0.67
Finetune
15.10
0.16
0.33
0.17
0.00
0.00
75.57
0.69
NegGrad+
17.41
0.09
61.00
1.14
0.44
0.05
69.57
1.19
CF-k
15.29
0.02
0.04
0.04
0.00
0.00
75.73
0.34
EU-k
17.05
0.07
97.48
0.28
0.05
0.01
54.20
2.27
Bad-T
19.56
1.44
11.34
1.82
94.67
6.12
54.33
0.31
SCRUB 15.33
0.06
100.00
0.00
0.08
0.01
52.20
1.71
SCRUB+R 15.33
0.06
100.00
0.00
0.08
0.01
52.20
1.71
Table 24: Basic MIA for ResNet architecture on CIFAR-10 for class unlearning, for the User Privacy
(UP) application.
Test error
Forget error
Retain error
MIA
method
mean
std
mean
std
mean
std
mean
std
Retrain
1.50
0.08
0.33
0.47
0.00
0.00
52.00
2.16
Original
1.57
0.24
0.00
0.00
0.00
0.00
59.00
2.16
Finetune
1.40
0.16
0.00
0.00
0.00
0.00
57.33
3.30
NegGrad+
3.60
0.14
14.33
1.25
0.87
0.07
51.00
1.63
CF-k
1.57
0.12
0.00
0.00
0.00
0.00
58.33
2.49
EU-k
3.90
1.47
0.00
0.00
0.76
0.63
52.00
3.56
Bad-T
4.90
2.10
1.34
1.20
0.67
0.94
67.67
6.94
SCRUB
1.67
0.19
0.00
0.00
0.00
0.00
57.67
0.94
SCRUB+R 1.67
0.19
0.00
0.00
0.00
0.00
57.67
0.94
Table 25: Basic MIA for AlI-CNN architecture Lacuna- 10 for selective unlearning, for the User
Privacy (UP) application.
30

---

Test error
Forget error
Retain error
MIA
method
mean
std
mean
std
mean
std
mean
std
Retrain
1.67
0.09
100.00
0.00
0.00
0.00
55.67
2.62
Original
1.70
0.21
0.00
0.00
0.00
0.00
58.00
1.63
Finetune
1.67
0.27
0.00
0.00
0.00
0.00
56.33
1.25
NegGrad+
2.00
0.00
14.27
0.74
0.00
0.00
54.33
2.05
CF-k
2.07
0.14
0.00
0.00
0.00
0.00
52.33
2.05
EU-k
4.15
1.22
62.08
44.26
0.81
0.53
52.67
3.68
Bad-T
2.56
0.09
38.65
36.80
0.37
0.18
63.33
2.49
SCRUB
1.96
0.34
100.00
0.00
0.00
0.00
50.33
2.62
SCRUB+R
1.96
0.34
100.00
0.00
0.00
0.00
50.33
2.62
Table 26: Basic MIA for AII-CNN architecture on Lacuna-10 10 for class unlearning, for the User
Privacy (UP) application.
Test error
Forget error
Retain error
MIA
method
mean
std
mean
std
mean
std
mean
std
Retrain
2.50
0.24
1.67
0.94
0.00
0.00
49.67
3.09
Original
2.53
0.25
0.00
0.00
0.00
0.00
56.67
1.70
Finetune
2.67
0.05
0.00
0.00
0.00
0.00
53.67
0.94
NegGrad+
4.30
0.43
12.67
3,30
0.95
0.08
54.00
2.16
CF-k
2.47
0.25
0.00
0.00
0,00
0.00
56.00
0.82
EU-k
2.60
0.00
0.00
0.00
0.03
0.00
56.00
2.83
Bad-T
7.30
2.20
3.26
1.83
0.33
0.47
67.33
3.40
SCRUB
2.97
0.25
6.00
3.27
0.00
0.00
50.67
4.03
SCRUB+R 2.97
0.25
6.00
3.27
0.00
0.00
50.67
4.03
Table 27: Basic MIA for ResNet architecture on Lacuna-10 for selective unlearning, for the User
Privacy (UP) application.
Test error
Forget error
Retain error
MIA
method
mean
std
mean
std
mean
std
mean
std
Retrain
2.52
0.19
100.00
0.00
0.00
0.00
55.00
2.94
Original
2.81
0.28
0.00
0.00
0.00
0.00
56.00
2.45
Finetune
3.04
0.19
0.00
0.00
0.00
0.00
54.67
1.25
NegGrad+
2.74
0.26
9.48
0.64
0.00
0.00
53.67
4.03
CF-k
2.81
0.28
0.00
0.00
0.00
0.00
56.00
2.45
EU-k
2.48
0.14
7.71
2.52
0.00
0.00
54.33
3.09
Bad-T
3.37
0.50
67.60
24.26
1.06
0.47
58.00
2.94
SCRUB
3.26
0.38
99,90
0.15
0.07
0.05
54.33
2.49
SCRUB+R 3.26
0.38
99.90
0.15
0.07
0.05
54.33
2.49
Table 28: Basic MIA for ResNet architecture on Lacuna- 10 for class unlearning, for the User Privacy
(UP) application.
31


Reviews:
[{"review_id": "3vnCs90zxI", "reviewer": "Reviewer_QAcd", "summary": "This paper introduces the methodology SCRUB, a teacher-student unlearning algorithm which personalises its forget metrics per application and dataset using. This method is demonstrated under three use cases: removing bias, resolving confusion, and user privacy. The authors support their theoretical claims with empirical experiments for each use case.", "questions": "- RB, RC, and UP should be defined again in the introduction\n- In Section 4, it would be nice to elaborate on why Bad-T performs poorly compared to SCRUB?\n", "limitations": "Yes", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The motivation of this paper is well-founded, and a major contribution to the field of unlearning. The paper is well-written and clear, with the contribution evident. The notation used throughout the paper is easy to follow. The empirical experiments have been well-chosen and SCRUB outperforms other baselines consistently.", "weaknesses": "Some of the experiments seem a little simple (specifically the standard attack for the user privacy setting), but given the larger scope of the paper, this is understandable. If additional experiments could be added in this setting, it would greatly strengthen this specific use case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer QAcd for the positive feedback. We are glad to hear that the reviewer found our work to be a major contribution to the field of unlearning. We address QAcd’s comments and questions below.\n\n**RE: RB, RC, and UP should be defined again in the introduction**\n\nGreat suggestion, we have updated the intro to include that\n\n**RE: In Section 4, it would be nice to elaborate on why Bad-T performs poorly compared to SCRUB?**\n\nTo answer this, let’s first consider the differences between SCRUB and Bad-T (please see also our general response for all reviewers for additional details on their differences.\n SCRUB and Bad-T share the (only) similarity that they can both be formulated using the teacher-student framework (like many other papers within various fields within machine learning). Within the said framework, however, they have fundamentally different objectives:\n1) SCRUB considers the original model as the teacher and encourages the student to stay close to the teacher on retain examples and move away from it on forget set examples. \nOn the other hand\n2) Bad-T considers two sets of teachers: the original model is the “good teacher”, and they consider a set of *random weights* as a “bad teacher”. They encourage the student to follow the good teacher for retain examples and the bad teacher for forget examples.\n\nSo, the two are substantially different, making it hard to draw any conclusions with confidence about the causes of Bad-T’s poor performance. It is not clear why *moving close to random weights* constitutes a good forgetting algorithm. So, we weren’t surprised by the fact that it’s not a strong-performing approach. An adverse effect of moving close to random weights, for instance, may be that the forget set accuracy becomes much lower than it ideally would have been (note that, ideally, the unlearned model has a forget set accuracy similar to that of re-training from scratch without the forget set; not similar to that of a random model). This may be the cause of Bad-T’s poorer performance on MIAs. In addition, we find empirically that Bad-T (with its formulated training objectivEs) fails badly to properly reconcile the trade-offs between good performance on forget set examples vs examples from the retain set. And its generalization ability (performance on test set) is often very poor.\n\nPlease note as well that the Bad-T paper does not compare against state-of-the-art methods for unlearning (they only consider one baseline in their paper, which is an older/outdated approach), so an understanding of its empirical performance remained elusive prior to the results we report here.\n\nWe hope the above responses address all key concerns and we are very keen to engage in order to further clarify/address comments and questions! \n"}, {"review_id": "CmbmoRnDn0", "reviewer": "Reviewer_Xo5D", "summary": "The paper presents a Machine Unlearning method and shows its utility in Removing bias, Resolving confusion along with protecting user privacy. The experiments are conducted with different sets of evaluation metrics for each application. The latest Membership Inference Attack is adopted for Unlearning to evaluate the information leak in the unlearned model. The paper also stresses on relaxing the requirements of theoretical proofs for the bounds on the Machine Unlearning algorithms in general. This argued is due to the Unlearning methods utility in other applications than Data Privacy in ML.", "questions": "1. The authors could let me know their thoughts on Weakness-1 and Weakness-2. \n\n2. Since, several existing Machine Unlearning methods have been presented in the ‘unbound’ setting (refer to weakness section). Maybe, it would be nice to discuss the idea of ‘unbound’ as a general category of methods and not something novel presented in this paper?\n\n3. In Bad-T, there is a metric called Zero Retrain Forgetting (ZRF) which also aims to measure the generalization power of the unlearned model and not focus just on the Df and Dr accuracies. Is there any way the ZRF metric could be incorporated in this method?\n\n4. I have few doubts about Removing Biases experiments – Are you unlearning a class or a subset of class or a set of random samples here? Also, it is unclear to me how the experiments on Removing biases are different from standard machine unlearning experiments. In fact, how exactly it is removing bias in the model? Could you elaborate?\n\n5. Doubts about Resolving Confusion -  I would like to know more about how exactly the confusion is resolved through unlearning?", "limitations": "1. The experiments are performed only on CIFAR-10 which is pretty small-scale given the authors mention one of their motivation to design this method is to address the shortcomings of the existing Unlearning methods that work only on small datasets.\n\n2. Several existing methods have already shown results on CIFAR-100 [1,2,3]. The proposed method could outperform some of these methods in 10-class dataset. It would be interesting to see how well the methods perform on 100-class dataset.\n\n3. Sequential Unlearning? – A common use case of Machine Unlearning would require repeated unlearning over the period of time. The experiments on repeated unlearning  is missing in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The following are some of the strength of the paper:\n1. The paper shows the utility of Machine Unlearning in removing Bias and Resolving confusion. \n2. The paper is well written and easily understandable.\n3. Adoption of LiRA MIA attack is a good addition and to the evaluation of Unlearned Model.\n4. Several experiments are conducted and comparison with some of the existing methods is discussed.", "weaknesses": "1. Excessive Similarity with an existing paper – The main idea of this paper shares a lot of similarity with the following paper https://ojs.aaai.org/index.php/AAAI/article/view/25879/25651 (mentioned as Bad-T in the paper)\n\nIn Page-6, Line 239-241, it is mentioned that “further differences with Bad-T is discussed in Appendix”. However, I could not find any such comparison. Since, there is a lot of similarity between the proposed method and Bad-T, it would have been appropriate to give a technical difference between the proposed and Bad-T method. But only the experimental results are compared in the Appendix.\n\nThere are excessive similarities in the method described in Section 3.1 and Bad-T paper. The list of the similarities are:\n\na. Page 3, Line 129-136. This is same as the Bad-T paper which also treats the original model as a teacher and the student selectively obeys the teacher for retain (Dr) and forget (Df) samples. Refer to the “Section: Proposed Method” in Bad-T Paper. The training object in Bad-T is also based on the KL-Divergence between the output distributions of the teacher and student model. \n\nb. Similar to Bad-T method, the teacher model is frozen here. Similar to Bad-T paper, the student is initialized with the weights of the teacher. Only difference is that the optimization is done alternatively for the retain (Dr)  and forget (Df) samples. This is again quite similar to Bad-T optimization where the optimization for retain (Dr) samples are done using the competent teacher (original model) and optimization for the forget (Df) samples are done using the bad teacher which acts as a good proxy for the student model to move away from the original model. Particularly, if we take the variation of Bad-T where the bad teacher is just a random generator, wouldn’t that give a better robustness against MIA attacks? Because it is training the model predict random outputs instead of deliberately attempting to move the weights towards the opposite direction for forget samples (Df).\n\nc. The proposed method might be more prone to being unstable as already discussed in the limitation section. A comparison with Bad-T in terms of Stability was desired.\n\nd. Overall, it seems the entire method is almost similar to the Bad-T paper. Besides, the authors did not clearly mention the technical difference with Bad-T. Furthermore, as mentioned in the paper, Bad-T is significantly faster than the proposed method. In Machine Unlearning, speed is one of the most important factor. It is quite possible that the Bad-T method (or its other variants) might give similar results on all the experiments presented in the paper with better speed.\n\n2. Inadequate Literature Study:\n\nIn Page 2, Line 54-58, the authors explains that there is a lack of Machine Unlearning methods that are free from the assumption of the stability of SGD and can work with bigger datasets. In the literature study Section (Page 5) also, the authors primarily highlight the 3 papers by Golatkar et al. and derive their motivation for this paper. \n\nIn past few years, there have been several papers that have highlighted the impracticality, inefficiency, and non-scalability issues present in the methods by Golatkar et al. Some of the papers that have shown Unlearning results without relying on the stability of SGD and other assumptions in Golatkar et al.: \n[1] “Amnesiac Machine Learning”\n[2] “Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks Using an Incompetent Teacher”\n[3] “Fast Yet Effective Machine Unlearning”\nSome more papers can be found here: https://github.com/jjbrophy47/machine_unlearning\n\nThe idea of ‘unbound’ is not new as claimed in this paper. All these 3 papers have in fact shown good unlearning in ‘unbound’ setup. As far scalability is concerned, [3] have shown results on CIFAR-100 and VGGFACE-100 (class-level unlearning). The paper fail to mention these salient points in the literature study and in the experiments. \n\n3. Considering the above mentioned points, the in the title of Section 2, the word ‘Unbound’ appears trivial. This Section gives a generic definition and discusses existing trade-offs in Machine Unlearning (MU). The content in this section doesn’t add anything significant to define ‘Unbound’. In fact, the Section title is little confusing. For example, in case of Privacy, we would like the MU method to be bound? Or in general unbounded methods are fine as they can be used to address multiple type of problems?\n\n4. About SCRUB+R: Maybe this could have been part of the experiments and analysis section? It is just Early Stopping….", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Xo5D for their comments and effort.\n\n**RE: the similarity with Bad-T** \nThank you for bringing up this important topic. Please see our answer in the general response, above. \n\n**RE: “Wouldn’t that give better robustness against MIAs?”**\nNo, that goes against key findings (cf. Carlini et al. for insights on MIAs). To summarize: a model is robust to MIAs if the attacker can’t tell apart training examples from held-out examples (e.g. by inspecting loss values/ confidences). For unlearning the goal is that the forget examples “look\" like unseen examples. Having the forget set losses/confidences be similar to those that would come from a random model is thus *not* a good solution, unless the unseen (test) examples’ losses/confidences also looked like they come from a random model (which Bad-T does not encourage and generally is certainly not true). So this approach may lead to increased MIA vulnerability. \n\nOn the other hand, SCRUB’s objective for forget examples is to move away from the original model in a controlled manner; via our carefully-crafted min-max steps and rewinding procedure. So SCRUB intuitively may be more successful than Bad-T at driving the unlearned model closer to a region of the desired “retrain-from-scratch” oracle (which again, is very unlikely to be close to random weights). We will add text to explain this better.\n\n**RE: \"The proposed method ... prone to being unstable\"**\nNB: we developed a practical recipe for optimization that works very well in practice. SCRUB’s very strong performance across a very wide range of scenarios that we consider is a testament to the success of that recipe. On the other hand, we find that Bad-T performs poorly.\n\n**RE: “Bad-T is significantly faster”**\nBad-T is faster. But it is running for only 1 iteration (as recommended in the Bad-T paper). And when running it for longer it did not further improve! So, it’s worth considering that Bad-T is fast because it doesn’t keep improving with more iterations. SCRUB strikes a great balance, being faster than most, being consistently a top performer across all metrics, applications, datasets, and architectures,  *unlike Bad-T*. Please see also the results from new experiments addressing limitation 2 substantiating the above.\n\n**RE: “Inadequate Literature Study”**\nReviewer Xo5D is correct: some previous works have moved towards ‘unboundedness’. But SCRUB is the only one shown to be ‘unbound’ in the application domain. We specifically identify that different unlearning applications require different performance metrics for forget quality. The SCRUB/SCRUB+R framework is the only one that is consistently a top performer for all metrics of forget quality for different applications while not sacrificing retain- and test-set accuracy. This is the essential feature of ‘unboundedness’ contributed by this paper and no previous work has achieved this! We thank the reviewer for raising this and plan to make the necessary edits in the Introduction sections to explain this better.\n\n**RE: the rewinding procedure and early stopping**\nThere is probably a misunderstanding: SCRUB+R is *fundamentally different to  “just early stopping”*. Early stopping refers to stopping early based on a criterion computed during training (e.g. validation accuracy). SCRUB+R, does not do this! As explained in Section 3.2, it first *must train “all the way”*, until  maximal error on the forget set is obtained. Then, based on that, it determines which previous checkpoint to use (“rewinding”). In addition, SCRUB+R uses an especially-selected subset of the validation set. Overall, our rewinding is not standard. It is a dedicated novel procedure, designed specifically to obtain an appropriate reference point for the forget error and avoid vulnerability to MIAs. Both the mechanism and underlying design principles are far from “just early stopping”.\n\n**RE: the ZRF metric**\nPlease note that we don’t “focus just on the Df and Dr accuracies”. We actually do additionally measure generalization. And we are doing so using the most commonly-accepted way of doing so: the accuracy on a test set. In fact, we’re surprised and concerned that test accuracy isn’t reported in the Bad-T paper. The ZRF metric sounds interesting but is definitely a non-standard way to measure generalizability - albeit an interesting idea for future research.\n\n**RE: using CIFAR-100 dataset**\nIndeed we ended up using smaller datasets. This was driven by the fact that several of the competing unlearning algorithms had very poor performance for larger datasets. However, Reviewer Xo5D brings up an interesting issue here.  To address this, we performed experiments in CIFAR-100, using a larger model, namely VGG+BN, with ~138M parameters. Our results in Table 1 to Table 3 (see general response/rebuttal) show that Bad-T fails (one more time) to reconcile model utility with good forget quality. And, (partially) addressing also weakness 1.d. Specifically, Bad-T performs really poorly for retain-set errors and test errors (many times larger than the error of retrain-from-scratch and SCRUB).\n\n**RE: RB and RC experiments**\n\nRB: We consider both class (forgetting a whole class) and selective (forgetting a subset of a class’ examples) unlearning. We assume users/owners of the model have identified the examples producing the biases in an operational model, and place them in the forget set. If unlearning is successful, the biases will have been removed as the effect of those examples are removed from the model. \n\nRC: a subset of the training set is mislabelled and this is discovered after the original model is trained. We place all and only the mislabelled examples in the forget set and run unlearning. If successful, the resulting model will no longer make mistakes due to its past confusion. \n\nIn each application, we use the proper metrics to measure forget quality as appropriate.\n\nReferences:\n- [Carlini et al] Membership inference attacks from first principles. IEEE S&P 2022"}, {"review_id": "dY3eyHgiRq", "reviewer": "Reviewer_LTMH", "summary": "The paper proposes three distinct goals to machine learning: removing bias, resolving class confusion from mislabelling of training data, and privacy through the right to be forgotten, and suggests that these goals require different technical approaches. The paper proposes an linearized multi-objective optimization problem (minimize loss on retained examples, minimize average KL divergence between softmax outputs to the original ones on retained examples, and maximize it on forget examples) with an alternating step optimizer. To limit the membership inference risks associated with ensuing high loss on forget examples, the paper proposes early stopping with a specially crafted validation set. The paper evaluates the algorithm for three goals on two datasets and using two network architectures.", "questions": "---", "limitations": "See W2.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- S1. The proposed optimization objective is simple and seems to be more effective than a multitude of more complex approaches from prior work.\n- S2. Relatedly, the performance of the algorithm seems to be significantly more effective with multiple goals in mind.\n- S3. The paper delinates three different goals for unlearning that could need different technical approaches.", "weaknesses": "- W1. *Impossible to evaluate the technical contributions without a detailed reading of the appendix.* The main body of the paper does not contain sufficient technical details to understand the contributions, deferring many crucial details to the appendix: the alternating step algorithm which is crucial to the contribution, training dynamics of the proposed objective that is inherent to the proposed optimization objective, the exact construction of the validation set for \"rewinding\", etc. Although the paper does not read bad, this is the reason I give the lowest score to the presentation rubric. It is not possible to judge the technical contributions without a reading of the appendix, which goes against the point of the page limit.\n- W2. *Insufficient evaluation in some aspects.* Although in many regards, the evaluation is comprehensive, e.g., with respect to the coverage of the methods from prior work, it is rather incomplete in other aspects. In particular:\n   - We only see the results on two datasets and two models.\n   - We do not necessarily know how significant is the performance improvement under optimization uncertainty. Such uncertainty might have non-trivial effect due to the unstable nature of the training dynamics of the proposed optimization objective.\n   - The proposed optimization objective is simple and is only slightly different from the prior NegGrad work. I do not think this constitutes lack of novelty, especially as it seems that the proposed simple approach outperforms many complex ones. However, in this case, it is important to see ablation studies with respect to the design choices of this objective. For instance, why KL divergence in softmax, not loss for both retain and forget sets, or for one of the two sets? Are there any possible stability inducing terms that would get rid of periodic training dynamics?\n   - Although I understand this is computationally expensive, especially if comparing to the retraining baseline, it would be good to see the performance in at least one large-scale dataset setting. It is important as machine unlearning seems most useful in such settings; otherwise, retraining might be good enough. In choosing the evaluation settings the paper cites prior work, but some of the prior work has evaluated, e.g., on Imagenet-1K.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer LTMH for raising the important points. \n\n\n**RE: Weaknesses, W1**\n\nThank you for this fair comment. However, we wish to point out that the “scale” of this contribution is very large, vis-a-vis the page limit of NeurIPS submissions. This is, by the way, a common problem for many NeurIPS papers. We tried hard to convey all essential elements of our methods and problem setting and comprehensive experiments and results. Nonetheless,  we will revise the paper write-up, incorporating many of the above key elements in the paper properly.\n \n**RE: \"The proposed optimization objective is simple and is only slightly different from the prior NegGrad work\"**\n\nThank you very much for raising this point, as it is critical. We would like to clarify that *the NegGrad baseline we use in this work is a novel unlearning algorithm in its own right* – also our contribution. To make this clearer, we will rename it to NegGrad+ (or something similar). A similar NegGrad baseline has been used in previous works that simply performs gradient ascent on the forget set. We found this to be very poor, so we won’t report it. That observation, though, inspired us to build an improved version of NegGrad that shares structural similarity with SCRUB (due to balancing two terms; one on the retain and one of the forget set). Our motivation for constructing this additional baseline is twofold: 1) it allows us to measure the added gain from SCRUB’s additional complexity *over our improved NegGrad*, and 2) contribute another baseline to the field, which we show is a particularly strong one, and we hope future work reports results with it too. From a pedagogical perspective, we are hoping that reviewers and readers will notice the similarity and then be able to identify/appreciate the added benefits coming with SCRUB/SCRUB+R.\n\n**RE: More/Larger datasets and models**\n\nThank you for this meaningful comment as well, as indeed in small-scales perhaps retrain-from-scratch is viable. In fact, we have already performed experiments on a larger dataset/network (namely, a 100-way classification task using CIFAR-100 and the VGG+BN network, with ~138M parameters). The results shown in Table 1 to Table 3 (please see the tables in our general response, above) clearly show that SCRUB has retain, test, and forget errors very close to retrain-from-scratch! \n\n**RE: Optimization's uncertainty**\n\nThis is a great point. Indeed the min-max optimization of SCRUB may lead to increased variance due to the oscillating behaviour. We have acknowledged this fact as a limitation of SCRUB in the appendix. However, there are two key points to consider: Firstly, we've demonstrated through practical experiments that we can mitigate the impact of this uncertainty by following the training procedures outlined in Algorithm 1 and Algorithm 2. You can find related ablation results in Figure 6 to Figure 9. Secondly, in all the experiments detailed in the paper, we've taken a thorough approach to comparison. We have run each method multiple times using different random seeds and compared their results using 95% confidence intervals. These multiple runs inherently capture some of the uncertainty arising from the optimization process itself. Importantly, the results reveal that the width of SCRUB's 95% confidence interval is not broader when compared to other baseline methods. This suggests that in practice SCRUB in fact does not introduce additional uncertainty beyond what is already present in the underlying setting. And, we find that SCRUB is by far the most consistent top performer across applications, metrics, and architectures out of all considered baselines.\nWe leave it to future work to propose more stable optimization algorithms for min-max optimization problems like SCRUB’s but in the meantime, we hope that the reviewer appreciates that we have performed a very thorough investigation that reveals that SCRUB is a top-performing unlearning method across the board. We believe that this is a valuable contribution to the community.\n\n**RE: Ablation studies**\n\nIn fact, we have conducted sensitivity analysis against alpha and gamma parameters that show the sensitivity of scrub’s performance w.r.t those parameters, Please see Fig 5 in the appendix (supplementary document), which shows that SCRUB is not very sensitive to these hyperparameters. Additionally, we have ablated the cross-entropy loss in our formulation, please see Fig 10 in the appendix.\nFurthermore, it is worth noting that certain modifications to the terms in our loss formulation will result in some of the other baselines. For instance, using cross-entropy only for the retain-set would yield the ‘Fine-tune’ baseline. Using cross-entropy only for the forget-set yields the simple NegGrad in the existing literature, that tends to catastrophically forget everything, and replacing both KL-divergence terms by cross-entropy recovers our improved proposed NegGrad. Therefore, we can view those baselines as constituting different ablations / variations of SCRUB's objective function.\n\nWe hope the above responses address all key concerns and we are very keen to engage in order to further clarify/address comments and questions!\n"}, {"review_id": "sqIyP6Ibt2", "reviewer": "Reviewer_YAUt", "summary": "This paper studies machine unlearning from an empirical perspective, proposes a scalable unlearning method SCRUB based on knowledge distillation, and evaluates the performance under different metrics along with other available unlearning approaches. Overall, this paper may provide insights and empirical metrics for designing other scalable unlearning methods in practice.", "questions": "Here I summarize some questions from the weakness section.\n\n1. Why the adaptation of LiRA is needed? Is it because allowing the attacker to know the unlearning algorithm can greatly increase the attack accuracy?\n\n2. What would change in the unlearning algorithm when we only want to unlearn a very small portion of the training set?\n\n3. What is the real difference between Removing Biases (RB) and Resolving Confusion (RC)? From the description, it seems that their goals are both not to predict the original labels of the forget set, and they share the same algorithm. I believe they can be both viewed as data cleaning for model training.", "limitations": "The authors have listed detailed limitations in the Appendix. First and the most important limitation would be it is hard to analyze SCRUB theoretically, as it always is in deep learning. However, when it comes to user privacy, sometimes a theoretical guarantee is required, which prevents the usage of SCRUB. The authors also mention the difficulty in training the min-max objective function, and how it will affect the model performance. Finally, the authors show cases where the rewinding procedure fails to prevent the MIA model to recognize the forget set, leading to privacy leakage in such cases.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Firstly, unlike other unlearning papers focusing primarily on user privacy, this paper summarizes three possible applications (removing biases, resolving confusion, and protecting user privacy) for unlearning algorithms to work in practice. Indeed, one should consider different metrics to evaluate the performance of unlearning algorithms under different scenarios. This is a valuable observation for designing other unlearning methods. The authors also show that previous state-of-the-art methods fail to obtain good performance on all tasks, which is exactly due to the fact that previous methods only focus on one criterion and thus cannot have universal good performance.\n\nSecondly, the proposed method SCRUB is based on a teacher-student model formulation, which has high scalability that can support large forget sets. Many previous unlearning methods, especially those having theoretical guarantees, would need some kind of sequentially unlearning process and repeatedly computing the Hessian matrix or Fisher information which leads to unavoidable high computational complexity. On the other hand, SCRUB treats the forget set as a whole and aims to have a consistent performance on the forget set.\n\nLastly, this paper provides a comprehensive empirical evaluation to validate the superior performance of SCRUB. Moreover, for user privacy applications, it uses the latest LiRA membership inference attack accuracy as a metric and adapts the SCRUB method with a rewinding step, which has indeed proven to be helpful by simulations. The full code and documents are included in the SI for reproducibility.\n\nOverall this paper is well-written and easy to follow.", "weaknesses": "Firstly, as the authors have pointed out in the paper, the idea of using knowledge distillation to achieve unlearning is not new, and the metrics, including using MIA accuracy to evaluate the unlearning performance are also not new. So my biggest concern for this paper is that the contribution might seem incremental. Also for the adaptation of the LiRA attacker, although the authors claim that \"the first adaptation of LiRA for unlearning\", but it is not clear why this type of adaptation is really needed. Is it because this adaptation can greatly increase the attack accuracy? Or is it technically challenging to apply LiRA directly within the framework? I would assume it is the first one, but some clarifications on these would make the paper clearer.\n\nSecondly, I am not sure if SCRUB can work well when the forget set is small but the training set is large. Consider the case where a model uses millions of data points for training, but we only want to unlearn one (or a very small portion) data point. This can happen frequently, especially in the UP case, where only a few users want to withdraw their data at the same time. Based on SCRUB+R, we would need to create a validation set of the same distribution as the forget set, which is basically sample another point from the same class as the validation set, and the goal is to make the model performance on forget point and validation point as close as possible. Since the choice of validation point could be any point from that class, such kind of requirement might lead to unstable training and the final model performance would largely depend on which point we choose. This is just an extreme example, but such kind of trend also appears in Figure 3, where the difference between SCRUB+R and Retrain becomes more significant when the forget set is smaller.\n\nHowever, despite these weaknesses exist, I think the observations, applications, and evaluation metrics proposed in the paper is still meaning for the unlearning community. So I would recommend boardline accept for the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer YAUt for their insightful comments. \n\n**RE: incremental contribution**\nPlease see our response in the general rebuttal for similarity against other methods.\n\n**RE: Smaller forget-sets and validation distribution**\n\nThank you for this comment. We should have clarified better and we will in the revised paper. Please note that we’re not restricted to picking a few validation examples if the forget set is small. We’re free to pick as many validation examples as we like (assuming availability, of course), and, as the reviewer points out, we will pick sufficiently many to avoid high-variance issues. The requirement that we have is to have a similar distribution between the validation and forget sets. In the context of classification problems, we simply do this via inspecting the class labels. So, for example, if the forget set contains only cats, we will pick only cats for the validation set too.\nIn addition, we have now performed additional experiments where the forget set size is much smaller than those reported in the paper. We used a larger dataset/network (namely, a 100-way classification task using CIFAR-100 and the VGG+BN network, with ~138M parameters). The results shown in Table 1 to Table 3 (please see the tables in our general response, above) clearly show that SCRUB has retain, test, and forget errors very close to retrain-from-scratch, even when the forget set sizes are just 10/40,000, 20/40,000, and 400/40,000 (that is, 0.1%, 0.2%, and 1% of the dataset size) - in addition to the results for 100/40,000 in the paper, showing robustness across forget set sizes)!\n\n**RE: \"Why the adaptation of LiRA is needed?\"**\n\nGreat question! Let us explain. There has been an extensive literature of MIAs (LiRA is the state-of-the-art) *outside of the context of unlearning*. The goal of the attacker there is to infer whether a given example was part of the training set of the model that is under attack. Now, the evaluation protocol in the context of unlearning is different: we don’t care to attack the original model in this case, and we don’t care to attack any example. Instead, we aim to use MIA tools to gauge unlearning quality: we want to see if the unlearned model (the original model, after having undergone an unlearning procedure) is vulnerable to attacks w.r.t the forget examples. That is, to estimate the quality of unlearning, what we care about is to measure whether the forget-set examples are indistinguishable from held-out examples, in the context of the unlearned model.\nTherefore, to build the strongest possible attack in this setting, we take exactly this setup into consideration when constructing shadow models for the attacker. One could of course try applying MIAs out-of-the-box but we suspect (and found in preliminary results in earlier stages of this research project) that those attacks will be generally less successful.\nOverall, the study of MIAs as a tool for evaluating unlearning quality is an open research area (as our privacy-expert colleagues assure us). In our paper, we actually include two different types of MIAs, (one LiRA-like and one not) to show the resilience of SCRUB/SCRUB+R under all these metrics. We sincerely hope that this will be appreciated in terms of (i) the comprehensiveness of the evaluation and (ii) in terms of the consistently strong-performing methods we contribute! As an aside, we strongly believe that our proposed adaptation of LiRA is a fundamental step in that direction (and a significant contribution of its own accord).\n\n**RE: \"What would change in the unlearning algorithm when we only want to unlearn a very small portion of the training set\"**\n\nGreat question! We believe that an extensive experimentation of different unlearning algorithms with different types of forget sets, and different forget set sizes would be a very valuable contribution. However, such an investigation is outside the scope of our paper. We hope the reviewer agrees that our experimentation is already very thorough and well-substantiates our claim that SCRUB is a consistently top-performing algorithm. \n\nNonetheless, to address this fair comment by the reviewer, we have performed additional experiments. We used a larger dataset/network (namely, a 100-way classification task using CIFAR-100 and the VGG+BN network, with ~138M parameters). The results shown in Table 1 to Table 3 (please see the tables in our general response, above) clearly show that SCRUB has retain, test, and forget errors very close to retrain-from-scratch, even when the forget set sizes are just 10/40,000, 20/40,000, and 400/40,000 (that is, 0.1%, 0.2%, and 1% of the dataset size) - in addition to the results for 100/40,000 in the paper, showing robustness across forget set sizes)!\n\n**RE: \"What is the real difference between Removing Biases (RB) and Resolving Confusion (RC)?\"**\n\nThis is another great point. Yes, to a large extent, the reviewer is correct. However, for the RC case, one can (and we argue that one should) employ application-specific metrics (additionally to retain-, forget-, and test-errors) that address the ability of the model to specifically resolve class mislabeling (confusion). And, for the RC cases, the forget set includes mislabelled examples, whereas for the RB cases, the forget set consists of examples causing biases.\n\n\nWe hope the above responses address all key concerns and we are very keen to engage in order to further clarify/address comments and questions! \n"}]

Author rebuttal:
Dear reviewers, 
Thank you very much for your dedication and insightful comments. We have thoroughly gone through your comments, taken your suggestions into deep consideration, and provided comprehensive responses. 

We answer below a comment that we believe is of interest to all reviewers. It concerns the similarity between SCRUB/SBRUB+R and the Bad-T baseline.  Also, we present 3 tables displaying results from new experiments we have run to address comments by more than one reviewer. These tables serve as supporting evidence for several of the arguments we make in our rebuttal.  The experiments here are conducted on a larger model, i.e VGG16+BN with ~138M parameters, and a larger dataset (in terms of the number of classes), i.e CIFAR-100 and with a variety of forget set sizes.

We address other details in each reviewer's rebuttal section.

**RE: the similarity with Bad-T** 
1. SCRUB and Bad-T are only similar in that they both follow the teacher-student (T-S) framework (like many papers within various fields in machine learning). However, the two are *fundamentally different* in terms of their objectives. Specifically: For forget examples, SCRUB encourages the student to move away from the original model (closer in spirit to gradient ascent ideas like NegGrad and *our improved version of NegGrad that we proposed in this paper*) whereas Bad-T encourages it to move towards a random model, yielding fundamentally different unlearning algorithms.

2. NB: the T-S framework is very widely used in machine learning, for a wide variety of methods, including semi-superivsed learning (Xie et al), self-supervised learning (Grill et al), to name a few applications. By no means would one consider those methods to be the same as each other, or that the existence of one deducts ‘novelty points’ from others, just because they both can be thought of as using some kind of a teacher and some kind of a student. 

3. KL-divergence between model outputs is a very generic and popular similarity metric that almost every method formulated in a T-S framework utilizes. The choice of this metric, by no means contributes to the novelty of such papers. 

4. Our results show SCRUB and Bad-T to have very different performance profiles -- a more reliable indicator of their differences than the common use of terms like “teacher” and “student” in the 2 papers. 

There are also other important differences between SCRUB and Bad-T:

5. As an example, SCRUB includes the task loss as well as the distillation loss, making it not strictly a T-S but rather a hybrid objective: aside from obeying/disobeying the teacher, it also optimizes the task loss w.r.t ground-truth labels on the retain set). 

6. Further, SCRUB comes with a novel reminding procedure (SCRUB+R) to be triggered in applications where we want to avoid producing an unlearned model whose error is “too high”; making SCRUB uniquely able to handle different application scenarios. Again, we wish to stress as well that our empirical comparison between SCRUB and Bad-T shows SCRUB to significantly outperform Bad-T. 

References
- [Xie et al] Self-training with Noisy Student improves ImageNet classification. CVPR 2020.
- [Grill et al] Bootstrap Your Own Latent A New Approach to Self-Supervised Learning. NeurIPS 2020.

**More supporting results**

Table 1: Results for class-unlearning. Class-0 (400 examples) of Cifar-100 (40k examples). The numbers are averaged over 3 runs ± 95% confidence intervals. 

| **Method**    | **test-error**| **retain-error**| **forget-error**  | **loss-MIA** |
|:--------------|:-------------:|:-------------:|:-------------:|:-------------:|
| Original      | 35.84±0.47    | 0.07±0.01     | 0.00±0.00     | 61.67±10.04   |
| Retrain       | 36.92±0.63    | 0.09±0.04     | 100.00±0.00   | 55.67±21.13   |
| NegGrad       | 43.16±1.94    | 6.86±3.20     | 49.50±7.93    | 64.67±11.20   |
| Bad-T@epoch1  | 44.14±1.14    | 8.78±1.29     | 63.83±15.27   | 61.67±12.75   |
| Bad-T@epoch10 | 43.33±0.79    | 7.74±1.99     | 85.33±2.59    | 63.33±17.45   |
| SCRUB+R       | 37.60±0.73    | 0.12±0.06     | 100.00±0.00   | 57.33±17.97   |
| SCRUB         | 38.27±1.91    | 0.78±1.53     | 100.00±0.00   | 48.33±7.59    |


Table 2: Results for selective unlearning. 10 examples from class-0 of Cifar-100 dataset.The numbers are averaged over 10 runs ± 95% confidence intervals

| **Method**    | **test-error**| **retain-error**| **forget-error**  | **loss-MIA** |
|:----------------|:-------------:|:-------------:|:-------------:|:-------------:|
| Original        | 37.46±3.09    | 0.54±1.08     | 0.00±0.00     | 58.00±5.64    |
| Retrain         | 37.54±3.25    | 0.57±1.14     | 19.00±9.80    | 45.00±11.80   |
| NegGrad         | 45.98±4.21    | 11.64±7.45    | 44.00±6.91    | 55.00±14.80   |
| Bad-T@epoch1    | 47.23±3.57    | 13.79±6.37    | 66.00±11.29   | 62.00±18.72   |
| Bad-T@epoch10   | 47.09±3.50    | 13.80±6.07    | 96.00±5.00    | 77.00±10.69   |
| SCRUB+R         | 38.40±2.91    | 0.69±1.28     | 40.00±8.26    | 61.00±10.90   |
| SCRUB           | 38.18±2.96    | 0.48±0.89     | 45.00±8.43    | 56.00±12.25   |


Table 3: Results for selective unlearning. 20 examples from class-0 of Cifar-100 dataset.The numbers are average over 10 runs ± 95% confidence intervals

| **Method**    | **test-error**| **retain-error**| **forget-error**  | **loss-MIA** |
|:-------------|:------------:|:------------:|:------------:|:------------:|
| Original     | 37.46±3.09   | 0.54±1.08    | 0.00±0.00    | 63.00±6.79   |
| Retrain      | 37.74±3.29   | 0.56±1.10    | 19.50±7.44   | 49.00±7.11   |
| NegGrad      | 45.56±4.34   | 11.14±7.79   | 47.00±6.13   | 67.50±8.64   |
| Bad-T@epoch1 | 46.46±3.89   | 12.95±7.26   | 55.50±10.04  | 55.00±13.38  |
| Bad-T@epoch10| 45.88±3.53   | 12.03±5.85   | 92.00±6.35   | 70.00±6.74   |
| SCRUB+R      | 38.39±2.87   | 0.68±1.24    | 23.00±5.39   | 49.50±7.24   |
| SCRUB        | 38.29±2.86   | 0.49±0.88    | 29.00±10.09  | 54.50±11.49  |



Meta-review: The paper is clearly written and the experimental evidence for the improvements of the method is rather solid (together with the new results during rebuttals - which the authors should add to the final version!) and they speak for themselves. All in all, it's a classical "method works better" paper with significant improvements and thorough experimental work that the reviewers appreciated. The authors did a good job in the rebuttal as well.
Decision: Accept (poster)