The Brainy Student: Scalable Unlearning by Selectively Disobeying the Teacher
Authors: Meghdad Kurmanji
Peter Triantafillou
Eleni Triantafillou
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep machine unlearning
machine unlearning
scalable unlearning

Abstract: Deep machine unlearning is the problem of removing the influence of a cohort of data from the weights of a trained deep model. This challenge has enjoyed increasing attention recently, motivated to the widespread use of neural networks in applications involving user data: allowing users to exercise their `right to be forgotten' necessitates an effective unlearning algorithm. Deleting data from models is also of interest in practice for removing out-of-date examples, outliers or noisy labels. However, most previous unlearning methods consider simple scenarios where a theoretical treatment is possible. Consequently, not only do their guarantees not apply to deep neural networks, but they also scale poorly.  In this paper, drawing inspiration from teacher-student methods, we propose a scalable deep unlearning method that breaks free of previous limiting assumptions. Our thorough empirical investigation reveals that our approach significantly improves upon previous methods in being by far the most consistent in achieving unlearning in a wide range of scenarios, while incurring only a minimal performance degradation, if any, and being significantly more scalable than previous methods.

Under review as a conference paper at ICLR 2023
THE BRAINY STUDENT: SCALABLE DEEP UNLEARN- ING BY SELECTIVELY DISOBEYING THE TEACHER
Anonymous authors Paper under double-blind review
ABSTRACT
Deep machine unlearning is the problem of removing the influence of cohort of data from the weights of a trained deep model. This challenge has enjoyed in- creasing attention recently, motivated to the widespread use of neural networks in applications involving user data: allowing users to exercise their 'right to be for- gotten' necessitates an effective unlearning algorithm. Deleting data from models is also of interest in practice for removing out-of-date examples, outliers or noisy labels. However, most previous unlearning methods consider simple scenarios where theoretical treatment is possible. Consequently, not only do their guaran- tees not apply to deep neural networks, but they also scale poorly. In this paper, drawing inspiration from teacher-studert methods, we propose scalable deep un- learning method that breaks free of previous limiting assumptions. Our thorough empirical investigation reveals that our approach significantly improves upon pre- vious methods in being by far the most consistent in achieving unlearning in a wide range of scenarios, while incurring only minimal performance degrada- tion, if any, and being significantly more scalable than previous methods.
INTRODUCTION
Can we make a deep learning model 'forget' a subset of its training data? Aside from being scien- tifically interesting, achieving this goal of Deep Machine Unlearming is increasingly important and relevant from a practical perspective. Regulations such as EU's General Data Protection Regulation (Mantelero] 2013) stipulate that individuals can request to have their data 'deleted' (the 'right to be forgotten"). ). Nowadays, given the ubiquity of deep learning systems in a wide range of applications, including computer vision, natural language processing, speech recognition and healthcare, allow- ing individuals to exercise this right necessitates deep unlcarning algorithms. Further, in addition to privacy considerations, this type of data deletion may also be desirable for several practical ap- plications: removing out-of-date examples, outliers, poisoned samples (Jagielski et al. 2018), noisy labels (Northcutt et al. 2021), or data that may carry harmful biases (Fabbrizzi et al. (022).
However, truly removing the influence of a subset of the training set from the weights of a trained model is hard, since deep models memorize information about specific instances (Zhang al. 2020
2021 Arpit et al. 2017) and their highly non-convex nature makes it difficult to trace the effect of each example on the model's weights. Of course, we can apply the naive solution of re-training the model from scratch without the cohort of data to be forgotten. This conceptually-simpt procedure indeed guarantees that the weights of the resulting model aren't 't influenced by the instances to forget (it performs 'exact unlearning)) ), but the obvious drawback is computational inefficiency: re-training a deep learning model to accommodate each new forgetting request isn't viable practice.
To mitigate the large computational cost of exact unlearning, recent research has turned to approxi- mate unlearning [Iz70 et al. 2021 Golatkar et al., 2020a [b). The goal, as defined in these works, to modify the weights of the trained model in order produce new set of weights that approximates those that would have been obtained by the exact procedure re-training from scratch. That is, they strive to achieve iidistinguishabilitt" between the models produced by the exact and approximate solutions, typically accompanied by theoretical guarantees for the quality of that approximation.
Unfortunately, though, most previous approaches suffer from the following important problems. Firstly, their guarantees are derived in simple scenarios of convex loss functions, where theoretical analysis is feasible, thus excluding the arguably most relevant application of unlearning: deep neural

---

Under review as a conference paper at ICLR 2023
networks. Secondly, in light of recent evidence, model indistinguishabiiity may not be necessary nor sufficient condition for successful unlearning (Tudii et al. 2022] Goel et al. 2022), putting into in the cases where they do apply. Finally, previous
question the soundness of these guarantees approaches suffer from poor scalability, both to the size of the training dataset (and, consequently, the size of the models) as well to the size of the cohort that will be forgotten (Goel et al. 2022).
The desire to scale to larger training datasets and model sizes needs little motivation. Indeed, this scaling has been a large contributing factor of the continuing progress of deep learning (Kaplan et al. 2020), creating an increasing need to also scale deep unlearning algorithms accordingly. In addition, scalability to the size of the forget set very important since, Goel al. (2022) also point out, even a single user might own many samples from the training set that should be deleted, or several users may submit deletion requests in 'bursts' after certain events like revelations of privacy leakages by organization. More generally, applications of removal of out-of-date or poisoned data, noisy labels or data that carries harmful biases don't come with assumptions on the size of those deletion sets. For instance, an organization may wish to delete all data that is older than prescribed retention period. In all these cases, the forget sets can be very large.
Motivated by the above desiderata, we propose approach that breaks free from limiting assump- tions made in previous work and is scalable both to training set and forget set sizes. To that end, propose
drawing inspiration from the teacher-student methodology. which we dub SCalable Remembering and Unlearning with a Brainy Student (SCRUBS). Con- cretely, train "brainy' student model that is smart enough to disobey the teacher to avoid inher- iting information about the forget cohort, while obeying it otherwise, to distill all useful information that it allowed keep. Our thorough empirical investigation reveals that our approach by far the most consistent in achieving unlearning in a wide range of scenarios, while incurring only minimal performance degradation, if any, and being significantly more scalable than previous methods.
unlearning approach
PROBLEM DEFINITION
Notation and preliminaries Let D = y.). denote training dataset containing N exam- ples, where the i'th example is represented by vector of input features 2; and a corresponding class label y/, Let f(=: w) denote a function parameterized by trainable weights w. In this work, we study the case where ff is represented by deep neural network, trained in supervised manner via empirical risk minimization. Specifically, we define the loss function as
1 N
[(w)=
Σ
where denotes the cross entropy loss.
Deep Machine Unlearning We now formalize the problem of deep machine unlearning. We assume that we are given model f(: that has been trained on D, where f denotes neural network and w" its trained parameters, obtained by minimizing Equation [ We will refer to this as the 'original model' (i.e. before any unlearning intervention is performed). We are also given a 'forget set' DD = (0, y/ N C DD comprised of N, examples from D and a 'retain set' DD of N, training examples that the model is still allowed retain information for. For simplicity, consider the standard scenario where D, is the complement of D or, in other words, UDD = D.
Given f(; w ), Dy and D,, the goal of deep machine unlearning is to design a scalable and efficient algorithm for producing a new set of weights w" that contains as little information as possible about Dj while not sacrificing performance. That is, f(; w") and f(; w²) should perform equally well on the retain set, as well as on held-out test data outside of D.
Formally, we want the following two equations to hold:
0
=
(3)
where [(p q) stands for the mutual information between the probability distributions of p and q and (D;()) is the distribution over outputs produced by ʃ with parameters on the dataset D.
2

---

Under review as a conference paper at ICLR 2023
Intuitively, states that the weights w" obtained from unlearning should yield a probability distribution that shares no information with that of the weights w" of the original model, when ingesting data from the forget set. On the other hand, Equation [ states that those two probability distributions should agree on data from the retain set. Therefore, satisfying both Equations 2 and simultaneously would lead to retaining all information about making predictions examples from Dr, while losing all information about making predictions on examples from D
Trade-off's In practice, there may be trade-offs between three desiderata: scrubbing the weights clean of information from y, maintaining accuracy on D,, and computational efficiency. For example, the naive solution of training from scratch on D is a good solution with respect to the first two, but very poor with respect to the third. As another example, setting w" to be random weights indeed satisfies the first and third, but would suffer from very low performance on the second and, finally, not performing any unlearning violates the first but satisfies the second and third.
In agreement with Goel et al. (2022). we define our goal as searching for unlearning algorithms that live on the Pareto frontier of these three desiderata. However, different practitioners may have differ- ent preferences and priorities weighting the amount of deletion versus the amount of performance degradation on the retain set, and offer a knob to express these ipplicationnspeciffe preferences in our framework through the hyperparameters in Equation 6 that we introduce in the next section.
SCALABLE REMEMBERING AND UNLEARNING WITH A BRAINY STUDENT (SCRUBS)
In this section, we present a practical algorithm for optimizing Equations 2 and 3 by casting the unlearning problem into a student-teacher framework. Specifically, we will think of the original model w" as the 'teacher' in this context, and we formulate our goal as training 'student' model w that selectively obeys that teacher. Intuitively, our goal for w" twofold: unlearn D, f while still remembering D,. To that effect, w should be obeyed when teaching about DD,, but it should be disobeyed when teaching about D, Consequently, we desire a 'brainy" student that is smart enough to selectively disobey the teacher when necessary.
En route to presenting our training objective, let us first define:
d(x; w") = DKI, [lP[((
In words, d(r; w") is the KL-divergence between the student and teacher distributions for the ex- ample We make the dependence of dl on w" explicit, since we will optimize the student weights w" while keeping the teacher weights frozen w", treating them as a constant.
Concretely, begin by initializing the student w" to the weights of the teacher w". Since the teacher weights w were trained on all of D, we assume that the teacher performs well on both constituents of D. D, and Dy. Therefore, this initialization step ensures that the student has good performance on D, at initialization time, already fulfilling one our desiderata. Can we then start from this solution and modify it unlearn Dy? In principle, one could do this by optimizing:
min
Σ d(z);w")
(4)
NJ
ZJED
However, in practice, when performing this maximization of the distance between the student and teacher on the forget set, we noticed that, while indeed the performance on D, degrades, as desired, it unfortunately also results in degrading performance on D,, To amend that, we propose to simulta- neously encourage the student to 'stay close" to the teacher on retain examples, while encouraging to 'move away" from the teacher forget examples. Formally, the optimization objective becomes:
min N
1
Σ IPED.
dd w²)
1 N
Σ ZjED,
d(y;;w")
Finally, we empirically found it beneficial also simultaneously optimize for the task loss the retain set, to further strengthen the incentive to perform well there. Our final training objective is then the following:
3

---

Under review as a conference paper at ICLR 2023
min 10
a N
Σ d(Ir; E,ED,
7 Nr
Σ (EDD
(U(xr;"");;n N ByED
Σ d(zy;w")
(6)
where/ stands for the cross-entropy loss and the a and y are scalars that we treat as hyperparameters.
In practice, we found that optimizing the objective in Equation o challenging, due oscillations in the loss throughout training. Intuitively, this is due to the student aiming to simultaneously satisfy two objectives, which may interfere with each other. To address this, we provide a practical recipe for optimization, reminiscent of common 'tricks' used in other min-max problems like in Generative Adversarial Networks (GANs) (Goodfellow al. 2014) where, in each iteration, the discriminator is trained for several steps before performing a single update to the generator. In our case, we iterate between performing an epoch of updates the retain set (the min-step) followed by an epoch of updates on the forget set (the max-step), in an alternating fashion. We also found it helpful to perform a sequence of additional min-steps the end of the sequence. For clarity, we provide pseudocode in Al Algorithm inn the Appendix and discuss these choices in Section
4 RELATED WORK
Unlearning definitions Defining the problem of unlearning is nuanced and an open problem in and of itself. A popular definition of the goal of unlearning is to produce model weights which are indistinguishable from those of model trained from scratch without the cohort to forget (Ginart
ct al. 2019 Graves et al. 2021: Guo et al. 2019; Golatkar et al., 2020a b 202I).
However, there is recent line of criticism against this definition of unlearning. Thudi et al. (2022) theoretically show that we can obtain arbitrarily similar model weights from training on two non- overlapping datasets. This implies that the ability achieve particular set of parameters as the end result of training does not always speak to the training dataset that was used. To strengthen this con- clusion, (Shumailov et al. (2021) recently showed that it's possible to find different training examples that would yield a similar gradient for updating the model, due to the averaging operation over the minibatch when using stochastic gradient descent. Therefore, having an unlearning algorithm output parameters similar to those of model trained from scratch isn't 't a sufficient condition for having achieved unlearning. Furthermore, Goel et al. (022) argue that it's not necessary condition either: by tweaking the training procedure, or changing hyperparameters, one will arrive at significantly different model distributions (Yang & Shami 2020). Therefore, a successful unlearning method may be unnecessarily penalized for not matching a particular wewrrineddffoo-sceeee
model.
Inspired by this criticism, we instead opt for a different definition of the unlearning in our work, as presented in Section [ Specifically, instead of using model retrained from scratch as reference point, we instcad start from the original model trained on all of D, and use that as a reference. Unlearning can then be defined in terms of removing information about D, from that model while retaining the information it already contains about D,..
Unlearning methods for convex models The problem of machine unlearning was first introduced in (Cao & Yang| 2015), where they provide an exact forgetting algorithm for statistical query learn- ing. (Bourtoule et al. (022) proposed a training framework by sharding data and creating multiple models, enabling exact unlearning of certain data partitions. (Ginart et al.) 2019)
the first paper
to introduce a probabilistic definition for machine unlcarning which was inspired by Differential Privacy (Dwork al. 2014), and formed the origin of the idea of the model indistinguishability definition discussed above. More recently, Guo et al. (201)): [zzo et al. (202T): [Sekhari et al. (2021) built upon this framework and introduced unlearning methods that can provide theoretical guaran- tees under certain assumptions.
Unlearning methods for deep networks Recently, approximate unlearning methods were devel- oped that can be applied to deep neural networks. (Golatkar et al. 2020a) proposed an information- theoretic procedure for removing information about D, from the weights a neural network and (Golatkar et al.) 2020b 2 propose methods to approximate the weights that would have been obtained by unlcarning via a linearization inspired by NTK theory (Jacot et al. 2018) in the first case, and based on Taylor expansions in the latter.
4

---

Under review as a conference paper at ICLR 2023
Forget 3 classes (50%)
Forget 4 classes (~ 67%)
Test error (L)
Retain error (L)
Forget error (f)
Test error (L)
Retain error ({)
Forget error (1)
Model
mean
std
mean
std
mean
std
mean
std
mean
std
mean
std
retrain
32.2
2.22
0.0
0.0
99.9
0.19
14.5
2.18
0.0
0.0
99.8
0.14
Original
37.7
0.0
0.0
0.0
1.04
Finetune
33.2
0.0
0.0
4.9
0.5
20.3
2.75
0.0
0.9
Fisher
37.4
2.7
15.4
7.6
84.4
6.4
21.8
7.09
7.83
9.09
84.3
288
NTK
35.2
2.9
0.0
0.0
25.7
22.3
18.5
3.8
0.0
0.0
46.9
5.9
NegGrad
31.1
2.2
1.2
1.5
89.2
1.8
12.8
1.8
0.0
0.0
78.8
1.4
CF-k
32,6
3.1
0.0
0.0
8.2
1.0
18.8
2.4
0.0
0.0
14.3
1.2
EU-k
32.6
3.9
0.2
0.2
55,4
2.2
15.7
1.8
0.2
0.3
64.3
0.7
SCRUBS
30.8
1.3
0.0
0.0
98.6
2.0
15.3
2.3
0.0
0.0
99.6
0.7
Table 1: For large forget sets, class unlearning on CIFAR-6 (with ResNet), SCRUBS achieves significantly better forgetting than all alternatives and doesn't degrade performance.
However, (Golatkar al., 2020a by scale poorly with the size of the training dataset, as comput- ing the forgetting step scales quadratically with the number of training samples. (Golatkar et al.) 2021] addresses this issue, albeit under different restrictive assumption. Specifically, they assume that a large dataset is available for pre-training that will remain 'static", in the sense that no for- getting operations are expected to be applied on it. Unfortunately, this assumption many important realistic applications of unlearning, like in healthcare where, for instance, assume availability large static dataset medical images specific to each problem of interest. (Chundawat al. 2022) is recent preprint that explores another teacher-student framework for unlearning. We include a section in the Appendix where we extensively discuss the differences with and empirically compare against it.
not met in can't can't
Related research areas Differential privacy (Dwork et al. 2014) and life-long learning (Parisi et al. 2019) have shared goals with unlearning. Differential Privacy seeks to ensure that the trained model doesn't It store information about any training instance; strong goal that is difficult to achieve for deep neural networks (Abadi et al. 2016). In contrast, in machine unlearning, we only desire to remove the influence of the training instances in the given forget set, which can be seen as a relax- ation of the same problem. Life-long learning seeks to continuously update model to solve new sets of tasks, without 'catastrophically forgetting' previously- tasks. The notion of forgetting task, though, is distinct from that of forgetting training example: the performance degrade, without having removed influence of particular training examples from the model weights.
task can
Related training objectives The teacher-student methodology has been used in variety of con- texts, including self-supervised (Grill et al. 2020 Chen He (02T) l and semi-supervised learning (Xie al., 2020). A related idea Knowledge Distillation (Hinton et al., 2015), which can be thought as a teacher-student approach with the additional desire for compression, where the stu- dent is chosen to have smaller architecture than the teacher. In addition, our training objective has a contrastive flavour (Chen et al.) 2020) He et al. 2020). due to both pulling the student close to the teacher, and pushing away from it, for different parts of the training set.
5 EXPERIMENTS
The goal of our experimental investigation is to answer the following questions: Q1: How do dif- ferent methods perform for the important understudied case of larger forget sets? Q2: How does SCRUBS fare against the state-of-the-an art models in the established benchmarks? Q3: How do dif- ferent methods fare in terms consistency of performance across different scenarios (datasets, ar- chitectures, training and forget set sizes, class selective forgetting)? Q4: How fast do the different methods run, compared to the assumed-to-be prohibitively-xxensiie retraining from scratch?
5.1 EXPERIMENTAL SETUP
Methods
We compare against state-of-the-art approaches as well various baselines:
5

---

Under review as a conference paper at ICLR 2023
lest accuracy
accuracy
Anetuning
Anctuning
Rsher
100
Rahar
NTK
NTK
NagGrad
NeqGrad
CF-K
CF-c
30
EU-K
EU-K
SCRURS
25
SCRURS
23
10
Forget error
Retain accuracy
Forget
Retain acturaty
error
(a) CIFAR-6.
(b) Lacuna-6.
Figure I:
In large forget sets, SCRUBS is the only top-performer across settings. Each point represents the average across 6 settings: (ResNet, All-CNN} x (class-50%, class-67% selective- 50%); }. using the data from Tables
13 14
and
15
Test accuracy
Test accuracy
Annetninng
Anetuning
Riheer
100
Hshar
100
NaqGrad
75
NeqGrad
CF-K
EUli
EU-k
SCRUAS
25
SCRUBS
10
10
Forget erfor
Retain accuracy
Forget
Retain accuracy
error
(a) CIFAR-S.
(b) Lacuna-5.
Figure 2: In the small scale setting, selective unlearning, SCRUBS is the only top-performer across settings. Each point represents the average across ResNet and All-CNN. On the left, CF-k and Finetuning perform similarly (they perform poorly) so their lines occlude one another, and the same is true of NTK, CF-k and EU-k on the right. For reference, this data lives in Tables and 5
Retrain: Retraining from scratch without the cohort to forget. This method is assumed to not be viable in practice due to its prohibitive computational cost.
Original: The 'original' model trained on all of 'D without any unlearning intervention.
Finetuning: Finetunes the original model on data from the retain set D,,
Similar to Finetuning, starts from the original model and finetunes it both on data from the retain set and the forget set, negating the gradient for the latter. Previous work considered weaker baseline that only trains the forget set, with a negative gradient. We tune this stronger baseline to achieve a good balance between the two objectives.
NegGrad:
Fisher: The Fisher Forgetting method from (Golatkar et al. 2020a)
NTK: The NTK Forgetting method from (Golatkar et al.
2020b).
CF-k: Catastrophic Forgetting-k (CF-k) (Goel et al.
2022) freezes the first (bottom-most)
k layers of the original model and finetunes the remaining (top-most) layers on D,.
EU-k: Exact Unlearning-k (EU-k) (Goel et al.) 2022) freezes the first (bottom-most)
layers of the original model and trains from scratch the remaining (top-most) layers on D,..

---

Under review as a conference paper at ICLR 2023
Counts
Counts
Originas
Foeturting
Fothars
NTK
Neytraad
ScRUBS
Original
Fourtmming
Ruthar
Negdrad
Chi
ScRUBS
Original
Finauriing
Robeer
NTK
Nogpiredd
SERUBS
(a) Small scale (4 settings).
(h) Large scale (8 settings).
(c) Large forget sets (12 settings).
Figure 3: The number of times that each method was (one of the) top performer(s) in terms of the forget enror, across all settings (architectures, datasets, selective class forgetting, forget set size, etc.). We consider a model a top-performer if its 95% confidence interval overlaps with that of the best mean. Small scale counts are over (ResNet, All-CNN) {CIFAR, Lacuna) total), large scale additionally x { {class, selective) (8 total), and large forget set additionally x {class-50%, class-67%; %. selective-50% (12 total). SCRUBS is the most consistent in terms of its ability unlearn.
Metrics We desire the model to have forgotten about D,, without hurting its performance the retain set nor its generalization ability. We therefore report forget error, where higher values are better (denoted f in the tables), and retain error and test error where lower values are better (denoted +. in the tables). For larger-scale experiments, we also consider scale-up factor as an additional metrie: the fraction of the runtime of retraining from scratch over the runtime of the given unlearning algorithm. Finally, we posit that consistency in good performance on the above metrics crucial for practical applicability, so experiment with large number of settings, including different datasets, sizes of forget and training sets, architectures and selective or class forgetting.
Experimental setup We base our investigation on the same two datasets that are used in previous work: CIFAR-10 (Krizhevsky et al., 2009) and Lacuna-10 10 (Golatkar et al., 2020a), which is a dataset derived from VGG-Faces (Cao Yang 2015). We also use same two architectures as previous work: All-CNN (Springenberg al.) (014)) and ResNet-18 18 (He et al. 2016). For fair comparisons with previous work, we follow the standard setup of using a model that was pre-trained on CIFAR- 100 / Lacuna- 100 for the CIFAR- 10 Lacuna- 10 experiments, respectively (Golatkar et al. 2020a b). We run each experiment with 3 random seeds and report the mean and standard deviation. We utilize the public code for the NTK and Fisher methods in our experiments, ensure correctness. We also plan to release our own code upon publication to facilitate future comparisons.
We conduct our empirical investigation in 3 setups, described in further detail in the Appendix:
Large forget set sizes: We significantly increase the % of the training that belongs Dy; y; an axis orthogonal to that of increasing the training set size. We investigate forgetting up 67% of the training set (for reference, standard benchmarks consider up 10%). For this, we train on 6 classes from each of CIFAR and Lacuna ("CIFAR-6' / "Lacuna-6") and perform both selective unlearning. where we forget 50% of the training examples of each class, and class unlearning where we forget either 3 (50%) or classes (~ 67%).
We exactly follow the setup in (Golatkar et al. 2020b) that uses only 5 classes with 100 train, 25 validation and 100 test examples per class. The forget set contains 25 examples from class 0 (5%). This small-scale setting allows comparing to NTK, which doesn't 't scale to larger datasets.
Small-scale:
from each of CIFAR and Lacuna
Large-scale: We exactly follow the setup in (Golatkar al. 2020a) that uses all 10 classes of each of CIFAR and Lacuna, and considers both a class unlearning scenario where the forget set is the entire training set for class 5 (10%). as well as a selective unlearning one where 100 examples of class 5 are forgotten (0.25% in CIFAR and 2% in Lacuna).
5.2 FINDINGS AND TAKE-AWAYS
We present results exploring larger forget set sizes (Q1) in (an Tables [5). and Figure (and Figures] B) In investigating the competitiveness of SCRUBS on previously V- established benchmarks (Q2), we report results for the small-scale in Figure (and Tables and

---

Under review as a conference paper at ICLR 2023
Test accurary
Test accuracy
NegGrad
CF-4:
75
SCRUBS
Scale-up
Retain accuracy
Scale-up
Retain accuracy
Finetunina
Fisher
NegGradi
EU-k
SCRUDD
Forget
Forget
error
(a) CIFAR-10 class unlearning.
(b) CIFAR-10 selective unlcarning.
Figure 4: In the large-scale setting, SCRUBS is the only top-performer across settings. Selective forgetting is much harder for all, but especially for EU-k and Fisher. Scale-up is multiplied by 10 for visualization purposes. Each point represents the average across ResNet and All-CNN variants. both subplots, CF-k and Finetuning perform similarly (they perform poorly) so their lines occlude one another. For reference, the data lives in Tables and 7 (left) and Tables 8 and 9 (right).
We showcase the consistency of each method's ability to unlearn (Q3) in Figure and we report results for the scale-up factor in Figures (and Figure (Q4). Tables and Figures mentioned in brackets can be found in the Appendix.
and for the large-scale in Figure 4 (and Tables 6
7
We summarize main findings below.
SCRUBS is the sole method that is consistenily top-performer in terms of achieving unlearning (Figure 3) while incurring minimal performance degradation, if any (Table
Figures and and being very computationally efficient (Figure
Original, expected, enjoys good performance the retain and test sets, but it has low forget set error, reflecting that no effort was made to unlearn Dy-
Finetuning retains the performance of Original for the most part, but also fails at forgetting.
The previous state-the-art NTK isn't among the top-performers in terms of forgetting, though it at least doesn't 't hurt the model's performance (Figures [ and 2). However, our thorough investigation of larger forget set sizes revealed notable exception to its poor performance, on selective unlearning with large forget set sizes (Figure [8. This warrants further investigation to gain deeper understanding of NTK, though it isn't viable in prac- tice since it can't 't scale beyond very small datasets and performs poorly on several settings.
The previous thate-the-art Fisher sometimes achieves forgetting (Figures Tb 14b and Fal 8). though not consistently (Figure and unfortunately usually degrades the model's per- also very slow (Figure 4 actually exceeding the runtime of the retrain from scratch; observation also made in (Goel et al. 2022).
formance substantially.
The baseline of NegGrad turns out to be a strong one in terms of achieving a balance between unlearning and retaining performance and encourage future work also port this bascline. However, SCRUBS outperforms it in terms of forget error in several scenarios, as seen in Figure and is much more computationally efficient (see Figure 4
CF-k completely fails to forget in most cases, whereas EU-k is more successful in that regard (though not consistently (Figure 3). This is expected, since EU-k trains the top layers from scratch, thus increasing the chances of removing information about D, whereas CF-k finetunes those layers instead. notable failure case that we discovered for EU-k is selective forgetting (notice the contrast between EU-k's forget error between Figures 4a and and between Figures 7b and [b). This interesting finding may speak to where class instance information is stored in neural networks and warrants further investigation.
Analysis of Training Dynamics Next, we illustrate the training dynamics of SCRUBS and the importance of different design choices. As reminder, the student is initialized from the teacher and
8

---

Under review as a conference paper at ICLR 2023
100
mtain set
forgel-set
25
validation-se
80
15
retain-set
20
forget-set
validationset
0
10
spach
opoch
(a) Only max steps (Equation 4
(b) SCRUBS with insufficient max steps.
70
retaii-sse
retain-set set
80
forget-set
Iorget-set
velidalion-set
validation set
92
10-
50
10.0 17.
175 200
10
opach
(c) SCRUBS with too many max steps.
(d) SCRUBS.
Figure 5: Illustration of training dynamics of SCRUBS variants, on CIFAR-5 with a ResNet model. Performing the right interleaving of min-steps and max-steps is important for achieving a good balance between high forget error and low retain and validation errors.
subsequently undergoes an alternating sequence of max-steps and min-steps; the former encouraging the student move far from the teacher on the forget set, and the latter encouraging stay close to the teacher on the retain set. We also found it useful to perform sequence of additional min-steps after the alternating sequence. We now explore the effect of these decisions.
First, we show that performing only max-steps, by optimizing Equation 4 is not a good solution. Simply pushing the student away from the teacher on the forget set achieves forgetting but unfor- tunately also hurts the retain and validation set performance (Figure 5a Therefore, alternating between max-steps and min-steps is necessary. However, is important to find the right balance. For instance, as seen in Figure performing too few max-steps leads to the unwanted consequence of the forget error dropping. On the other hand, removing the final sequence of only min-steps is also harmful, as shown in Figure 5c that trains for larger number of epochs of an equal number of (alternating) max-steps and min-steps without achieving a good balance at any point throughout the trajectory. On the other hand, SCRUBS (Figure (5d) achieves good balance of high forget error and low retain and validation error simultancously. The Appendix shows additional examples of training dynamics (Figures 9 TO TTI. hyperparameter sensitivity (Table 21], possible trade-off's of wwwsseeeeerrosoeaaeeeee which provides a small but consistent added protection against degrading performance (Figure 12
forgetting- versuus performance (Table 20) and an ablation of the cross-entropy term in Equation 6
CONCLUSION
In this paper, we have proposed SCRUBS, new approach deep machine unlearning, wherein 'brainy' student selectively obeys the teacher model, in order to distill useful information from it, while the same time erasing information pertaining to the cohort to forget. Our thorough empirical investigation revealed that SCRUBS is by far the most consistent approach in achieving forgetting, while only incurring a minimal performance degradation, if any, in all scenarios we considered, and being significantly more computationally efficient than previous methods. Areas for future work include developing practical algorithms to improve the stability of optimization of SCRUBS and proving theoretical guarantees for the mutual information bounds. Overall, compared to previous works, we take a practical perspective and propose a strong- strong-performing model that is also very computationallly-fficii efffiiicil and consistent top top-performer across wide range of scenarios. We hope that our work opens the door to practical applications of unlearning, enabling efficient deletions of (any amount of) ofte noisy, outlier data in large models, trained on large datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In International conference machine learning, pp- 233-242. PMLR, 2017.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 202/ IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pp. 463-480. IEEE, 2015.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning- PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli, Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. arXiv preprint arXiv:2205.08096, 2022.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, ((3-4):211 1-001,4-77,2000 2014.
Simone Fabbrizzi, Symeon Papadopoulos, Eirini Ntoutsi, and loannis Kompatsiaris. survey on bias in visual datasets. Computer Vision and Image Understanding, pp. 103552, 2022.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Shashwat Goel, Ameya Prabhu, and Ponnurangam Kumaraguru. Evaluating inexact unlearning requires revisiting forgetting. arXiv preprint arXiv:2201.0664/ 2022.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In European Conference on Computer Vision, pp. 383-398. Springer, 2020b.
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 792-801, 2021.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, Advances in neural information processing systems, 27, 2014.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11516-11524, 2021.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraaro Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284. 2020.
10

---

Under review as a conference paper at ICLR 2023
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/1 2(7), 2015.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data dele- tion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2008-2016. PMLR, 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances neural information processing systems, 31, 2018.
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Ma- nipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP), pp. 19-35. IEEE, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alessandro Mantelero. The proposal for general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 29(3):229-22) 220, 2013.
Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103. 2021.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 11::54771, 2019.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Pro- cessing Systems, 1188075180888 2021.
Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Er- dogdu, and Ross J Anderson. Manipulating sgd with data ordering attacks. Advances in Neural Information Processing Systems, 34:1 18021-880222 2021.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1412.6644, 2014.
Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22), pp. 4007-4022, 2022.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves ImageNet classification. In Conference on Computer Vision and Pattern Recognition, pp. 10687-10698. 2020.
Li Yang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415:295-316, 2020.

---

Under review as a conference paper at ICLR 2023
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 2021.
Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret re- vealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 253-261, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DISCUSSION OF LIMITATIONS AND ADDITIONAL FUTURE WORK
SCRUBS has shown impressive results in terms of being consistently a top-performer in terms of unlearning, with a minimal drop in performance compared to previous works. However, SCRUBS has limitations that we hope future work will address. Primarily, the lack of theoretical guarantees a limitation of our method, While this is an important drawback, theoretical guarantees are chal- lenging for deep neural networks. Previous works associated with guarantees, despite offering great insights, suffer from practical limitations. We aim to fill this gap instead. However, we look forward to future work that strives to strike compromise between effective unlearning, good performance, scalability, and theoretical insights. Another limitation of SCRUBS is the difficulty and instability associated with tuning the min-max objective, as shown in the literature e.g. for GANs. For instance, this can lead to oscillating behaviour, as we show in Figure 5 We remedy this to a large extent in practice by providing a practical algorithm that works well, showing consistently improved results over prior work, but there is room for improvement on this front for future work.
We hope that future work also continues to push the limits of scalability. In this work, we have made important strides in this direction. However, the datasets and models we consider aren't too large, in order to allow comparisons to previous works that would not be feasible to run for larger scale experiments. An interesting topic of future work is investigating the interplay between SCRUBS and other scalable algorithms like NegGrad with increasing amounts of scale.
Further, our extensive investigation has revealed interesting trade-offs between different models. However, a solid understanding of the stengths and weaknesses of each, as well as which factors im- pact their performance the most, remains elusive. For instance, our investigation revealed a scenario (selective unlearning in cases with large forget set sizes) where NTK and Fisher, albeit performing very poorly in most cases, perform uncharacteristically well (see e.g. Figure (8) This is a surprising finding that we've dicted it. Therefore, we hope that future work carefully studies existing methods to gain a deeper understanding of them.
ve uncovered, in the sense that previous knowledge could not have trivially pre-
Another really interesting future work direction is to investigate how different unlearning algorithms interact with different architectures, like Transformers, and loss functions, like self-supervised learn- ing. In addition, future work can explore the effect of class imbalance on unlearning. For instance, is it easier to unlearn a class that has few training examples, compared to a class that has many training examples? Do the evaluation metrics need to be adapted for this scenario?
A.2 MORE EXPERIMENTAL DETAILS
Datasets. We have used CIFAR-10 and Lacuna-10 10 datasets for evaluation purposes. CIFAR-10 10 consists of 10 classes with 60000 color images of size 32 x 32. In our experiments, the train, test, and validation sizes are 40000, 10000, and 10000 respectively. Lacuna- 10 is a dataset derived from VGG-Faces (Cao & Yang 2015). We have followed the same procedure described in (Golatkar) let al. 2020a) to build Lacuna. We randomly select 10 celebrities (classes) with at least 500 samples. We use 100 samples of each class form the test-set, and the rest make the train-set. All the images are resized to 32 x 32. We also use CIFAR- 100 and Lacuna-100 100 to pretrain the models. Lacuna-100 built in a similar way as Lacuna- 10, and there is no overlap between the two datasets. We have not applied any data augmentation throughout the experiments.
Small-Scale datasets. We followed the same procedure as decribed in (Golatkar et al. 2020b) to create the small versions of CIFAR-10 and Lacuna- 10, namely CIFAR-5 and I Lacuna-5. To this end, we take the first classes of each dataset and randomly sample 100 images for each class. We make the train and test sets by sampling from the respective train and test sets of CIFAR-10 and Lacuna- 10. We also make 25 samples from each class from the train set to create the validation sets. For large forget-set experiments, CIFAR-6 and Lacuna-6, we repeat the same process by taking the first 6 classes of each dataset.
Metrics Here, define mathematically the metries used in our tables: retain error, forget error and test error. Let D,, D, and D, denote the retain and forget portions of the training dataset, and a test dataset of heldout examples, respectively. We define error (Frr) follows:
13

---

Under review as a conference paper at ICLR 2023
Err(D) =1
Σ (D.D)ED
1[arg max (xx;(( yel
where f. parameterized by the neural network model (comprised of a feature extractor followed by a softmax classifier layer), arg max(f w)) is the label that the model thinks is most likely for example 2i, and [xx] the indicator function that returns 1 if is True and 0 otherwise.
Based on the above, the retain error, forget error and test error are computed as Err(D,), Err(D)) and (rr(D;), respectively.
Models. We use the same models with the same architectural modifications in (Golatkar et al. b). For AlI-CNN, the number of layers reduced and batch normalization added before each non-linearity. For Resnet, ResNet-18 architecture is used. For small scale experiments, the number filters is reduced by 60% in each block. For the large-scale experiments, the exact archi- tecture has been used.
Pretraining. Following the previous work for consistency, we apply pretraining. Specifically, for CIFAR datasets, we have pretrained the models CIFAR- 100. For Lacuna, have pretrained the models on Lacuna-106 100. We pretrain the models for 30 epochs using SGD with a fixed learning rate of 0.1, Cross Cross-Entropy loss function, weight decay 0.0005, momentum 0.9, and batch size 128.
Baselines. 'Original' is the model trained on the entire dataset D. For 'Retrain', we train the same architecture on D,,, with the same hyperparameters used during training of the original model. For Finetune', we fine-tune the 'original' model on D, for 10 epochs, with fixed learning rate of 0.01 and weight-decay 0.0005. For 'Negative Gradient', fine-tune the 'original' model using the following loss:
Del Σ
D4,l Σ (f(zz;
1
((w) =
(8)
D
D
Where 3 E [0, 1]. We have tuned / to get a high forget-error while not destroying retain-error. For small-scale experiments, weight-decay. For large-scale experiments 3 = 0.9999 and we have trained for 5 epochs, with SGD, 0.01 Ir, and 0.0005 weight-deay. Please note that small 3 result in explosion quickly. For 'CF-k', we freeze the first k layers of the network and finetune the rest layers with D,. We use the same setting as 'Finetune' baseline. For 'EU-k' we freeze the first k layers, and re-initialize the weights of the remaining layers and retrain them with D,. As all the models are pretrained on larger datasets, for re-initializing we use the weights of the pretrained models. For 'EU-k' we use the same settings as the 'Retrain' baseline. In both 'EU-k' and "CF-k' baselines, for both ResNet and All-CNN we freeze all the layers except for the last block of the network. For Resnet the last block is block4 and for All-CNN, the last block of layers is the 9th sequential block.
0.95 and we have trained for 10 epochs, with SGD, 0.01 Ir and 0.1
SCRUBS parameters. We train SCRUBS using Algorithm [ Throughout the experiments, we tune the parameters to get high forget-error while retaining the retain and validation error of the original model. We use the same optimizer for both min and max steps. We observed that for small- scale settings 'Adam' optimizer works better, while in large-scale settings both 'Adam' and 'SGD' could be used. For all experiments, we initialize the learning rate at 0.0005 and decay it by 0. after a number of min and max steps. Decaying the learning rate crucial to control the oscillating behaviour of our min and max optimization. We apply a weight decay of 0.1 for small-scale setting and 0.0005 for large scale experiments, with a momentum of 0.9. Finally, we use different batch sizes for the forget-set and the retain-set to control the number of iteration in each direction, i.e the max and the min respectively. We report these in Table 2
System specification. For scale-up experiments, All the codes are written and executed in Python 3.8, on an Ubuntu 20 machine with 40 CPU cores, a Nvidia GTX 2080 GPU and 256GB memory.
A.3 PSEUDOCODE
For clarity, we provide pseudocode for training SCRUBS in Algorithm T code public upon acceptance facilitate future research.
We also plan to make our
14

---

Under review as a conference paper at ICLR 2023
model
dataset
unlearning-type
forget-set bs
retain-set bs
steps
min steps
CIFAR-I0
class
512
128
3
CIFAR-10
selective
16
64
5
5
Lacuna-I 10
class
128
128
5
Lacuna-li 10
selective
32
32
4
4
CIFAR-5
selective
32
10
ResNet
Lacuna-5
selective
32
32
5
10
CIFAR-6
class
64
32
10
CIFAR-6
selective
64
32
5
10
Lacuna-6
class
32
32
2
10
Lacuna-6
selective
64
32
4
8
CIFAR-10
class
512
256
4
CIFAR- 10
selective
16
64
5
5
Lacuna- 10
class
32
32
4
Lacuna-10 10
selective
8
32
2
4
CIFAR-5
selective
16
AII-CNN
Lacuna-5
selective
32
32
5
10
CIFAR-6
class
32
32
10
CIFAR-6
selective
64
32
3
10
Lacuna-6
class
32
32
10
Lacuna-6
selective
32
32
3
10
Table 2: SCRUBS hyperparameters for each experiment
Algorithm Training Brainy Student
Require: Teacher weights wo
Require: Total max steps MAX-STEPS
Require: Total steps STEPS
Require: Learning rate
w + w
while < STEPS do
i MAX-STEPS then t +- DO-MAX-EPOCH(u
end
nh + DO-MIN-EPOCHH end while
Algorithm 2 DO-MAX-EPOCK
Require: Student weights "
Require: Learning rate e
Require: Batch size
Require: Forget set Dy
Require: Procedure NEXT-BATCH
4 NEXT-BATCHDD,,B
while do
2h <- th cVa..
1
Σ d(xy:
um b
ByEb
NEXT-BATCH(D,, B)
end while
15

---

Under review as a conference paper at ICLR 2023
Algorithm 3 DO-MIN-EPOCH
Require: Student weights wn
Require: Learning rate
Require: Batch size B
Require: Retain set D,
Require: Procedure NEXT-BATCH
f TEXT-BATCH(D,, while do
w <- w - S
Σ (E
+//"""++++++0)))) od(3,; + (xx w²)),
b NEXT-BATCH(D,..) end while
CIFAR-10
Lacuna-10
ResNet
AlI-CNN
ResNet all
All-CNN
Model
all
selective
all
selective
selective
all
selective
Finetune
3.8
3.09
3.33
3.03
1.7
2.03
2.16
2.00
Fisher
0.08
0.07
0.16
0.14
0.08
0.07
0.16
0.15
NegGrad
3.4
2.96
2.30
2.97
1.66
1.5
2.41
2.27
CF-k
3.55
3.17
3.37
2.91
3.42
3.20
3.27
3.11
EU-k
1.41
1.26
1.34
1.20
1.39
1.28
1.32
1.26
SCRUBS
7.84
7.41
6.36
5.33
2.17
1.95
2.81
2.48
Table 3: Scale-up factor: the fraction of the runtime retrain from scratch over the runtime of
each given unlearning algorithm. That scale-up value of for an unlearning algorithm means that that algorithm runs X times faster than retrain from scratch.
A.4 ADDITIONAL TABLES
In this section, we provide the results for all scenarios we studied, for completeness. Each of these tables has been used in aggregated results reported in the main paper (e.g. in the barplot of counts of each method being a top-performer w.r.t forget error, in Figure as well as in the spider plots where averages across settings reported, in Figures 7
Overall, our thorough investigation reveals interesting trade-offs between methods, summarized in Section [ in the main paper. Our broad take-away is that SCRUBS is the most consistent in achieving unlearning, while causing only a minimal, if any, performance degradation and being
very computationallly-effiiaa
CIFAR-5 Retain error (L)
Lacuna-5 Retain error ({)
Test error (L)
Forget error (t) mean
Test error (L) mean
Forget error (f) mean
Model
mcan
std
mean
std
std
std
mean
std
std
Retrain
24.9
2.5
0.0
0.0
28.8
5.9
5.8
0.4
0.0
0.0
4.8
3.4
Original
24.2
0.0
0.0
0.0
0.4
Finetune
24.3
0.0
0.0
0.0
0.3
0.0
0.0
0.0
Fisher
31.6
3,4
14.0
6.0
4.8
5.2
14.0
3.6
6.7
3.3
6.4
8.3
NTK
24.4
2.6
0.0
0.0
22.4
9.2
5.6
0.5
0.0
0.0
0.0
0.0
NegGrad
25.5
1.1
0.0
0.0
41.3
6.1
6.1
0.7
0.0
0.0
1.3
2.3
CF-k
22.6
1.9
0.0
0.0
0.0
0.0
5.8
0.4
0.0
0.0
0.0
0.0
EU-k
23.5
1.1
0.0
0.0
10.7
2.3
5.9
0.6
0.0
0,0
0.0
0.0
Bad-T
29.93
2.34
7.44
1.79
9,33
6.
13.6
0.57
1,58
1.34
0.0
0.0
SCRUBS
24.2
1.6
0.0
0.0
40.8
1.8
6.2
0.73
0.0
0.0
24.8
5.2
Table Small-scale results with ResNet. SCRUBS is the top-performet in terms of forgetting with minimal performance degradation.
16

---

Under review as a conference paper at ICLR 2023
CIFAR-5
Lacuna-5 Retain error (L)
Test (L)
Retain error (4)
Forget error (t)
Test (})
Forget error
Model
std
std
mean
std
std
Retrain
24.36
1.61
0.13
0.28
9.12
4.6
0.38
0.0
0.0
4.67
Original
24.08
1.86
0.17
0.38
0.0
0.0
4.53
0.47
0.0
0.0
0.0
Finetune
23.48
0.04
9.77
10.76
6.63
13.22
19.33
40.03
Fisher
42.64
31.83
10.47
16.83
$2.53
13.87
$1.09
14.54
39.33
NTK
24.16
0.17
0.38
13.6
8.29
4.47
0.47
0.0
0.0
3.33
4.68
NegGrad
26.07
1.21
0.56
0.49
36.00
10.58
5.27
0.76
0.14
0.12
12.00
13.86
CF-k
22.67
1.55
0.00
0.00
0.00
0.00
4.67
0.70
0.00
0.00
0.00
0.00
EU-k
25.87
0.64
3.23
1.69
8.00
6.93
5.20
0.20
0.00
0.00
0.00
0.00
Bad-T
29.27
0.83
1.05
1.05
0.0
0.0
9.5
1.84
0.95
0.74
0.0
0,0
SCRUBS
23.88
1.78
0.08
0.12
40.8
8.2
3.87
0.72
0.0
0.0
25.33
4.13
Table 5: Small-scale results with AII-CNN. SCRUBS is the top-performer terms of forgetting with minimal performance degradation.
CIFAR-I0
Lacuna-10 Retain error (L)
Test error (L)
Retain error (L)
Forget error (f)
Test error (L)
Forget error (t)
Model
mean
std
mean
std
mean
std
mean
std
mean
std
mean
std
Retrain
14.72
0.16
0.0
0.0
100.0
0.0
2.87
0.34
0.0
0.0
99.75
0.56
Original
16.56
0.1
0.0
0.0
0.0
0.0
3.07
0.26
0.0
0.0
0.0
0.0
Finetune
16.41
0.09
0.0
0.0
0.0
0.0
3.02
0.37
0.0
0.0
0.0
0.0
Fisher
26.42
1.41
2.45
0.84
100.0
0.0
3.33
0.54
0.0
0.0
100.0
0.0
NegGrad
17.84
1.46
1.74
2.55
91.26
7.73
3.41
0.17
0.00
0.00
14.90
1.78
CF-k
15.31
0.12
0.00
0.00
0.03
0.01
2.89
0.22
0.00
0.00
0.00
0.00
EU-k
18.73
0.42
0.00
0.00
98.79
0.18
3.19
0.17
0.01
0.02
4.06
0.83
SCRUBS
15.73
0.17
0.51
0.02
100.0
0.0
3.69
0.36
0.28
0.23
100.0
0.0
Table 6: Large-scale, class unlearning results with ResNet. SCRUBS and EU-k are the top- performers in this setting in terms of forgetting with minimal performance degradation. Note, how- ever, that EU-k doesn't perform strongly across the board (Figure 3 and in particular performs very poorly in selective unlearning (notice the contrast between EU-k's forget етто between Figures 4a and 4b and between Figures 7b and 8b). Fisher is also a top-performer in terms of forget error in this setting too, but CIFAR causes large degradation in test error, is often observed for this method.
CIFAR-10
Lacuna- 10 Retain error (L)
Test
Retain
(L)
Forget error error (f)
Test (})
Forget error
Model
mean
mean
std
mean
std
mean
std
mean
Retrain
13.97
0.19
0.0
0.0
100.0
1.59
0.36
0.0
0.0
100.0
Original
15.56
0.25
0.0
0.0
0.0
0.0
1.56
0.33
0.0
0.0
0.0
0.0
Finetune
15.39
0.22
0.0
0.0
0.0
0.0
1.67
0.44
0.0
0.0
0.0
0.0
Fisher
27.4
2.28
3.66
1.03
99.0
0.0
1.78
0.29
0.0
0.0
89.0
0.0
NegGrad
17.87
0.31
0.58
0.13
87.22
1.67
0.17
0.00
0.00
6.56
1.13
CF-k
14.99
0.23
0.00
0.00
0.00
1.48
0.36
0.00
0.00
0.00
BU-k
15,30
0.13
0.14
100.00
1.74
0.45
0.00
77.19
39.51
SCRUBS
15.06
0.14
0.12
0.03
100.0
100.0
Table 7: Large-scale, class unlearning results with AII-CNN, SCRUBS is the top-performer in terms of forgetting with minimal performance degradation.
17

---

Under review as a conference paper at ICLR 2023
CIFAR-10
Lacuna- 10
Test error (L)
Retain error (L)
Forget error (t)
Test error (U)
Retain error (Q)
Forget error (1)
Model
mcan
std
mean
std
mcan
std
mean
std
mean
std
mean
std
Retrain
17.4
0.14
0.0
0.0
29.67
3.21
2.7
0.2
0.0
0.0
1.0
1.0
Original
17.36
0.14
0.0
0.0
0.0
0.0
2.73
0.15
0.0
0.0
0.0
Finetune
17.37
0.11
0.0
0.0
0.0
0.0
2.63
0.12
0.0
0.0
0.0
0.0
Fisher
21.23
0.27
2.88
0.54
3.0
2.65
3.1
0.35
0.0
0.0
0.0
NegGrad
22.7
0.6
4.1
0.5
53.7
6.8
4.7
0.2
0.9
0.1
13.0
1.0
CF-k
17.4
0.1
0.0
0.0
0.0
0.0
2.7
0.2
0.0
0.0
0.0
0.0
EU-k
21.8
0.2
0.4
0.6
23.7
3.5
2.9
0.1
0.0
0.0
0.0
0.0
SCRUBS
18.04
0.2
0.0
0.0
70.33
4.16
3.0
0.0
0.0
0.0
4.67
3.06
Table 8: Large-scale, selective unlearning results with ResNet. SCRUBS and NegGrad are the top-performers in terms of forgetting, though NegGrad has worse test performance than SCRUBS in both cases. Note also that NegGrad isn't as consistent at forgetting across settings as SCRUBS, as can be seen in Figure
CIFAR-10
Lacuna- 10
Test error (L)
Retain error (L)
Forget error (f)
Test error (L)
Retain error (L)
Forget error (1)
Model
mean
std
mean
std
mean
std
mean
std
mean
std
mean
std
Retrain
16.47
0.21
0.0
0.0
25.67
2.31
1.6
0.44
0.0
0.0
0.67
0.58
Original
16.43
0.08
0.0
0.0
0.0
0.0
1.53
0.31
0.0
0.0
Finetune
16.5
0.18
0.0
0.0
0.0
0.0
1.43
0.21
0.0
0.0
0.0
Fisher
21.39
1.22
4.0
1.44
13.0
11.27
1.87
0.21
0.01
0.02
0.0
NegGrad
21.36
0.34
3.23
0.37
45.33
2.89
2.77
0.25
0.40
0.07
8.67
0.58
CF-k
16.29
0.07
0.00
0.00
0.00
0.00
1.53
0.31
0.00
0.00
0.00
0.00
EU-k
17.62
0.61
0.11
0.11
0.33
0.58
1.83
0.47
0.00
0.00
0.00
0.00
SCRUBS
16.55
0.11
0.0
0.0
29.33
3.21
2.07
0.31
0.0
0.0
1.67
0.58
Table 9: Large-scale, selective unlearning results with AlI-CNN. SCRUBS and NegGrad are the top-performers in terms of forgetting, though NegGrad has worse test performance than SCRUBS in both cases. Note also that NegGrad isn't as consistent at forgetting across settings as SCRUBS, as can be seen in Figure
Forget classes (50%)
Forget 4 classes (~ 67%)
Test error (L)
Retain error (L)
Forget error (t)
Test error (L)
Retain error ({)
Forget error (f)
Model
mean
std
mean
std
mean
std
mean
std
mean
std
mean
std
retrain
32.2
2.22
0.0
0.0
99.9
0.19
14.5
2.18
0.0
0.0
99.8
0.14
Original
37.7
0.0
0.0
0.0
1.04
Finetune
33.2
0.0
0.0
4,9
0.5
20.3
2.75
0.0
0.9
Fisher
37.4
2.7
15.4
7.6
84.4
6.4
21.8
7.09
7.83
9.09
84.3
2.8
NTK
35.2
0.0
0.0
25.7
22.3
18.5
3.8
0.0
0.0
46.9
5,9
NegGrad
31.1
2.2
1.2
1.5
89.2
1.8
12.8
1.8
0.0
0.0
78.8
1.4
CF-k
32,6
3.1
0.0
0.0
8.2
1.0
18.8
2.4
0.0
0.0
14.3
1.2
EU-k
32.6
3.9
0.2
0.2
55.4
2.2
15.7
1.8
0.2
0.3
64.3
0.7
Bad-T
32.7
2.4
0.22
0.19
99.7
0.6
14.5
2.6
0.0
0.0
97.1
2.6
SCRUBS
30.8
1.3
0.0
0.0
98.6
2.0
15.3
2.3
0.0
0.0
99.6
0.7
Table 10: For large forget sets, class unlearning on CIFAR-6 (with ResNet), SCRUBS achieves significantly better forgetting than all alternatives and doesn't degrade performance.
18

---

Under review as a conference paper at ICLR 2023
Forget 50% Retain error (4)
Forget 67% Retain error (Q)
Test
(L)
Forget error (1)
Test error (U)
Forget error
Model
mean
std
mean
std
mean
std
mean
std
mean
std
mean
std
Retrain
28.56
1.68
0.0
0.0
100.0
0.0
13.17
2.25
0.0
0.0
100.0
Original
34.22
0.0
9.81
30.33
9.28
1.17
1.61
4.08
Finetune
29.11
3.5
0.33
0.58
25.44
20.21
13.5
5.07
0.0
0.0
21.67
7.35
Fisher
42.33
4.51
19.89
4.34
75.67
13.13
27.67
13.34
12.67
5.35
80.42
14.05
NTK
31.56
1.17
0.0
0.0
27.11
8.95
25.33
5.25
0.83
1.04
24.33
9.96
NegGrad
29.78
4.03
2.44
2.04
100.0
0.0
11.0
2.78
0.33
0.58
100.0
0.0
CF-k
31.67
3.48
0.0
0.0
6.89
10.22
19.0
1.5
0.0
0.0
6,33
7.72
EU-k
33.67
1.67
5.0
0.33
100.0
0.0
13.33
2.84
1.67
1.15
100,0
0,0
Bad-T
31.22
4.79
0.67
0.88
98.33
0.67
15.33
2.89
0.67
0.76
87.33
5.11
SCRUBS
28.78
2.36
0.0
0.0
100.0
0.0
12.67
1.44
0.0
0.0
99.92
0.14
Table Large forget sets, class unlearning on CIFAR-6 with AII-CNN. SCRUBS, NegGrad and EU-k are the top-performers in this setup, though EU-k sometimes incurs performance degradation. Note that, generally, EU-k doesn't 't perform strongly across the board (Figure 3) and in particular performs very poorly in selective unlearning (notice the contrast between EU-K's forget error be- tween Figures [4 and (4 and between Figures 76 and 8b). NegGrad also isn't nearly as consistent SCRUBS in forgetting across different scenarios (Figure
Forget 50% Retain error (L)
Forget 67% Retain error (L)
Test error (L)
Forget error (t) mean
Test error (L)
Forget error (t)
Model
mean
mean
std
std
mean
std
mean
mean
Retrain
3.9
0.7
0.0
0.0
97.9
3.7
1.5
0.9
0.0
0.0
96.3
6.1
Original
6.1
1.3
0.2
0.2
0.1
0.2
6.0
1.7
0.3
0.3
0.1
0.1
Finetune
3.8
0.4
0.0
0.0
4.1
1,4
2.5
1,8
0.0
0.0
4.1
2.6
Fisher
7.4
1.9
3.4
0.8
95.6
4.0
4.3
1.3
1.8
1.2
95.3
3.5
NTK
5.0
0.9
0.0
0.0
6.0
0.9
4.5
1.3
0.0
0.0
7.9
2.5
NegGrad
3.9
0.7
0.0
0.0
64.7
9.5
2.7
1,3
0.0
0.0
40.5
4.8
CF-k
4.1
0.7
0.0
0.0
0.8
0.2
3.5
1.3
0.0
0.0
0.3
0.4
EU-k
3.3
0.0
0.0
24.7
6.2
0.5
0.0
0.0
4.9
Bad-T
4.44
0.19
2.0
0.33
99.89
0.19
1.17
0.76
1.17
0.76
91.08
SCRUBS
0.0
0.0
99.2
1.4
0.3
Table 12: Large forget sets, class unlearning Lacuna-6 with ResNet. SCRUBS is the top- performer in terms of forgetting with minimal performance degradation.
Forget 50% Retain error (L)
Forget 67%
Test error (L)
Forget error (f) mean
Test error (+)
Retain error (J)
Forget error (1)
Model
mean
std
mean
std
std
mean
std
mean
std
mean
std
Retrain
2.89
0.77
0.0
0.0
100.0
0.0
0.17
0.29
0.0
0.0
100.0
0.0
Original
4.44
0.0
0.0
0.0
0.58
Finetune
2.44
0.19
0.0
0.0
10.22
1.02
4.17
5.97
2.83
4.91
36.0
50.77
Fisher
44.78
11.6
39.78
11.97
83.22
20.61
47.67
4.54
44.5
7.47
84.75
16.45
NTK
3.56
0.38
0.0
0.0
6.44
0.51
2.83
0.76
0.0
0.0
7.17
3.19
NegGrad
2.33
0.33
0.0
0.0
100.0
0.0
0.33
0.29
0.0
0.0
100.0
0.0
CF-k
3.0
0.58
0.0
0.0
0.22
0.19
2.17
0.29
0.0
0.0
0.0
0.0
EU-k
3.33
0.58
0.0
0.0
89.22
18.67
0.17
0.29
0.0
0.0
84.5
26.85
Bad-T
5.33
1.2
1.11
0.69
100.0
0.0
1.17
0.29
0.83
0.76
99.25
0.43
SCRUBS
2.56
0.77
0.0
0.0
99.44
0.96
1.0
1.0
0.0
0.0
100.0
0.0
Table 13: Large forget sets, class unlearning on Lacuna-6 with AII-CNN. SCRUBS and EU-k are
the top-performers in this setting in terms of forgetting with minimal performance degradation. But note that, generally, EU-k doesn't I perform strongly across the board (Figure 3) and in particular per- forms very poorly in selective unlearning (notice the contrast between BU-k's forget error between Figures anl and 4b and between Figures and 8b).
19

---

Under review as a conference paper at ICLR 2023
CIFAR-6
Lacuna-6 Retain error (L)
Test error (L)
Retain error (L)
Forget error (f)
Test error (L)
Forget error (t)
Model
mean
std
mean
std
mean
std
mean
std
mean
std
mean
std
Retrain
35.06
1.02
0.0
0.0
38.22
2.5
6.56
0.35
0.0
0.0
6.11
0.19
Original
32.06
0.69
0.0
0.0
0.0
0.0
5.5
0.73
0.0
0.0
0.0
0.0
Finetune
32.11
1.17
0.0
0.0
0.0
0.0
5.72
0.54
0.0
0.0
0.0
0.0
Fisher
38.06
1.95
14.44
4.83
17.56
7.31
14.44
4.11
7.0
2.31
6.89
1.39
NTK
33.28
0.69
0.0
0.0
27.33
4.26
6.17
0.67
0.0
0.0
5.22
0.38
NegGrad
32,0
1.09
0.0
0.0
0.0
0.0
5.67
0.58
0.0
0.0
0.0
0.0
CF-k
32.0
1.32
0.0
0.0
0.0
0.0
5.72
0.67
0.0
0.0
0.0
0.0
EU-k
34.11
0.96
1.11
0.38
15.78
2.27
5.56
0.51
0.11
0.19
0.89
0.69
Bad-T
37.33
2.75
2.67
0.58
23.56
3.67
9.33
0.33
0.0
0.0
5,56
1.02
SCRUBS
33.89
2.58
0.11
0.19
23.11
2.83
6.22
0.1
0.0
0.0
7.33
0.88
Table 14: Large forget sets, selective unlearning with ResNet. In this setting, SCRUBS and NTK are top-performers on forgetting without sacrificing model performance. While Fisher also achieves forgetting some extent, notice that it substantially degrades performance. On the other hand, NTK in general suffers from poor unlearning performance most other settings considered (e.g. see Figures and 3) and it additionally fails to scale to datascts of the size of CIFAR-IO Lacuna- 10.
CIFAR-6
Lacuna-6 Retain error (L)
Test emor (L)
Retain error (4)
Forget erorr (f) mcan
Test emorr (L)
Forget error (f)
Model
mean
std
std
mean
std
mean
std
Retrain
35.11
4.25
1.11
1.92
2.65
5.22
0.42
0.0
0.0
4.11
Original
31.17
5.83
2.89
5.0
2.78
4.81
4.5
0.44
0.11
0.19
0.0
0.0
Finetune
30.33
3.06
0.0
0.0
0.0
0.0
4.78
1.93
0
0.0
0.0
0.0
Fisher
46.28
5.62
30.78
12.54
31.56
8.77
$2.06
12.86
46.0
11.22
51.33
13.12
31.83
5.45
2.56
4.43
2.85
4.89
0.25
0.0
0.0
3.67
NegGrad
29.94
3.67
0.0
0.11
0.19
4.28
0.54
0.11
0.19
0.78
0.84
CF-k
30.0
3.03
0.11
0.19
0.0
4.5
0.6
0.0
0.0
0.0
FU-k
41.22
0.77
16.22
2.69
18.89
2.01
5.83
0.73
0.11
0.19
0.44
0.38
Bad-T
39.06
2.7
0.22
0.19
35.56
3.91
6.33
1.26
0.0
0.0
3.44
1.95
SCRUBS
30.83
2.91
0.0
0.0
12.44
1.26
4.61
1.07
0.0
0.0
5.89
0.84
Table 15: Large forget sets, selective unlearning with AII-CNN. Fisher the top-performer in terms of forget set error, but introduces large degradation in both test and retain performance. EU-k also performs well on forgetting (though only on one of the 2 datasets) but also degrades performance. SCRUBS strikes good balance between forgetting while retaining performance.
20

---

Under review as a conference paper at ICLR 2023
lest accuracy
lest accuracy
Anctuning
100
Negärad
CF-K
50
EU-K
50
SCRUBS
23
Scale-up
Retain accuracy
Scale-up
10
Retain
ecturacy
Finetuning
Fisher
NegGrad
CF-k
EU-K
SCRUBS
Forget
Forget error
error
(a) Lacuna- 10 class unlcarning.
(b) Lacuna- 10 selective unlcarming.
Figure 6: In the large-scale setting on Lacuna-10, SCRUBS is among the rop-performers: it is slightly slower than CF-k, but CF-k doesn't achieve unlearning (its forget error is 0%) and it is slightly outperformed by NegGrad on the selective setting, but NegGrad is not consistently strong- performing and, in fact performs very poorly class unlearning. Interestingly, selective forgetting is significantly harder for all methods, but especially for EU-k and Fisher which completely fail to unlearn in the selective case. Scale-up is computed the fraction of runtime of the retrain from scratch baseline over the runtime of the given unlearning algorithm. This fraction is multiplied by 10 in this plot for visualization purposes. Each point represents the average across ResNet and All- CNN variants. For reference, the data used to create this plot lives in Tables 6 and (for the left subplot) and Tables 8 |aand (for the right subplot).
A.5 ADDITIONAL FIGURES
In this section, we include additional figures that didn't fit in the main paper.
Specifically, report the large-scale spider plot for Lacuna- 10 in Figure We also separate out the settings of class- and selective- unlearning for large forget set sizes, in Higures 7 and g These two complement Figure in the main paper which aggregates over both class- and selective- unlearning. Separating them out allows to observe different trends. Selective unlearning is substantially harder for all methods when it comes forgetting. Uncharacteristically, NTK and Fisher are among the top performers in that regard when it comes to selective unlearning. Fisher, though, causes significant performance degradation, isn't generally a top performer in most settings we considered, and is very slow to run, making it impractical (Figure 4). NTK also generally suffers from poor unlearning performance most other settings considered (e.g. see Figures [
and it additionally fails to scale to datasets of the size of CIFAR-10 10/ Lacuna-10. 10. Regardless, these results reveal the difference in behaviours across the diverse set of settings we consider, and we hope consistutes a source of inspiration for future investigations.
In addition, we include additional plots of SCRUBS' training dynamics, for different datasets and architectures, in Figures 9 10 and [T] where the conclusions are in agreement with our discussion of this topic in the main paper. Finally, we ablate the effect of cross-entropy in Figure I2 where we find that using cross-entropy offers additional protection to maintaining the model's performance during the unlearning procedure. This sometimes comes at the cost of smaller forget set error, compared the forget set error that would have been achieved if cross-entropy was omitted from the loss.
A.6 ADDITIONAL EVALUATION METRICS: RESOLVING CLASS CONFUSION VIA UNLEARNING
In this section, we explore the performance of SCRUBS and various other baselines on a different explore
set of metrics for measuring forgetting, inspired by Goel et al. (2022). Specifically, scenario where the forget set has confused labels (c.g. for two classes A and B, examples of A are labelled as B, and vice versa). The idea here is that, because mislabelled examples are only present
21

---

Under review as a conference paper at ICLR 2023
Test accuracy
Test accuracy
Finetuning
Anetuning
Richer
100
NTK
NTK
NegGrad
75
NegCrrad
75
CF-K
CF-K
50
SCRUBS
25
SCRUBS
25
10
Torgel
Relain accuracy
Torgel
Relain accuracy
(a) CIFAR-6.
Lacuna-6.
Figure 7: Large forget sets, class unlearning, SCRUBS is the only top-performer across settings. Each point represents the average across 4 settings: {ResNet, All-CNN} (50%, 67%). For refer- ence, the data used to create these plots lives in Tables
12 13
Test accuracy
Test accuracy
Anctuning
Anctuning
Rsher
Rshar
NagGrad
NeqErad
CF-k
EU-k
EU-K
SCRUAS
SCRURS
Forget error
Forget
accuracy
error
acturacy
(a) CIFAR-6.
(b) Lacuna-6.
Figure 8: Large forget sets, selective unlearning. Each point represents the average across ResNet and All-CNN variants. On the left, NegGrad, Finetuning and CF-k perform very similarly, so their lines cover another and, on the right, the same is true for NegGrad, Finetuning, CF-k and EU-k. The detailed results can be seen in Tables 14 and 15 Take-away: SCRUBS remains top per- former in terms of being one the best methods in terms of unlearning, with minimal performance degradation. Interestingly, in terms of forgetting, contrary to the their performance in most other scenarios considered (see e.g. Figures 2 Fisher, though, causes significant performance degradation, isn't generally top performer in most settings we considered, and is very slow to run, making it impractical (Figure 4.. NTK also generally suffers from poor unlearning performance in most other settings considered (e.g. see Figures [ 2 and 3 and it additionally fails to scale to datasets of the size of CIFAR-10 10 / Laauna-10. Regardless, these results reveal the difference in behaviours across the diverse set of settings we consider, and we hope consistutes source of inspiration for future investigations. For reference, the data used to create this plot lives in Tables [4 and 15
observe that NTK and Fisher perform uncharacteristically well here
22

---

Under review as a conference paper at ICLR 2023
retain-set
error
forget-set
validation-set
retain-set
forget-set
velidation-set
epoch
epoch
(a) Performing only max steps (Equation 4
(b) SCRUBS with insufficient max steps.
retain-set
retain-set
forget-set
forget-set
validation-set
validation-set
6STOP
30
10
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
10
epoch
epoch
(c) SCRUBS with too many max steps.
(d) SCRUBS.
Figure 9: Illustration of training dynamics of SCRUBS variants, on CIFAR-5 with AlI-CNN. Per- forming the right interleaving of min-steps and max-steps is important for achieving good balance between high forget error and low retain and validation errors.
100
retain-set
forget-set
80
60
OTTOF
retain-set
20
forget-set
validation-set
epoch
epoch
(a) Performing only max steps (Equation 4
(b) SCRUBS with insufficient max steps.
retain-set
retain-set
forget-set set
forget-set
valldation-set
validation-set
81
CTTor
epoch
epoch
(c) SCRUBS with too many max steps.
(d) SCRUBS.
Figure 10: Illustration of training dynamics of SCRUBS variants, on Lacuna-5 with ResNet. Per- forming the right interleaving of min-steps and max-steps is important for achieving good balance between high forget error and low retain and validation emror..
23

---

Under review as a conference paper at ICLR 2023
retain-set
forget-set
validation-set
retain-set
erfor
forget set
validation
epoch
epoch
(a) Performing only max steps (Equation 4
(b) SCRUBS with insufficient max steps.
retain-set
forget-set
validation-set
retain set
errOr
forget-set
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
10
epoch
epoch
(c) SCRUBS with too many max steps.
(d) SCRUBS.
Figure 11: Illustration of training dynamics of SCRUBS variants, on Lacuna-5 with All-CNN. Per- forming the right interleaving of min-steps and max-sleps is important for achieving good balance between high forget error and low retain and validation errors.
retain-set
forget-set
forget-set
validation-set
validation-set
validation-set
validation-set
OTTOr
error
6
10
epoch
epoch
(a) CIFAR-5 with ResNet.
(b) CIFAR-5 5 with AII-CNN.
100
80
60
forget-set set
CTTOF
forget set
validation-set
validation-set
validation-set
10
validation-se
20
epoch
epoch
(c) Lacuna-5 with ResNet.
(d) Lacuna-5 with AII-CNN.
Figure 12: Effect of adding the cross-entropy loss in Equation Dashed lines omit cross-entropy while solid lines use it. We find that the addition of cross-entropy offers an additional protection maintaining the model's performance during the unlearning procedure. This sometimes comes at the cost of smaller forget set error, compared to the forget set error that would have been achieved cross-entropy was omitted from the loss.
24

---

Under review as a conference paper at ICLR 2023
in the forget set, successful unlearning (removing the influence of the forget set) would lead to model that is not at all confused between classes A and B.
In more detail, the setup follow is: 1) We first mislabel some portion of the training dataset (we mislabelled examples between classes 0 and I of each of CIFAR-5 and Lacuna-5 in our exper- iments), 2) train the 'original model' the (partly mislabelled) training dataset (it has mislabelled examples for classes O and but correct labels for the remaining classes), 3) perform unlearning where the forget set contains all and only the confused examples. Given this, the goal for the un- learning algorithm is to resolve the confusion of the original model.
We consider the following metrics (using terminology consistent with Goel al. 2022)). They presented in order of decreasing generality, and increasing focus on measuring degrees of confusion between the two classes considered.
Error (e.g. test error, retain error, forget error). This counts all mistakes, so anytime that an example of some class is predicted to be in any other class, it will be counted. These are the same metries that we use for the rest of the paper (see Equation 77. For test and retain error, lower is better, whereas for forget error, higher is better.
Interclass Confusion IC-ERR (e.g. IC test error, IC retain error). This counts only mis- takes that involve examples from the confused classes A and B. Specifically. it counts in- stances of any example of class A being predicted be in any other class and analogously for class B. Compared to Error, this metric is more focused towards understanding the re- sult of the introduced confusion, since only considers cases that relate the confused classes. A successful unlearning method would make such errors, so lower is better for each of IC test error and IC retain error.
FGT-ERR (e.g. Fgt test error, Fgt retain error). This metric counts only misclassification between the confused classes and B. Here, mistake of an example of class A (or B) being predicted to be in class other than A or B will not be counted. Only mistakes of an example of class A being predicted to be in class B, and vice versa, are counted. This the most focused metric that explicitly measures the amount of confusion remaining after unlearning. A successful unlearning method would make no such errors, so lower is better for each of Fgt test and Fgt retain.
More formally, Error is the same defined in Equation Let now mathematically define IC- ERR and FGT- ERR. We denote by C"",P the confusion matrix for model parameterized by w on the dataset D, and let D denote the part of the dataset D that belongs to class A. So, for example Dr denotes the part of the retain set D, that belongs to class A, and the entry C..P of the confusion matrix stores the number of times that sample belonging class was (mis)classified belonging to class B in the dataset D by the model parameterized by w. Then, we have:
Σe C...D + +E C...D
IC- IC,
[D + [DB]
where k # A, #,K' B.
So, for example, the 'IC test error' column in our tables is computed via IC-ERR(D,, 0, 1; w). where Dr denotes the test set, and 0 and are the two classes confused in our experiments. Analogously, 'IC retain error' is computed IC-ERR(D,.) 0, w)
Finally:
FGT-ERR(D, A, B:w) = C + B,A
(10)
Tsat is, FOT-ERR only measures the misclassification between the two confused classes A and B. So, for example, the 'Fgt test error' in tables is computed as FGT-ERR(D,. 0, 1; w) and analogously 'Egt retain error' is computed as FGT- (CTT-ER((D,.. 0, 1; w).
The results are shown in Tables 16 and Lacuna-5. We observe that across the board, SCRUBS is a top poo-performer on this metric too (see the captions of the individual tables for more details about performance profile).
18 and for each of ResNet and All-CNN on both CIFAR-5
25

---

Under review as a conference paper at ICLR 2023
model
Test error (
Retain error (4)
Forget error (1)
IC test error (U)
IC retain eroor (1)
Fgt test error (U)
Figt retain error (
std
mcaa
sid
mocn
std
mcan
std
mcaan
std
macan
std
26.67
90.33
1.53
18
0.0
0.0
18.33
Original
41.0
2.09
0.0
0.0
0.0
0.0
56.0
3.04
0,0
0,0
92.0
7.94
0.0
0.0
Finctune
79.33
0.0
NegGrad
36.27
0.42
0.0
0.0
12.67
21.94
47.5
5.27
0.0
0.0
69.0
13.53
0.0
0.0
CF-K
39.6
1.64
0.0
0.0
54,83
2.02
0.0
0.0
85,33
7.02
0.0
0.0
17.47
1.26
2,08
47.0
4.77
8.33
4.73
63.33
9.71
3.67
2.52
Fisher
44.8
2.36
21.33
3.45
32.0
11.53
51.5
7.47
26.33
9.5
79.0
3.61
20.0
7.94
NTK
32.6
2.51
60.33
0.58
37.5
4.0
0.0
0.0
52.0
10.58
0.0
SCRUBS
25.93
3.13
1.08
0.52
96.0
1.73
19.0
3.91
0.0
0.0
19.67
7.51
0.0
0.0
Table 16: Confusion metrics on CIFAR-5 with ResNet. (Confused class 0,1; 50-50 samples). SCRUBS is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first groups of columns).
Test emor (1)
Retain emtor (1)
Forget ertor (1)
IC test emmr (L)
IC retain emorr (1)
Fgt test error (L)
Figt retain emnrr (3)
model
meun
mean
mean
meun
meun
std
meun
mean
Ratmin
24.4
2.75
0.0
0.0
90.67
4.04
19.0
1.32
0.0
0.0
11.33
4.62
0.0
0.0
Original
37.07
4.67
1.5
2,6
5.67
9.81
49.0
4.77
6.0
10.39
80.67
12.58
6.0
10.39
Finetune
34.33
3.35
0.9
0.0
3.0
5.2
43.67
7.29
0.0
0.0
67,33
16.04
0.0
0.0
NegGrad
33.53
4.47
0.0
0.0
13.33
21,36
42:33
11:34
0,0
0.0
62.0
22.65
0.0
0.0
CT.k
0.0
76.33
BU-k
51.6
1.0
27.67
3.5
52.67
6.03
59.5
5.22
38.33
6.66
68.67
15.57
19:67
10.41
Fisher
51,93
2.95
35.17
3.92
31.0
11.53
56.83
8.69
31.67
14.01
78.33
15,53
17.67
11.5
NTK
32.2
2.84
0.75
1.3
43,35
14.15
36.67
4.07
3.0
5.2
54,33
9.02
3.0
5.2
SCRUBS
25.0
3.14
0.0
0.0
93.33
2.52
26.0
4.44
0,0
0.0
18.0
11.14
0.0
0.0
Table 17: Confusion metries on CIFAR-5 with AlI-CNN. (Confused class 0,1; 50-50 samples). SCRUBS is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first 3 groups of columns).
Test mean
(
Retain mean
Forget mean
retain error sid
Fgt mean
Fgt retain
())
model
sid
std
sid
mean
std
mean
std
mean
sid
Retmin
0,2
0.9
99.67
0.58
7.17
2.57
0.0
0.0
0.0
0.0
0.0
0.0
Original
27.07
3.33
1.67
0.88
4.33
1.53
57.5
6.26
6.67
3.51
108.0
14.18
6.67
3.51
Finetone
18.8
4.26
0.0
0.0
14.67
6.03
37.67
11.15
0.0
0.0
63.67
22.01
0.0
0.0
NegGrad
5.25
5.33
CF-k
22.27
4.31
0.08
0.14
10.67
5.03
46,33
10:97
0.33
0.58
81.67
23.01
0.33
0.58
EUL-k
15.27
3.19
0.83
0.38
62.0
12.49
29.33
9.0
2.33
1.53
43.67
16.29
0.33
0.58
Fisher
35.87
3.33
17.75
3.78
27.35
3.79
60.0
5.27
31.0
7.94
109.0
14.53
30.0
7.0
NTK
14.53
5.22
0.0
0.0
51.67
23,18
27.17
11.3
0.0
0.0
43.33
25,32
0.0
0.0
SCRUBS
11.33
3.82
0.58
9.33
0.58
Table 18: Confusion metrics on Lacuna-5 with ResNet. (Confused class 0,1; 50-50 samples). SCRUBS is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first 3 groups of columns). NTK in some cases is able to resolve confusion, but not consistently, and also suffers from higher Test Error.
model
Test error (
Retain mean
(1)
Forget emorr (f) mean
IC test error mean
IC retain emor Fgt error (L) mean
Figt retain mean
(1)
mean
std
sid
std
mean
std
std
std
Retrain
4.2
0.87
0.0
0.0
100.0
0.0
5.33
2.25
0.0
0.0
0.0
0.0
0.0
0.0
Original
25.47
2.32
5.75
5.63
20.33
25.74
56.17
4.93
23.0
22.54
105
8.08
23.0
22.54
Finetune
12.8
2,8
0.0
0.0
23.0
7.94
25.83
7.75
0.0
0.0
39.67
12.74
0.0
0.0
NegGrad
12.8
9.06
3.12
90.0
6.56
20.33
17:04
5.0
6,24
12.67
11.68
2.67
3.79
CF-k
21.27
1.63
0.58
0.8
9.33
0.58
47.0
4.58
2.33
3.21
82.67
10.12
2.33
3.21
BU-k
17.0
8.91
3.92
3.99
92.33
4,93
35.0
18.26
13.0
11.36
3.67
4.73
0.0
0.0
49.6
9.54
57.67
10.79
42,33
11.59
11.68
29.67
NTK
12.87
6.63
2.83
4.91
72,33
12.06
25.5
17.88
11.33
19.63
33.67
24.05
10.0
17.32
SCRUBS
3.87
100.0
4.33
1.26
0.0
0.0
0.0
0.0
Table 19: Confusion metries on Lacuna-5 with AII-CNN. (Confused class 0,1; 50-50 samples). SCRUBS is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see egg. the usual Error metries in the first 3 groups of columns).
26

---

Under review as a conference paper at ICLR 2023
Test error
Forget
error
Retain error
mean
std
mean
std
mean
std
0.77
100.0
0.0
66.67
0.0
100.0
0.0
0.79
100.0
0.0
66.67
0.0
100.0
0.0
0,80
56.2
18.0
100.0
0.0
50.6
27.7
0.81
62.4
7.3
100.0
0.0
60.8
10.0
0.83
61.3
5.0
100.0
0.0
57.0
8.7
0.85
56.8
8.3
100.0
0.0
51.3
14.9
0.89
37.3
4.7
90.7
13.3
14.3
10.7
0,91
38.4
6.7
86.0
23.1
19.6
16.7
0.93
34.2
2.3
63.2
16.6
1.4
1.3
0.95
31.1
2.2
89.2
1.8
1.2
1.5
0.97
35.0
1.5
20.9
7.5
0.0
0.0
0.99
36.8
2.2
8.1
1.5
0.0
0.0
Table NegGrad trade-off between forgetting and retaining. We observe that smaller 33 tends to forget more and larger / tends to overfit more to the retain-set based on Equation [ The results correspond to the experiment Table for the experiment of large forget set, class unlearning on CIFAR-6 (forget classes, i.e. 50%), with ResNet.
A.7 TRADE-OFEE THAT ARISE WHEN TUNING MODELS
In this section, we discuss trade-offs present when tuning models. We begin with case study the NegGrad approach, which is the second strongest model overall, after SCRUBS. This will allow to better understand its performance profile under different settings the hyperparameters. As shown in Equation 8 this model is trained by finetuning on both the retain and forget set, but reversing the gradient for the latter. There is a trade-off parameter 3 that controls the relative importance each of those two terms. As can be seen from Table 20 this induces trade-off: smaller values of 3 tend to forget more successfully (higher forget error), at the expense of poor performance (higher retain and test error), whereas lower values of /3 lead to the opposite effect.
In our main experiments, we have carefully tuned this model in the same way that we do for SCRUBS: achieve high error on the forget set while sacrificing performance.
Next, to better understand trade-offs for both NegGrad and SCRUBS, we visualize the Pareto Front of each of these methods, in Figure 13 Our approach for this is the following. First, we vary hyperparameters for NegGrad (as seen in Table 20) and for SCRUBS (as seen in Table 22 see the next section on Sensitivity Analysis for SCRUBS for more details). Then, construct the Pareto Front for each of these two methods as follows: from those tables, we include only the rows that represent (retain error, test error, forget error) points such that no other such point 'strictly dominates' it, i.c. other point performs strictly better than it on all of those 3 metrics.
We draw the following conclusions from Figure
1. SCRUBS' Pareto front contains many more points that are more effective forgetting (higher forget error) compared to NegGrad's (visually: there are a lot of orange points the right of blue points, both subplots). Interestingly, NegGrad can achieve high forgetting (as seen by the blue points in the top right corner), but this comes at the cost of retain error and test error for NegGrad (large y-values). On the other hand, SCRUBS' Pareto front contains points that have the best possible forget error, without sacrificing performance (bottom right corner of the plot).
2. Both NegGrad and SCRUBS have some points on their Pareto Front that our model selection procedure (that balances good forgetting with good performance on the re- tain set) would never choose as the model that we use for evaluation (e.g. to produce results for our tables). In the case of SCRUBS, these points are the orange ones on the top right corner. These are included in the Pareto Front, because they have the best possi- ble forget eπror, are 'tied' with the point that does (so aren't 't strictly dominated by any other points), but they have poor retain error. Therefore, during model selection, we would instead choose the points the bottom right corner, which have equally high forget
27

---

Under review as a conference paper at ICLR 2023
refain emror va forget-ene error
lest error vs forget error
NegGirdd SCRUBS
NegGrad SCRUBS
60
60
retain-erorr
40
Nss--cror
-50
0-
70
100
90
100
Girget-arror
Eoges--croo
(a) retain vs forget errors.
(b) test forget errors.
Figure 13: A visual comparison of the Pareto Front of NegGrad and SCRUBS. These points are obtained by Tables 20 and 22 which explore the performance of each of NegGrad and SCRUBS with respect to different hy hyperparameter settings. From those tables, we include only the rows that represent points of the Pareto Front. More concretely, a (retain error, test error, forget error) point on the Pareto Front if no other such point 'strictly dominates' it, i.e. no other point performs strictly better than it on all of the metrics.
ror but without sacrificing performance. In the case of NegGrad, some points of the Pareto Front fail to forget (bottom left corner, very low forget error), and similarly we would never choose those during model selection, since we can find other points in the Pareto front that have similar performance on the retain set but with higher forget error.
3. Overall: SCRUBS' Pareto Front has significantly more points than NegGrad's Pareto Front on the most desirable 'corner' of the plots: the bottom right corner, which represents best forget error well as best retain/test error.
A.8 SENSITIVITY ANALYSIS
Next, we analyze the sensitivity of SCRUBS to different hyperparameters that impact optimization.
As can be seen from Table 2I setting both a and y to (which leads to only applying the max- step, push the student away from the teacher forget set examples) Ieads to high forget error, as expected, but significantly hurts performance. This is expected, since this variant has no incentive to retain useful information from the original model. Further, best value that we found, which we used in our main results), can protect from this performance degradation, even if setting only to O, at least for the values considered in this table. Our general take-away, however, that our results aren't t overly sensitive to these hyperparameters. As long as they aren't set to O, there is wide range of them that yields similarly well-performing models.
observe that setting y (the
On the other hand, setting only y to 0 leads a worse performance than
A.9 DETAILED COMPARISON AGAINST THE BAD-TEACHER APPROACH
In this section, we perform a more detailed empirical comparison against an idea presented in recent concurrent work (preprint paper (Chundawat al. (022)). Their approach also utilizes seacher-student framework, albeit in a different way. As in SCRUBS, the teacher is the original model trained on all available training data, and the student initialized from the teacher and sub- sequently updated in a way that induces forgetting. However, the way in which this is achieved in their work is different than SCRUBS: order to achieve forgetting, the student is encouraged to match the behaviour of randomly iistoomll-iiiiaiizeedd model (a 'bad teacher') ) for the examples of the forget set (instead of pushing it away from the teacher for those examples as we do in SCRUBS).
Given that this idea is related to SCRUBS, we devote this section to detailed discussion of the empirical comparison between these two. For this, we have re-implemented their approach (which denote 'Bad-T', for 'bad teacher'), ) since the above preprint provides no available code for this model. We have tuned the learning rate and number of updates for 'Bad-T" thoroughly and perform model selection for this model in the same way that do for SCRUBS.
28

---

Under review as a conference paper at ICLR 2023
Test error
Forget error
Retain
mean
std
mean
std
mean
std
0
0.1
44.78
4.76
94.78
1.55
23.11
9.45
0.1
31.33
2.68
96.89
0.57
0.00
0.00
0.1
31.89
2.86
97.67
0.27
0.00
0.00
3
0.1
30.89
2.04
98.78
0.63
0.00
0.00
0.1
30.78
2.18
99.33
0.54
0.00
0.00
5
0.1
31.22
2.04
99.33
0.54
0.00
0.00
6
0.1
30.89
2.28
99.67
0.47
0.00
0.00
7
0.1
31.22
2.53
99.67
0.47
0.00
0.00
31.33
3.30
100.00
0.00
0.00
0.00
3
0.1
31.00
2.18
98.78
0.63
0.00
0.00
30.22
1.23
91.44
2.69
0.00
0.00
3
30.44
2.22
84.67
3.27
0.00
0.00
31.33
1.78
81.44
4.24
0.00
0.00
3
30.67
2.05
79.11
4.13
0.00
0.00
31.00
1.78
76.89
4.61
0.00
0.00
3
6
31.67
2.23
75.22
4.99
0.00
0.00
31.56
1.64
74.22
5.19
0.00
0.00
0
64.78
2.67
100.00
0.00
65.11
2.20
Table 21: Sensitivity analysis for SCRUBS, for the experiment of large forget set, class unlearning on CIFAR-6 (forget classes, i.e. 50%), with ResNet. Here we have analyzed the sensitivity of SCRUBS optimization
(the hyperparameter that controls the influence of the that mini- mizes the distance from the teacher on retain examples, in Eq. 6) and (the hyperparameter that controls the influence of the cross-entropy term, in Eq. (6 on SCRUB's performance. The number of max-steps is set to 2 here, which is the best value discovered for this experiment.
Test error
Forget error
Retain error
max-steps
?
mean
std
mean
std
mean
std
0
0
56.67
3.21
30.56
7.05
43.78
6.92
0
0.1
34.11
1.81
82.56
6.31
8.22
0.57
I
1
0.1
32.89
2.99
95.89
2.57
0.00
0.00
2
0.1
32.67
2.62
97.89
1.34
0.00
0.00
3
0.1
32.78
2.51
98.78
0.57
0.00
0.00
0
0
64.78
2.67
100.00
0.00
65.11
2.20
0
0.1
44.78
4.76
94.78
1.55
23.11
9.45
2
1
0.1
31.33
2.68
96.89
0.57
0.00
0.00
2
0.1
31.89
2.86
97.67
0.27
0.00
0.00
3
0.1
30.89
2.04
98.78
0.63
0.00
0.00
0
66.67
0.00
100.00
0.00
66.67
0.00
0
0.1
63.78
1.81
99.89
0.16
61.78
4.59
3
0.1
44.33
4.91
100.00
0.00
28.22
8.87
2
0.1
37.11
2.47
100.00
0.00
15.11
3.00
3
0.1
33.78
2.11
100.00
0.00
4.89
1.97
Table 22: Additional sensitivity analysis for SCRUBS. This is an expansion of Table 21 that consid- ers values for max-step aside from the best one.
29

---

Under review as a conference paper at ICLR 2023
lest accuracy
accuracy
Anctuning
Anctuning
100
NagGrad
NegErad
Bad-T
SCRUAS
SCRURS
5d
25
Forget error
Retain accuracy
Forget
Retain acturaty
error
(a) CIFAR-5.
(b) Lacuna-5.
Figure 14: In the small scale setting, selective unlearning, SCRUBS is the only top-performer across settings. Bad-T performs poorly compared to other models. Each point represents the average across ResNet and AlI-CNN.
Counts
Counts
2
Original
Roneuning
Fisher
NTK
NogGrad
CF
Bad-T
scrums
Original
Riistnning
Fisher
NTK
NopGred
CF-
BU-K
Bad-T
scrums
(a) Small scale settings).
Large forget sets (12 settings).
Figure 15: The number of times that each method was (one of the) top performer(s) in terms of the forget error, across all settings (architectures, datasets, selective class forgetting, forget set size, etc.). We consider model a top-performer if its 95% confidence interval overlaps with that of the best mean. Small scale counts are over (ResNet, All-CNN) x (CIFAR, Lacuna] (4 total), and large forget set additionally (class-50%, class-67%, sclective-50%) (12 total). SCRUBS is the most consistent in terms its ability to unlearn.
The results for Bad-T are shown in Tables 4 which is an adaptation of Figure that includes Bad and excludes some other moorly-performigg baselines reduce clutter and aid more clearly comparing Bad-T to other methods. We observe the scenario, and that it fails to exhibit particular, we observe that
12 and 15 and and Figure 14
that the performance of Bad-T largely varies depending consistency in being top-performer in contrast to SCRUBS. performs well in the scenarios with large forget sets (often in tie with SCRUBS) but performs very poorly in the small scale regime (as can be seen clearly in Figure| 14 for example).
To best understand its performance profile compared to SCRUBS, Figures 15 and 16 counts the times that a model
top-performer. We consider a model to the top-pertormer with respect to all other models considered, or if its
metric if it has the best mean (over trials) performance confidence intervals overlap with the model's that has the best mean. These figures show (i) that SCRUBS is significantly more consistent in terms of successfully forgetting across wide range of scenarios, and (ii) in addition, SCRUBS more consistent terms of not hurting the retain and test performance during unlearning.
30

---

Under review as a conference paper at ICLR 2023
(a) Small scale, retain (4 settings).
(b) Small scale, test (4 settings).
(c) Large forget sets, retain (12 settings).
(d) Large forget sets,
test (12 settings).
Figure 16: Focusing on the comparison between SCRUBS and Bad-T, we additionally show the number of times that each method was (one of the) top performer(s) in terms of the retain and test errors, across all settings (architectures, datasets, selective vs class forgetting, forget set size, etc.). We consider model top-performer if its 95% confidence interval overlaps with that of the model that has the best mean. Small scale counts are over (ResNet, AII-CNN) x (CIFAR, Lacuna} (4 total), and large forget set additionally (class-50% class-67% selective-50%} (12 total). SCRUBS is more consistent compared to Bad-T in its ability to not hurt performance during unlearning.
31


Reviews:
[{"review_id": "eql0YEV4h_8", "reviewer": "Reviewer_8kmh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of machine unlearning. The paper proposes a method that trains a student model starting from the original model, and updates the student model based on the loss so that the student's loss on the unlearned samples is maximized while the loss on the retained samples is minimized. Different from previous machine unlearning work, the paper focuses on the efficiency and performance of deep neural networks, and shows improved results in terms of forgetting errors, retaining accuracy, and test accuracy.", "summary_of_the_review": "Given the limited novelty and some questions about the current method and experiments, I think modifications and clarification need to be made to accept the current paper.", "strength_and_weaknesses": "Strengths:\n1. The proposed method is relatively efficient and can apply to a larger set of unlearning samples compared to previous methods.\n\n2. The proposed method is simple and easy to implement.\n\n3. The experimental result shows that the proposed method has better consistency in giving the best performance among various tasks in terms of forgetting errors, retaining accuracy, and test accuracy.\n\n\nWeaknesses:\n1. The proposed objective is a minimax objective, and training for such an objective could be potentially very tricky and it may require a lot of tuning. The paper mentions that it follows certain techniques from training GAN models, but it still seems careful tuning is required to optimize the objective effectively. Also, it is hard to know whether the optimization result is good or bad without any theoretical understanding of the proposed objective.\n\n2. The novelty of the proposed method is also limited. The proposed objective is similar to the consistency and contrastive loss used in self-supervised learning and the optimization procedure follows techniques of training GAN models.\n\n3. The proposed method has better consistency over a variety of tasks. However, in some tasks, the proposed methods do not give the best performance and more investigation seems required to understand what kind of tasks the proposed method does better and why.\n\n4. The proposed method does not have theoretical guarantees of any form. I.e., it is hard to know whether the unlearning set of samples has been truly forgotten. Showing theoretical results on deep models is very hard, and showing theoretical properties for simpler models could still be insightful.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bisUCH78VnA", "reviewer": "Reviewer_eMqy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a simple scheme for machine unlearning: learning to minimizing DL divergence with original model on the retain set (can also minimize test error), while maximizing DL divergence with original model on the forget set. It is shown to be effective across many task setups versus many baselines.", "summary_of_the_review": "The method is simple and effective, but authors might want to better quantify the \"scalable\" part. ", "strength_and_weaknesses": "The method seems simple yet consistently effective, and there are many analyses and insights discussed in the experiment parts. It also points out problems of previous theoretically-inspired methods.\n\nI'm not familiar with this domain, but seems the scales of experiments are modest, and even \"large scale\" setup is more about portion of forgetting instead of scale of model/data (as mentioned in intro)? The title, abstract, and intro might need to better characterize what is \"scalable\" in the method. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JEubHD580wr", "reviewer": "Reviewer_VJBS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The focus of this paper is deep machine unlearning. This problem refers to removing the influence of a subset of data from the weights of a trained deep model. For instance, an application of this kind of method is to remove user data to guarantee their \"right to be forgotten.\" Other applications include removing noisy labels, outliers, harmful biases, and out-of-date instances. Given the widespread use of DNNs, this is a significant problem.\n\nSince previous approaches to deep machine unlearning make unrealistic assumptions related to DNNs, such as assuming the convexity of loss functions, there is a need for scalable solutions. The student-teacher methodology inspires the proposed solution. Here, the student ignores to-forget instances and considers all others.\n\nA hypothesis of this paper is that by leveraging the student-teacher-inspired methodology are able to scale and simultaneously provide minimal performance loss. The authors designed a new method based on that methodology and conducted empirical experiments to validate their hypothesis. All research questions are clearly stated.\n\nSpecific comments:\n\n- I have difficulty understanding why, in practice, Eq. 4 does not work. I suggest the authors include some intuition on that.\n- In Sec. 3, Parag. 5, \"performing this maximization step\" ->  \"performing this minimization step\" (according to Eq. 4).", "summary_of_the_review": "This paper presents a scalable solution to the deep unlearning method by exploiting the student-teacher methodology. The results validate their hypothesis. Additionally, there are missing important discussions and missing original code implementation.", "strength_and_weaknesses": "Strengths\n\n- This paper is easy to follow and well-motivated.\n- In the context of new legislation and other relevant use cases, this paper has a clear motivation.\n- The proposed method seems to be practical.\n\nWeaknesses\n\n- It is a relevant point to include the source code and enable others to reproduce the experiments.\n- The current method is valid to other architectures, such as Vision Transformers? Or a new deep unlearning method needs to be envisioned? This paper deserves this discussion.\n- Given the current metrics used, I did not have a clear understanding of issues related to, for instance, class imbalance.\n- The evaluation metrics should be clearly presented, such as using equations.\n- Besides the points highlighted as future work, the authors should discuss the limitations of their method. Given the space restriction, some parts of related work could go to the Appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9VxObSf3vZO", "reviewer": "Reviewer_GpWt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a machine unlearning approach, Brainy student, to forget a subset of training data even at larger scales. The solution is inspired by a recent popular student-teacher architecture where a student network disobeys the teacher network to avoid inheriting information about the forget set while obeying the teacher network for the retain set. The experimental investigation reveals that the proposed method significantly improves upon previous methods in achieving unlearning in a wide range of scenarios.", "summary_of_the_review": "Overall, the authors propose a new framework, brainy student, a student-teacher network-based model for the purpose of machine unlearning at larger scales. However, the paper lacks model training details, lacks a discussion on why the proposed model is superior to the previously existing methods, and the robustness of the model is still to be validated.", "strength_and_weaknesses": "Strengths:\n* The idea of machine unlearning for larger scales, by simultaneously encouraging the student to ‘stay close to the teacher on retaining examples, while encouraging it to ‘move away from the teacher on forgetting examples, is an interesting contribution to the different domains of AI. \n* The problem formulation and its components are well explained with mathematical equations.\n\nWeaknesses:\n* This proposed approach of using student-teacher architecture for machine unlearning is not entirely new. There is a plethora of papers in machine unlearning that uses a student-teacher approach to train the models. \n* The proposed framework is similar to the recent work “Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher“, Chundawat et al. 2022. However, the authors have not cited this work, and experimental analysis is limited to previous works.\n* Figure 3 requires standard error bars for better comparison across all the methods. Also, the authors could perform a statistical significance test to compare the results. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject