SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Meghad Kurmanji*
Peter Triantafillou
Jamie Hayes
Eleni Triantafillou
Deep machine unlearning is the problem of 'removing' from a trained neural network a subset of its training set. This problem is very timely and has many applications, including the key tasks of removing biases (RB), resolving confusion (RC) (caused by mislabelled data in trained models), as well as allowing users to exercise their 'right to be forgotten' to protect User Privacy (UP). This paper is the first, to our knowledge, to study unlearning for different applications (RB, RC, UP), with the view that each has its own desiderata, definitions for 'forgetting' and associated metrics for forget quality. For UP, we propose a novel adaptation of a strong Membership Inference Attack for unlearning. We also propose SCRUB, a novel unlearning algorithm, which is the only method that is consistently a top performer for forget quality across the different application-dependent metrics for RB, RC, and UP. At the same time, SCRUB is also consistently a top performer on metrics that measure model utility (i.e. accuracy on retained data and generalization), and is more efficient than previous work. The above are substantiated through a comprehensive empirical evaluation against previous state-of-the-art.
Can we make a deep learning model 'forget' a subset of its training data? Aside from being scientifically interesting, achieving this goal of deep machine unlearning is increasingly important and relevant from a practical perspective and this research direction is enjoying significant attention recently [Triantafillou et al., 2023]. Regulations such as EU's General Data Protection Regulation [Mantelero, 2013] stipulate that individuals can request to have their data 'deleted' (the 'right to be forgotten'). Nowadays, given the ubiquity of deep learning systems in a wide range of applications, including computer vision, natural language processing, speech recognition and healthcare, allowing individuals to exercise this right necessitates new technology. Unlearning aims to fill this need.
However, unlearning has several additional important applications where privacy with respect to a subset of individual training examples isn't necessarily a consideration: removing out-of-date examples, outliers, poisoned samples [Jagielski et al., 2018], noisy labels [Northcutt et al., 2021], or data that may carry harmful biases [Fabbrizzi et al., 2022]. In fact, we are the first to argue that the definition of 'forgetting' is application-dependent, and unlearning comes with different desiderata and priorities in different scenarios. For instance, for a privacy application, 'forgetting' the data of users who requested deletion is the primary objective and we may be willing to sacrifice model performance to some degree, in exchange for reliably defending Membership Inference Attacks (MIAs). In other cases, for instance, when deleting outdated data to keep a model current, maintaining the model's performance throughout such 'cleanup' operations may be the primary consideration, and guaranteeing that no trace of the old data is left behind may be irrelevant. While previous work does not establish such distinctions, we study three scenarios and propose a method that is the most consistent top-performer across applications and metrics.
*Correspondence to Meghdad.Kurmanji@warwick.ac.uk.
Unfortunately, removing the influence of a subset of the training set from the weights of a trained model is hard, since deep models memorize information about specific instances [Zhang et al., 2020, 2021, Arpit et al., 2017] and their highly non-convex nature makes it difficult to trace the effect of each example on the model's weights. Of course, we can apply the naive solution of retraining the model from scratch without the cohort of data to be forgotten. This procedure indeed guarantees that the weights of the resulting model aren't influenced by the instances to forget (it performs 'exact unlearning'), but the obvious drawback is computational inefficiency: retraining a deep learning model to accommodate each new forgetting request isn't viable in practice. To mitigate that cost, recent work turned to approximate unlearning [Izzo et al., 2021, Golatkar et al., 2020a,b] where the goal is to modify the weights of the trained model to approximate those that would have been obtained by exact unlearning. That is, they strive to achieve 'indistinguishability' between the approximate and exact solutions, typically accompanied by theoretical guarantees for the quality of that approximation. This goal is also mirrored in the metrics used by these works: the model's error on the deleted data is desired to be just as high as the error of the 'exact' retrained model (which truly never saw that data). The argument is that an error higher than that reference point may cause susceptibility to membership inference attacks: a noticeably poor performance on an example can reveal that it was unlearned.
In this work, we show empirically that such approximate unlearning methods often perform poorly in practice (in one or more of RB, RC and UP). We hypothesize that this is due to potential violations of assumptions made by these methods, e.g. that the 'optimal' solution of unlearning is close in weight space to the original model [Golatkar et al., 2020a,b] (due to assumptions of stability of SGD and 'small enough' forget sets). Further, we find these previous methods scale poorly to the size of the training set and the model, often making them impractical. To address these issues, we propose a new unlearning model, SCalable Remembering and Unlearning unBound (SCRUB) that departs from that methodology. SCRUB bears a novel teacher-student formulation, where the student model selectively disobeys an all-knowing teacher, to inherit from it only knowledge that does not pertain to the data to be deleted. This method is 'unbound' from limiting assumptions (e.g., to facilitate formal guarantees), poor scalability limits, and is not tailored to a single unlearning application. We find that SCRUB can yield a very high error on the deleted data, which is desired in some scenarios but may cause susceptibility to membership inference attacks. To mitigate this when relevant, we extend SCRUB with a novel 'rewinding' procedure that pinpoints the 'checkpoint' of the unlearning process to use such that the error on the deleted data approximates the reference point of being 'just high enough'.
We also propose the first, to our knowledge, adaptation of the state-of-the-art LiRA MIA [Carlini et al., 2022] to the framework of unlearning, as a metric for UP. Overall, we perform a comprehensive evaluation using different baselines, datasets, architectures and three application scenarios. We find that SCRUB is by far the most consistent top performer out of all state-of-the-art methods considered. SCRUB is able to ensure high forget quality, satisfying different forget-quality metrics for different applications while also being a top performer with respect to model utility (accuracy on retained data, and generalization). Our findings highlight the need to consider a broad spectrum of applications when evaluating unlearning algorithms, to better understand their properties and trade-offs.
2 Problem Definition for Unlearning Unbound
Notation and preliminaries Let  $\mathcal{D} = \{x_i, y_i\}_{i=1}^N$  denote a training dataset containing  $N$  examples, where the  $i$ 'th example is represented by a vector of input features  $x_i$  and a corresponding class label  $y_i$ . Let  $f(\cdot; w)$  denote a function parameterized by trainable weights  $w$ . In this work, we study the case where  $f$  is represented by a deep neural network, trained in a supervised manner via empirical risk minimization. Specifically, we define the loss function as  $\mathcal{L}(w) = \frac{1}{N} \sum_{i=1}^{N} l(f(x_i; w), y_i)$  where  $l$  denotes the cross entropy loss.
Deep Machine Unlearning We now formalize the problem of deep machine unlearning. We assume that we are given a model  $f(\cdot; w^o)$  that has been trained on  $\mathcal{D}$ , where  $f$  denotes a neural network and  $w^o$  its trained parameters, obtained by minimizing the above loss function. We will refer to this as the 'original model' (i.e. before any unlearning intervention is performed). We are also given a 'forget set'  $\mathcal{D}_f = \{x_i, y_i\}_{i=1}^{N_f} \subset \mathcal{D}$ , comprised of  $N_f$  examples from  $\mathcal{D}$  and a 'retain set'  $\mathcal{D}_r$  of  $N_r$  training examples that the model is still allowed to retain information for. For simplicity, we consider the standard scenario where  $\mathcal{D}_r$  is the complement of  $\mathcal{D}_f$  or, in other words,  $\mathcal{D}_f \cup \mathcal{D}_r = \mathcal{D}$ .
Given  $f(\cdot; w^o)$ ,  $\mathcal{D}_f$  and  $\mathcal{D}_r$ , the goal of deep machine unlearning is to produce a new set of weights  $w^u$  such that the 'unlearned model'  $f(\cdot; w^u)$  has 'forgotten'  $D_f$  without hurting the 'utility' of the original model (its performance on the retained data  $D_r$  and generalization to data outside of  $D$ ). We argue that defining (and measuring) 'forgetting' is application-dependent. Our work is the first, to the best of our knowledge, to perform a thorough empirical evaluation of unlearning algorithms on three different application scenarios, each with its own definition of forgetting and associated metrics.
Defining and measuring 'forgetting' We consider three different application scenarios in this work, each with their own definition of forgetting and associated metrics. 1) Removing biases. We want maximal error on  $\mathcal{D}_f$ : the model should never predict the assigned labels of forget examples, since they reflect unintended behaviour. 2) Resolving confusion. A model may be 'confused' due to incorrect labels in its dataset. A successful unlearning algorithm resolves this if mislabeled examples are placed in the forget set (see metrics in Section 5). 3) User privacy. Success from a privacy perspective is associated with a forget error only as high as that of retraining-from-scratch; a higher forget error could lead to vulnerability to Membership Inference Attacks (MIAs). In all of the above cases, in addition to achieving forgetting, we seek unlearning algorithms that don't damage model utility, i.e. accuracy on retained data and generalization ability, and that unlearning is efficient enough to be preferable over the simple and expensive solution of retraining the model.
Fundamental trade-offs A top-performing unlearning algorithm must simultaneously achieve three feats: (i) obtain high 'forget quality', (ii) ensure continued high accuracy on retained and unseen data, and (iii) efficiency / scalability. In practice, there are fundamental trade-offs in two different dimensions. The first dimension involves trade-offs between (i), (ii) and (iii), for example achieving good 'forget quality' at the expense of efficiency. Retraining from scratch is such an example, but, as we will show later, some methods suffer from inefficiencies even worse than retraining. Another common issue we will observe is achieving good 'forget quality' at the expense of utility. The second dimension involves trade-offs within the context of achieving good 'forget quality' across applications, since the metric for this quality is application-specific, and an algorithm may perform strongly on one such metric but poorly on others. SCRUB is designed to handle these trade-offs and is shown empirically to consistently outperform previous methods across the board.
3 SCalable Remembering and Unlearning unBound (SCRUB)
We present SCRUB, a new unlearning algorithm based on a novel casting of the unlearning problem into a teacher-student framework. We design SCRUB to meet the desiderata of unlearning: efficiently 'forgetting' without hurting model utility. Because we argue that the definition of forgetting is application-dependent, we propose a recipe that works across applications: We first design SCRUB to strive for maximal forget error, which is desirable in some scenarios (e.g. RB and RC) but not in others (e.g. UP). To address the latter case, we complement SCRUB with a novel 'rewinding' procedure that can reduce the forget set error appropriately when required. SCRUB is 'unbound' from limiting assumptions (e.g., to facilitate formal guarantees), poor scalability limits and is by far the most consistent method in forgetting across applications while preserving model utility.
3.1 The SCRUB
We consider the original model  $w^{o}$  as the 'teacher' and we formulate our goal as training a 'student'  $w^{u}$  that selectively obeys that teacher. Intuitively, our goal for  $w^{u}$  is twofold: forget  $\mathcal{D}_f$  while still remembering  $\mathcal{D}_r$ . To that effect,  $w^{o}$  should be obeyed when teaching about  $\mathcal{D}_r$ , but disobeyed when teaching about  $\mathcal{D}_f$ . Our code is available for reproducibility<sup>2</sup>.
En route to deriving our training objective, let us first define:
$$
d (x; w ^ {u}) = D _ {\mathrm {K L}} \left(p \left(f \left(x; w ^ {o}\right)\right) \| p \left(f \left(x; w ^ {u}\right)\right)\right)
$$
In words,  $d(x; w^u)$  is the KL-divergence between the student and teacher output distributions (softmax probabilities) for the example  $x$ . We make the dependence of  $d$  on  $w^u$  explicit, since we will optimize the student weights  $w^u$  while keeping the teacher weights frozen  $w^o$ , treating them as a constant.
$^{2}$ https://github.com/Meghdad92/SCRUB
Concretely, we begin by initializing the student  $w^u$  to the weights of the teacher  $w^o$ . Since the teacher weights  $w^o$  were trained on all of  $\mathcal{D}$ , the teacher performs well on both constituents of  $\mathcal{D}$ , namely both of  $\mathcal{D}_r$  and  $\mathcal{D}_f$ . Therefore, the student has good performance on  $\mathcal{D}_r$  at initialization, already fulfilling one of our desiderata. Can we then start from this solution and modify it to unlearn  $\mathcal{D}_f$ ? In principle, one could do this by optimizing:
$$
\min  _ {w ^ {u}} - \frac {1}{N _ {f}} \sum_ {x _ {f} \in \mathcal {D} _ {f}} d \left(x _ {f}; w ^ {u}\right) \tag {1}
$$
However, in practice, when performing this maximization of the distance between the student and teacher on the forget set, we noticed that, while indeed the performance on  $\mathcal{D}_f$  degrades, as desired, it unfortunately also results in degrading performance on  $\mathcal{D}_r$ . To amend that, we propose to simultaneously encourage the student to 'stay close' to the teacher on retain examples, while encouraging it to 'move away' from the teacher on forget examples, adding a contrastive flavour. Formally, the optimization objective becomes:
$$
\min  _ {w ^ {u}} \frac {1}{N _ {r}} \sum_ {x _ {r} \in \mathcal {D} _ {r}} d \left(x _ {r}; w ^ {u}\right) - \frac {1}{N _ {f}} \sum_ {x _ {f} \in \mathcal {D} _ {f}} d \left(x _ {f}; w ^ {u}\right) \tag {2}
$$
Furthermore, SCRUB also simultaneously optimizes for the task loss on the retain set, to further strengthen the incentive to perform well there. Our final training objective is then the following:
$$
\min  _ {w ^ {u}} \frac {\alpha}{N _ {r}} \sum_ {x _ {r} \in \mathcal {D} _ {r}} d \left(x _ {r}; w ^ {u}\right) + \frac {\gamma}{N _ {r}} \sum_ {\left(x _ {r}, y _ {r}\right) \in \mathcal {D} _ {r}} l \left(f \left(x _ {r}; w ^ {u}\right), y _ {r}\right) - \frac {1}{N _ {f}} \sum_ {x _ {f} \in \mathcal {D} _ {f}} d \left(x _ {f}; w ^ {u}\right) \tag {3}
$$
where  $l$  stands for the cross-entropy loss and the  $\alpha$  and  $\gamma$  are scalars that we treat as hyperparameters.
In practice, we found that optimizing the objective in Equation 3 is challenging, due to oscillations in the loss. Intuitively, this is due to trying to simultaneously satisfy two objectives, which may interfere with each other, namely moving close to the teacher on some data points while moving away from it on others. To address this, SCRUB provides a practical recipe for optimization, reminiscent of 'tricks' used in other min-max-like problems like Generative Adversarial Networks [Goodfellow et al., 2014] where the discriminator is trained for several steps before each update to the generator.
Specifically, SCRUB iterates between performing an epoch of updates on the forget set (the max-step) followed by an epoch of updates on the retain set (the min-step), in an alternating fashion. Guarding against hurting retain performance due to this alteration of min-steps and max-steps, SCRUB also performs a sequence of additional min-steps at the end of the sequence to restore the retain performance in the event that it was harmed. SCRUB training stops when the forget error has increased without harming the retain set error. We find in practice this point can be reached with a small number of epochs. We provide pseudocode, training plots and ablations in the Appendix.
3.2 SCRUB and Rewind (SCRUB+R)
By construction, SCRUB encourages obtaining a high error on the forget set. This is desired for some application scenarios. However, an uncharacteristically high error on deleted examples can make them identifiable, causing vulnerability to membership inference attacks. SCRUB+R addresses this.
To this end, we are after a procedure for selecting the checkpoint of SCRUB where the forget set error is 'just high enough'. One could obtain a reference point for that by retraining from scratch without the forget set and recording the error of that model on the forget set. However, that would defeat the purpose of unlearning, as we want to avoid that computation. Therefore, we propose a different way of establishing a reference point. First, we create a validation set of the same distribution as the forget set. For instance, if the forget set has only examples of class 0, we keep only examples of class 0 in the validation set too. Next, we train SCRUB as usual, storing a model checkpoint every epoch. At the end of its training, where the forget error is typically the highest, we measure the error on the constructed validation set. This will serve as the reference point for the desired forget set error. Finally, we 'rewind' to the checkpoint where the forget error is closest to that reference point.
(a) CIFAR-10 class unlearning.
(b) CIFAR-10 selective.
Figure 1: RB results: SCRUB is the only consistent top-performer in terms of forgetting and preserving utility. It is also highly efficient: second only to Bad-T which, however, fails at forgetting and damages utility. In all subfigures, each point represents the average across ResNet and All-CNN variants. For large-scale results, we also compute the scale-up factor: the fraction of the runtime of retraining from scratch over the runtime of the given unlearning algorithm (multiplied by 5 here, for visualization purposes). We find that selective unlearning is harder for all methods, especially for EU-k and Fisher (see Figure 1a vs 1b). CF-k and Finetuning perform similarly (they perform poorly) in all cases, so their lines occlude one another. Complete tables are in the Appendix.
(c) CIFAR-5 selective.
The intuition is that the last step of unlearning approximates 'maximally forgetting'. Consequently, the error on the identically-distributed validation set approximates the error of a model that never learned about that distribution from the forget set: any correct predictions on the held-out examples are due only to the generalization power of the model that was trained only on the retain set. Therefore, we choose that validation set error to serve as the reference point for how high we would like the forget set error to be for UP applications. We show empirically that this rewinding procedure greatly increases the defense against membership inference attacks, making SCRUB a strong performer for applications where privacy is a consideration.
Unlearning definitions Despite prior work on unlearning, this research problem is still in its infancy and we lack a well-established formal definition. The term 'unlearning' was coined in Cao and Yang [2015], where, in the context of certain structured problems, unlearning is defined to be successful if deleting an example yields the same outputs on a dataset as if that example had never been inserted. Ginart et al. [2019] introduce a more relaxed probabilistic definition for machine unlearning inspired by Differential Privacy [Dwork et al., 2014] that requires the unlearned model to yield outputs that are similar to those that would have been produced by a model retrained from scratch without the forget set. This led to the development of 'approximate learning' methods that aren't identical to retrain-from-scratch ('exact unlearning') but 'similar' [Guo et al., 2019, Golatkar et al., 2020a]. Sekhari et al. [2021] and Neel et al. [2021] adopt similar DP-inspired definitions of (approximate) unlearning based on the goal of indistinguishability from retrain-from-scratch.
However, there is a recent line of criticism against the idea of defining or measuring unlearning success based on indistinguishability from retrain-from-scratch. Thudi et al. [2022b] theoretically show that we can obtain arbitrarily similar model weights from training on two non-overlapping datasets. This implies that reaching a particular point in parameters space isn't a sufficient condition for having unlearned. Furthermore, Goel et al. [2022] argue that it's not a necessary condition either: retraining from scratch may arrive at different model distributions due to changes in e.g. hyperparameters [Yang and Shami, 2020]. Therefore, a unlearning method may be unnecessarily penalized for not matching a particular retrained-from-scratch model. In our work, we take a practical viewpoint and define unlearning as 'forgetting' in an efficient manner and without hurting model utility, for three different application-specific metrics for forgetting that we discuss in the next section.
Unlearning methods Cao and Yang [2015] introduce unlearning and provide an exact forgetting algorithm for statistical query learning. [Bourtoule et al., 2021] proposed a training framework by sharding data and creating multiple models, enabling exact unlearning of certain data partitions. Ginart et al. [2019] introduce a probabilistic definition for machine unlearning inspired by Differential