SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Privacy [Dwork et al., 2014], and formed the origin of the idea of the model indistinguishability goal. More recently, Guo et al. [2019], Izzo et al. [2021], Sekhari et al. [2021] built upon this framework and introduced unlearning methods that can provide theoretical guarantees under certain assumptions. Further, Huang and Canonne [2023] provide bounds for 'deletion capacity' in the DP-based unlearning methods using the group property of DP. [Graves et al., 2021] proposed two methods applicable to deep networks. The first randomly re-labels the forget set and retrans for a small number of updates, while the second stores the index of examples that participated in each batch and the per-batch parameter updates, and unlearns by 'undoing' the affected updates. This requires a lot of storage and can also lead to a poor proxy of the model that would have been obtained had those parameter updates not happened in the first place, especially for larger sequences. [Golatkar et al., 2020a] proposed an information-theoretic procedure for removing information about  $\mathcal{D}_f$  from the weights of a neural network and [Golatkar et al., 2020b, 2021] propose methods to approximate the weights that would have been obtained by unlearning via a linearization inspired by NTK theory [Jacot et al., 2018] in the first case, and based on Taylor expansions in the latter. However, [Golatkar et al., 2020a,b] scale poorly with the size of the training dataset, as computing the forgetting step scales quadratically with the number of training samples. [Golatkar et al., 2021] addresses this issue, albeit under a different restrictive assumption: that a large dataset is available for pre-training that will remain 'static', in that no forgetting operations will be applied on it; an assumption that is not met in many important applications (e.g. healthcare, medical images). Some more recent works propose modifications to the original model training, to make the resulting model more amenable to unlearning. Thudi et al. [2022a] propose a regularizer to reduce the 'verification error', which is an approximation to the distance between the unlearned model and a retrained-from-scratch model. The goal is to make unlearning easier in the future. Additionally, they propose a single gradient update that reverses the effect of  $\mathcal{D}_f$ . This regularizer, however, may have a negative effect on the model's performance, and the single gradient unlearning step only removes the effect that  $\mathcal{D}_f$  has had in the first iteration of training. [Zhang et al., 2022] present a training process that quantizes gradients and applies randomized smoothing. This process is designed to make unlearning unnecessary in the future and comes with certifications under some conditions. However, a deletion request that causes too large a change to the data distribution (e.g. if doing class unlearning), may exceed the 'deletion budget' and invalidate their assumptions.
Related research areas Differential Privacy (DP) [Dwork et al., 2014] and Life-Long Learning (LLL) [Parisi et al., 2019] have shared goals with unlearning. DP seeks to ensure that the trained model doesn't store information about any training instance; a strong goal that is difficult to achieve for deep neural networks [Abadi et al., 2016]. In contrast, in unlearning we only desire to remove the influence of the training instances in the forget set, which can be seen as a relaxation of the same problem. LLL seeks to continuously update a model to solve new sets of tasks, without 'catastrophically forgetting' previously-learned tasks. The notion of forgetting a task, though, is distinct from that of forgetting a training example: the performance on a task can degrade, without having removed the influence of particular examples from the model weights.
Teacher-student and contrastive methods The teacher-student methodology has been used in a different contexts, e.g. self-supervised [Grill et al., 2020, Chen and He, 2021] and semi-supervised learning [Xie et al., 2020]. Knowledge Distillation [Hinton et al., 2015] can also be thought of as a teacher-student approach with the additional desire for compression, where the student is chosen to have a smaller architecture than the teacher. In addition, our training objective has a contrastive flavour [Chen et al., 2020, He et al., 2020], due to both pulling the student close to the teacher, and pushing it away from it, for different parts of the training set. For unlearning, [Kim and Woo, 2022, Wu et al., 2022] propose methods with a two-stage pipeline: 'neutralization' / forgetting followed by a 'retraining' / restoring, that restores performance that may be lost in the first phase. Both use distillation only in the second phase on only the retain set. In contrast, SCRUB employs a single phase where we carefully design a teacher-student formulation to both encourage forgetting the requested data and retaining the other knowledge, simultaneously. Most similar to our work in that regard is the method of [Chundawat et al., 2022] that employs a related but different teacher-student objective that encourages moving close to a 'Bad Teacher' for the forget set (we thus refer to this as 'Bad-T'). Moving close to a different, 'bad', teacher for forget examples is fundamentally different from our goal of moving away from the (single) teacher as in SCRUB. We find that Bad-T forgets poorly compared to SCRUB in all applications, and degrades model quality.
ResNet	All-CNN
Model	IC test error (↓) mean	std	IC retain error (↓) mean	std	Fgt test error (↓) mean	std	Fgt retain error (↓) mean	std	IC test error (↓) mean	std	IC retain error (↓) mean	std	Fgt test error (↓) mean	std
Retrain	24.0	1.8	0.0	0.0	18.3	4.2	0.0	0.0	19.0	1.3	0.0	0.0	11.3	4.6
Original	56.0	3.0	0.0	0.0	92.0	7.9	0.0	0.0	49.0	4.8	6.0	10.39	80.7	12.6
Finetune	52.0	3.1	0.0	0.0	79.3	10.1	0.0	0.0	43.7	7.3	0.0	0.0	67.3	16.0
NegGrad+	47.5	5.3	0.0	0.0	69.0	13.5	0.0	0.0	42.3	11.3	0.0	0.0	62.0	22.6
CF-k	54.8	2.0	0.0	0.0	85.3	7.0	0.0	0.0	47.8	5.8	0.0	0.0	76.3	14.4
EU-k	47.0	4.8	8.3	4.7	63.3	9.7	3.7	2.5	59.5	5.2	38.3	6.7	68.7	15.6
Fisher	51.5	7.5	26.3	9.5	79.0	3.6	20.0	7.9	56.8	8.7	31.7	14.0	78.3	15.5
NTK	37.5	4.0	0.0	0.0	52.0	10.6	0.0	0.0	36.7	4.1	3.0	5.2	54.3	9.0
Bad-T	47.3	9.7	6.0	6.2	27.0	8.7	0.3	0.6	39.3	8.2	3.0	1.0	25.7	5.0
SCRUB	19.0	3.9	0.0	0.0	19.7	7.5	0.0	0.0	26.0	4.4	0.0	0.0	18.0	11.1
Table 1: RC-application results: SCRUB is the strongest performer in terms of resolving class confusion. These results are on CIFAR-5 with ResNet and All-CNN, where we confused 50 samples of class 0 by mislabeling them as class 1, and 50 of class 1 by mislabeling them as class 0.
Overview of applications and metrics We consider three sets of forget-quality metrics, for different applications. First, we consider the scenario of Removing Biases (RB) where we desire the highest possible forget set error (reflecting that we want the model to never predict the associated labels of the forget set, since those capture an unintended behaviour / bias). Next, inspired by Goel et al. [2022], we consider unlearning for Resolving Confusion (RC) between classes, caused by a portion of the model's original training set being mislabelled. Finally, we consider a scenario of data deletion for User Privacy (UP). In this case, we use Membership Inference Attacks (MIAs) to measure success (including the first, to our knowledge, adaptation of the state-of-the-art LiRA MIA [Carlini et al., 2022] to unlearning), where unlearning succeeds if the attacker is unable to tell apart forgotten examples from truly unseen ones. In all cases, aside from achieving the application-specific forgetting goal, we also measure utility (performance on  $\mathcal{D}_r$  and a test set  $\mathcal{D}_t$ ) as an additional metric.
Experimental details We utilize the same two datasets from previous work: CIFAR-10 [Krizhevsky et al., 2009] and Lacuna-10 [Golatkar et al., 2020a], which is derived from VGG-Faces [Cao and Yang, 2015], and the same two architectures: All-CNN [Springenberg et al., 2014] and ResNet-18 [He et al., 2016]. For consistency with previous work, we pre-train the original model on CIFAR-100 / Lacuna-100 for the CIFAR-10 / Lacuna-10 experiments, respectively [Golatkar et al., 2020a,b]. We run each experiment with 3 random seeds and report the mean and standard deviation.
We consider two settings throughout the paper: small-scale and large-scale. The former allows comparisons with NTK, which
doesn't scale beyond it. For the small-scale, we exactly follow the setup in [Golatkar et al., 2020b] that uses only 5 classes from each of CIFAR and Lacuna ('CIFAR-5' / 'Lacuna-5'), with 100 train, 25 validation and 100 test examples per class. The forget set contains 25 examples from class 0 (5%). For the large-scale, we exactly follow the setup in [Golatkar et al., 2020a] that uses all 10 classes of each of CIFAR and Lacuna, and considers both a class unlearning scenario where the forget set is the entire training set for class 5 (10%), as well as a selective unlearning one where 100 examples of class 5 are forgotten (0.25% in CIFAR and 2% in Lacuna). We note that the Appendix contains complete results for all scenarios (several are omitted from the main paper due to space constraints) and also contains the complete experimental details and hyperparameters.
(a) Small-scale.
(b) Large-scale.
Figure 2: The number of times each method was a top performer in forgetting in RB-application. A model is a top-performer if its  $95\%$  confidence interval overlaps with that of the best mean. Small-scale counts are over  $\{\mathrm{ResNet,All - CNN}\} \times \{\mathrm{CIFAR,Lacuna}\}$  (4 total), large-scale additionally x {class, selective} (8 total). SCRUB is the most consistent top-performer.
Unlearning algorithms We compare against state-of-the-art approaches as well as various baselines: Retrain: Retraining from scratch without the forget set. This is assumed to not be viable in practice, but we include it as a reference. Original: The 'original' model trained on all of  $\mathcal{D}$  without any unlearning. Finetuning: Finetunes the original model on data from the retain set  $\mathcal{D}_r$ . NegGrad+: Similar to Finetuning, starts from the original model and finetunes it both on data from the retain and forget sets, negating the gradient for the latter. Previous work considered a weaker baseline that only trains on the forget set, with a negative gradient (NegGrad). We tune the stronger NegGrad+ to achieve a good balance between the two objectives in the same way as SCRUB. Fisher Forgetting [Golatkar et al., 2020a]. NTK Forgetting [Golatkar et al., 2020b]. Catastrophic Forgetting-k (CF-k) and Exact Unlearning-k (EU-k) [Goel et al., 2022]: they freeze the first k layers of the original model and either finetune the remaining layers on  $\mathcal{D}_r$  (CF-k) or train them from scratch on  $\mathcal{D}_r$  (EU-k). Bad-T: the concurrent teacher-student method of [Chundawat et al., 2022]. We utilize the public code for the NTK and Fisher for correctness. We implement Bad-T ourselves (no code was available).
5.2 Unlearning for Removing Biases (RB)
In RB, we desire to achieve the highest possible error on  $D_{f}$ , without hurting the error of  $D_{r}$  and  $D_{t}$ . Some example scenarios are removing biases from trained models, outdated or incorrect information. A maximal forget error on  $D_{f}$  in this case is desirable as it reflects that we want the model to never make the predictions that were deemed harmful or no longer valid or accurate.
We report the results for this scenario in Figures 1 and 2. We also report complete results in the Appendix for all architectures, datasets and baselines. As shown in Figure 2, SCRUB is by far the most consistent method in terms of achieving high forget quality (highest forget error), while it doesn't hurt retain and test errors.
5.3 Unlearning for Resolving Confusion (RC)
In this section, following Goel et al. [2022], we study unlearning for resolving confusion between two classes that the original model suffers from due to a part of its training set being mislabelled. Specifically, we place all and only the mislabeled training examples in the forget set. A successful unlearning algorithm would thus entirely resolve the model's confusion. In more detail, the setup is: 1) Mislabel some portion of the training dataset (we mislabeled examples between classes 0 and 1 of each of CIFAR-5 and Lacuna-5), 2) Train the 'original model' on the (partly mislabeled) training dataset, and 3) Unlearn with the confused examples as the forget set.
We consider the following metrics inspired from Goel et al. [2022] that are variants of the model's error on examples from the confused classes (lower is better for both). We outline them below and define them formally in the Appendix.
- Interclass Confusion IC-ERR (e.g. IC test error, IC retain error). This counts mistakes where an example from either of the two confused classes was incorrectly predicted to belong to any other class.
- FGT-ERR (e.g. Fgt test error, Fgt retain error). This metric counts only misclassification between the confused classes, i.e. an example of class 0 being predicted as belonging to class 1 or vice versa.
We observe from Table 1 that, across the board, SCRUB is by far the most consistent method in terms of eliminating class confusion without damaging the quality of the model (on retain and test sets).
5.4 Unlearning for User Privacy (UP)
Next, we consider a privacy-critical application: the deletion of the data of a user exercising their right-to-be-forgotten. We use Membership Inference Attacks (MIAs) as the metric where the goal is that, after applying unlearning, an attacker can't tell apart examples that were unlearned from those that were truly never seen, thus protecting the privacy of the user requesting deletion.
While previous unlearning papers have reported MIA results [Golatkar et al., 2020a], those MIAs are simple and far from state-of-the-art MIAs used in privacy and security papers. Indeed, it's not trivial to adapt such MIAs to the unlearning protocol. To initiate bridging this gap, we report results on two MIAs: 1) a simple one that is closer to the typical MIAs used in the unlearning literature, and
(a) Forget 100 examples.
(b) Forget 150 examples.
(c) Forget 175 examples.
(d) Forget 200 examples.
Figure 3: MIA results for different sizes of forget set on CIFAR-10 with ResNet. Error bars show  $95\%$  confidence intervals. Rewinding is most useful for smaller forget sizes (see the discussion in Section 5.4). SCRUB+R successfully defends MIAs, comparably to the Retrain oracle.
2) the first, to our knowledge, adaptation of the state-of-the-art LiRA attack [Carlini et al., 2022] to the unlearning protocol. We describe both at a high-level here and in detail in the Appendix.
Basic attack We train a binary classifier (the 'attacker') on the unlearned model's losses on forget and test examples for the objective of classifying 'in' (forget) versus 'out' (test). The attacker then makes predictions for held-out losses (losses the attacker wasn't trained on) that are balanced between forget and test losses. For a perfect defense, the attacker's accuracy is  $50\%$ , indicating that it is unable to separate the two sets, marking a success for the unlearning method.
We present Basic MIA results in Table 2 for CIFAR-10. We observe that SCRUB+R is a strong performer. EU-k can also perform strongly in this metric, though not as consistently. We also observe that rewinding was only triggered once, for selective unlearning. Recall that, in both selective and class unlearning, the forget set contains examples coming from a single class, but in the former, it doesn't contain all examples of that class; the rest are in the retain set. Thus, we indeed expect that rewinding would be more useful for selective unlearning: the smaller the portion of a class in the forget set, the larger the remainder of that class in the retain set and consequently, the better we expect the model to generalize on held-out examples of that class (lower error), making it in turn more likely to need to rewind, in order to also lower
the forget set error commensurately. In Figure 3 we plot results for different variants of selective unlearning, with different number of examples in the forget set (from the same class) and indeed find that for smaller numbers, rewinding is triggered. Our take-away is that, when needed, rewinding can substantially improve SCRUB's MIA results.
ResNet	All-CNN
Model	Class	Selective	Class	Selective
mean	std	mean	std	mean	std	mean	std
Retrain	49.33	1.67	54.00	1.63	55.00	4.00	48.73	0.24
Original	71.10	0.67	65.33	0.47	66.50	0.50	71.40	0.70
Finetune	75.57	0.69	64.00	0.82	68.00	1.00	74.97	1.27
NegGrad+	69.57	1.19	66.67	1.70	72.00	0.00	70.03	1.92
CF-k	75.73	0.34	65.00	0.00	69.00	2.00	72.93	1.06
EU-k	54.20	2.27	53.00	3.27	66.50	3.50	51.60	1.22
Bad-T	54.00	1.10	59.67	4.19	63.4	1.2	77.67	4.11
SCRUB	52.20	1.71	78.00	2.45	52.00	0.00	54.30	2.24
SCRUB+R	52.20	1.71	58.67	1.89	52.00	0.00	54.30	2.24
Table 2: Basic MIA results (for UP): SCRUB successfully defends MIA. Note the rewinding procedure of SCRUB+R was triggered only once; see discussion in Section 5.4 and Figure 3. In the Appendix, we also show that the forget error of SCRUB(+R) is close to that of Retrain, as desired.
LiRA-for-unlearning attack In the standard privacy setting, the LiRA attacker trains a large number of 'shadow models' [Carlini et al., 2022], for which it controls which examples are in the training set each time (by construction). To then predict the membership status of a 'target example', it estimates two Gaussian distributions: the distribution of confidences of that example under shadow models that trained on it, and the distribution of its confidences under shadow models that didn't. It predicts that the target example is 'in' if the likelihood of the former is larger than that of the latter.
We propose the first, to our knowledge, adaptation of LiRA for unlearning. This is a strong attack where we allow the attacker knowledge of the unlearning algorithm. Concretely, for each shadow model, the attacker also produces a 'shadow unlearned' model by applying the given unlearning algorithm several times, using a large number of forget sets (similar to Chen et al. [2021]). Now, for each 'target example', this setup allows the attacker to estimate a different pair of Gaussians: the distribution of (confidences of) that target example under models where it was forgotten, and as before, under models where it was not seen. The attacker predicts the example was forgotten if its likelihood under the former is larger than under the latter. We present all details and findings in the Appendix and observe that SCRUB+R outperforms the other methods in defending this attack.
5.5 Overview of results and take-aways
Finetuning is a poor method: it may retain model utility but fails to forget. The previous state-the-art NTK and Fisher models aren't among the top-performers either. The former performs poorly across the board in terms of forgetting and doesn't scale beyond small datasets. The latter sometimes performs well in RB in terms of forgetting, though not always and it is very slow (Figure 1), exceeding the runtime of Retrain; as also observed in [Goel et al., 2022].  $\mathbf{NegGrad}+$ , our proposed enhancement over the NegGrad baseline, is a strong baseline in terms of achieving a balance between forgetting and utility (it performs well in several settings in terms of RB, albeit not as consistently as SCRUB) and we encourage future work to report this baseline, though SCRUB outperforms it, especially for RC. CF-k inherits the performance profile of Finetune: it maintains utility but forgets poorly. On the other hand, EU-k more reliably forgets (performs strongly in UP, and often in RB), which is expected relative to CF-k due to retraining part of the network, but SCRUB outperforms it significantly in terms of RC and is more consistently a top performer in RB. A notable failure case that we discovered for EU-k in RB is selective unlearning (notice the contrast between EU-k's forget error between e.g. Figures 1a and 1b). This finding may speak to where class vs instance information is stored in neural networks and warrants further investigation. Bad-T is very efficient, but performs poorly in terms of forgetting, across all applications and also damages model utility. Overall, SCRUB is by far the most consistent in successfully forgetting under different metrics without hurting utility and being efficient.
In the Appendix, we discuss limitations of our work, broader impact, future work and present full experimental results and all details including hyperparameters and pseudocode.
Despite substantial recent attention [Triantafillou et al., 2023], unlearning is a young area of research.
In this work, we propose SCRUB, an unlearning method that is unbound from limiting assumptions and poor scalability limits. SCRUB(+R) is found empirically to perform very strongly: it is by far the most consistent method in achieving good forgetting quality with respect to different application-dependent metrics, while incurring only minimal utility degradation. As such, we believe that SCRUB fills the important need for scalable unlearning methods that perform well in practice on several metrics of interest. However, important questions remain open for future work.
Crucially, despite progress in this direction, a well-established formal definition of the problem of unlearning remains elusive, and consequently we also lack well-established metrics for measuring (theoretically or empirically) the quality of unlearning algorithms. In this work, we focus on empirical evaluation with respect to different application-dependent metrics for forgetting quality that we believe are relevant in real-world applications. However, as the unlearning community matures, we hope that success criteria for (different applications of) unlearning will be formalized and standardized.
Further, an important limitation of SCRUB is the absence of theoretical guarantees, making it perhaps ill-suited for certain application scenarios. While methods that come with guarantees exist, they either aren't applicable to deep neural networks, or we find through our empirical analysis that they perform poorly and don't scale beyond small datasets. Therefore, future work that theoretically studies scalable methods like SCRUB would be very valuable. In absence of that, continuing to study and better understand the trade-offs offered by different methods is an important direction.
On the practical side, we hope that future work continues to push the limits of scalability and studies unlearning in larger models, different architectures, different training objectives like self-supervised learning and different domains and modalities, including foundation language models.
Broader Impact While recent advances in deep learning represent exciting opportunities for our community, they also come with great responsibility. As researchers, we are responsible for understanding and mitigating the issues associated with the widespread use of deep learning technology. Machine learning models may carry harmful biases, unintended behaviours, or compromise user privacy. Our work is intended to take a step in addressing these issues via a post-processing 'unlearning' phase that makes progress over previous solutions in practice, as we show through an extensive empirical investigation. However, SCRUB does not come with theoretical guarantees and we can not prove that applying SCRUB perfectly mitigates those issues, so caution must be taken in practice and proper auditing of machine learning models is critical.
We would like to thank Fabian Pedregosa for countless insightful discussions, especially on the topic of membership inference attacks.