SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
Devansh Arpit, Stanisław Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242. PMLR, 2017.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159. IEEE, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pages 463-480. IEEE, 2015.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914. IEEE, 2022.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 896-911, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. arXiv preprint arXiv:2205.08096, 2022.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211-407, 2014.
Simone Fabbrizzi, Symeon Papadopoulos, Eirini Ntoutsi, and Ioannis Kompatsiaris. A survey on bias in visual datasets. Computer Vision and Image Understanding, page 103552, 2022.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Shashwat Goel, Ameya Prabhu, and Ponnurangam Kumaraguru. Evaluating inexact unlearning requires revisiting forgetting. arXiv preprint arXiv:2201.06640, 2022.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304-9312, 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In European Conference on Computer Vision, pages 383-398. Springer, 2020b.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303-319. IEEE, 2022a.
Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22), pages 4007-4022, 2022b.
Eleni Triantafillou, Fabian Pedregosa, Jamie Hayes, Peter Kairouz, Isabelle Guyon, Meghdad Kurmanji, Gintare Karolina Dziugaite, Peter Triantafillou, Kairan Zhao, Lisheng Sun Hosoya, Julio C. S. Jacques Junior, Vincent Dumoulin, Ioannis Mitliagkas, Sergio Escalera, Jun Wan, Sohier Dane, Maggie Demkin, and Walter Reade. Neurips 2023 - machine unlearning, 2023. URL https://kaggle.com/competitions/neurips-2023-machine-unlearning.
Chen Wu, Sencun Zhu, and Prasenjit Mitra. Federated unlearning with knowledge distillation. arXiv preprint arXiv:2201.09441, 2022.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves ImageNet classification. In Conference on Computer Vision and Pattern Recognition, pages 10687-10698, 2020.
Li Yang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415:295-316, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 253-261, 2020.
Zijie Zhang, Yang Zhou, Xin Zhao, Tianshi Che, and Lingjuan Lyu. Prompt certified machine unlearning with randomized gradient smoothing and quantization. Advances in Neural Information Processing Systems, 35:13433-13455, 2022.
This Appendix contains the following sections:
Section 8: Limitations and Future Work
Section 9: Experimental Details and Pseudocode
Section 10: Formal Description of Metrics
- Section 11: Membership Inference Attacks: Description and Additional Findings
Section 12: Ablations and Sensitivity Analysis
Section 13: Larger-scale settings
- Section 14: Additional Results for Removing Biases (RB)
- Section 15: Additional Results for Resolving Confusion (RC)
- Section 16: Additional Results for User Privacy (UP)
SCRUB has shown impressive results in terms of being consistently a top-performer in terms of unlearning, with a minimal drop in performance compared to previous works. However, SCRUB has limitations that we hope future work will address.
A significant step for future work is to develop theoretical guarantees for the gains provided by our methods. We opted to focus on an empirical approach for the following reasons: First, while theoretical guarantees abound for linear models, deep networks pose additional significant challenges. Second, methods accompanied with theoretical guarantees suffer from practical limitations with respect to accuracy and/or scalability. For these reasons we opted to approach the problem from a practical standpoint, pushing the envelope by developing unlearning algorithms which are top performers across many important different application scenarios, different evaluation metrics, different architectures and datasets. We look forward to future work that strives to strike a compromise between effective unlearning, good performance, scalability, and theoretical insights.
Another limitation of SCRUB is the difficulty and instability associated with tuning the min-max objective, as shown in the literature e.g. for GANs. For instance, this can lead to oscillating behaviour, as we show in Figure 6. We remedy this to a large extent in practice by providing a practical algorithm that works well, showing consistently improved results over prior work, but there is room for improvement on this front for future work.
SCRUB's rewinding procedure also has limitations. We find in practice throughout all of our experiments that it can help to substantially increase the success of SCRUB's defense on MIA in scenarios where the forget error obtained by SCRUB at the end of unlearning is 'too high'. However, a different failure case which can also appear is that SCRUB's forget error at the end of training is 'too low'. This can happen due to the way in which we tune hyperparameters, which is designed to not harm the retain and validation performance too much, and thus can in some cases lead to 'premature stopping' before the forget error reaches the same level as a reference point for how high it would be if the model had truly never seen those examples. We highlight that addressing all possible issues that can arise in all scenarios and provide an unlearning algorithm that performs strongly across the board is extremely challenging. The fact that we have observed failure cases for each algorithm, be it SCRUB or other baselines, is indicative of the extensiveness of the experimentation we conducted. Our work has made important strides in designing consistently strong-performing unlearning methods and we look forward to future contributions in this direction.
We hope that future work also continues to push the limits of scalability. We believe that our work constitutes an important step in this direction. However, the datasets and models we consider aren't too large, in order to allow comparisons to previous works that would not be feasible to run for larger scale experiments. An interesting topic of future work is investigating the interplay between SCRUB and other scalable algorithms like NegGrad+ with increasing amounts of scale.
Another really interesting future work direction is to investigate how different unlearning algorithms interact with different architectures, like Transformers, and loss functions, like self-supervised learning.
9 Experimental Details and Pseudocode
Algorithm 1 SCRUB
Require: Teacher weights  $w^o$    
Require: Total max steps MAX-STEPS   
Require: Total steps STEPS   
Require: Learning rate  $\epsilon$ $w^{u}\gets w^{o}$ $i\gets 0$    
repeat if  $i <   \mathrm{MAX - STEPS}$  then  $w^{u}\gets \mathrm{DO - MAX - EPOCH}(w^{u})$  end if  $w^{u}\gets \mathrm{DO - MIN - EPOCH}(w^{u})$    
until  $i <   \mathrm{STEP}$
Datasets. We have used CIFAR-10 and Lacuna-10 datasets for evaluation purposes. CIFAR-10 consists of 10 classes with 60000 color images of size  $32 \times 32$ . In our experiments, the train, test, and validation sizes are 40000, 10000, and 10000 respectively. Lacuna-10 is a dataset derived from VGG-Faces [Cao and Yang, 2015]. We have followed the same procedure described in [Golatkar et al., 2020a] to build Lacuna. We randomly select 10 celebrities (classes) with at least 500 samples. We use 100 samples of each class to form the test-set, and the rest make the train-set. All the images are resized to  $32 \times 32$ . We also use CIFAR-100 and Lacuna-100 to pretrain the models. Lacuna-100 is built in a similar way as Lacuna-10, and there is no overlap between the two datasets. We have not applied any data augmentation throughout the experiments.
Small-Scale datasets. We followed the same procedure as described in [Golatkar et al., 2020b] to create the small versions of CIFAR-10 and Lacuna-10, namely CIFAR-5 and Lacuna-5. To this end, we take the first 5 classes of each dataset and randomly sample 100 images for each class. We make the train and test sets by sampling from the respective train and test sets of CIFAR-10 and Lacuna-10. We also make 25 samples from each class from the train set to create the validation sets.
Algorithm 2 DO-MAX-EPOCH
Require: Student weights  $w^{u}$    
Require: Learning rate  $\epsilon$    
Require: Batch size B   
Require: Forget set  $D_{f}$    
Require: Procedure NEXT-BATCH  $b\gets$  NEXT-BATCH  $(D_{f},\mathrm{B})$  repeat  $w^{u}\gets w^{u} + \epsilon \nabla_{w^{u}}\frac{1}{|b|}\sum_{x_{f}\in b}d(x_{f};w^{u})$ $b\gets$  NEXT-BATCH  $(D_{f},\mathrm{B})$  until  $b$
Models. We use the same models with the same architectural modifications in [Golatkar et al., 2020a,b]. For All-CNN, the number of layers is reduced and batch normalization is added before each non-linearity. For Resnet, ResNet-18 architecture is used. For small scale experiments, the number of filters is reduced by  $60\%$  in each block. For the large-scale experiments, the exact architecture has been used.
Pretraining. Following the previous work for consistency, we apply pretraining. Specifically, for CIFAR datasets, we have pretrained the models on CIFAR-100. For Lacuna, we have pretrained the models on Lacuna-100. We pretrain the models for 30 epochs using SGD with a fixed learning rate of 0.1, Cross-Entropy loss function, weight decay 0.0005, momentum 0.9, and batch
size 128.
Baselines. 'Original' is the model trained on the entire dataset  $D$ . For 'Retrain', we train the same architecture on  $D_r$ , with the same hyperparameters used during training of the original model. For 'Finetune', we fine-tune the 'original' model on  $D_r$  for 10 epochs, with a fixed learning rate of 0.01 and weight-decay 0.0005. For 'NegGrad+', we fine-tune the 'original' model using the following loss:
$$
\mathcal {L} (w) = \beta \times \frac {1}{| D _ {r} |} \sum_ {i = 1} ^ {| D _ {r} |} l \left(f \left(x _ {i}; w\right), y _ {i}\right) - (1 - \beta) \times \frac {1}{| D _ {f} |} \sum_ {j = 1} ^ {| D _ {f} |} l \left(f \left(x _ {j}; w\right), y _ {j}\right) \tag {4}
$$
Where  $\beta \in [0,1]$ . We have tuned  $\beta$  to get a high forget-error while not destroying retain-error and validation-error. For small-scale experiments,  $\beta = 0.95$  and we have trained for 10 epochs, with SGD, 0.01 lr and 0.1 weight-decay. For large-scale experiments  $\beta = 0.9999$  and we have trained for 5 epochs, with SGD, 0.01 lr, and 0.0005 weight-deay. Please note that small  $\beta$  result in explosion quickly. For 'CF-k', we freeze the first k layers of the network and finetune the rest layers with  $D_r$ . We use the same setting as 'Finetune' baseline. For 'EU-k' we freeze the first k layers, and
model	dataset	unlearning-type	forget-set bs	retain-set bs	max steps	min steps
ResNet	CIFAR-10	class	512	128	2	3
CIFAR-10	selective	16	64	5	5
Lacuna-10	class	128	128	5	5
Lacuna-10	selective	32	32	4	4
CIFAR-5	selective	32	32	10	10
Lacuna-5	selective	32	32	5	10
All-CNN	CIFAR-10	class	512	256	3	4
CIFAR-10	selective	16	64	5	5
Lacuna-10	class	32	32	4	4
Lacuna-10	selective	8	32	2	4
CIFAR-5	selective	16	32	5	10
Lacuna-5	selective	32	32	5	10
Table 3: SCRUB's hyperparameters for each experiment
re-initialize the weights of the remaining layers and retrain them with  $D_r$ . As all the models are pretrained on larger datasets, for re-initializing we use the weights of the pretrained models. For 'EU-k' we use the same settings as the 'Retrain' baseline. In both 'EU-k' and 'CF-k' baselines, for both ResNet and All-CNN we freeze all the layers except for the last block of the network. For Resnet the last block is block4 and for All-CNN, the last block of layers is the 9th sequential block. For Bad-T, we follow the specifications given in Chandawat et al. [2022] with possible tuning of the parameters in different settings to get the highest forget-error without damaging retain-error. More specifically, for all models we perform one epoch of unlearning using Adam optimizer, and a temperature scalar of 4. Also, we use the whole retain-set compared to  $30\%$  reported in their paper as we empirically observed that using only  $30\%$  of retain-set for Bad-T yields high test errors.
Algorithm 3 DO-MIN-EPOCH
Require: Student weights  $w^{u}$
Require: Learning rate  $\epsilon$
Require: Batch size B
Require: Retain set  $D_r$
Require: Procedure NEXT-BATCH
$$
b \leftarrow \operatorname {N E X T - B A T C H} \left(D _ {r}, \mathrm {B}\right)
$$
$$
w ^ {u} \leftarrow w ^ {u} - \epsilon \nabla_ {w ^ {u}} \frac {1}{| b |} \sum_ {\left(x _ {r}, y _ {r}\right) \in b} \alpha d \left(x _ {r}; w ^ {u}\right) +
$$
$$
\gamma l \left(f \left(x _ {r}; w ^ {u}\right), y _ {r}\right)
$$
until  $b$
SCRUB pseudocode and parameters. We train SCRUB using Algorithm 1. Throughout the experiments, we tune the parameters to get a high forget-error while retaining the retain and validation error of the original model. We use the same optimizer for both min and max steps. We observed that for small-scale settings 'Adam' optimizer works better, while in large-scale settings both 'Adam' and 'SGD' could be used. For all experiments, we initialize the learning rate at 0.0005 and decay it by 0.1 after a number of min and max steps. Decaying the learning rate is crucial to control the oscillating behaviour of our min and max optimization. We apply a weight decay of 0.1 for small-scale setting and 0.0005 for large scale experiments, with a momentum of 0.9. Finally, we use different batch sizes for the forget-set and the retain-set to control the
number of iteration in each direction, i.e the max and the min respectively. We report these in Table 3.
System specification. For scale-up experiments, the code is executed in Python 3.8, on an Ubuntu 20 machine with 40 CPU cores, a Nvidia GTX 2080 GPU and 256GB memory.
10 Formal Description of Metrics
In this section, we give more details and mathematical definitions of the metrics that we use throughout the paper. We first mathematically define the forget, retain and test errors, and then other application-dependent metrics, for Resolving Confusion (RC) and User Privacy (UP).
Forget, retain and test errors Here, we define the retain error, forget error and test error. Let  $\mathcal{D}_r$ ,  $\mathcal{D}_f$  and  $\mathcal{D}_t$  denote the retain and forget portions of the training dataset, and a test dataset of heldout examples, respectively. We define error  $(Err)$  as follows:
$$
\operatorname {E r r} (\mathcal {D}) = 1 - \frac {1}{| \mathcal {D} |} \sum_ {\left(x _ {i}, y _ {i}\right) \in \mathcal {D}} \mathbb {1} [ \arg \max  \left(f \left(x _ {i}; w\right)\right) = = y _ {i} ] \tag {5}
$$
where  $f$ , parameterized by  $w$  is the neural network model (comprised of a feature extractor followed by a softmax classifier layer),  $\arg \max(f(x_i; w))$  is the label that the model thinks is most likely for example  $x_i$ , and  $\mathbb{1}[x]$  is the indicator function that returns 1 if  $x$  is True and 0 otherwise.
Based on the above, the retain error, forget error and test error are computed as  $Err(\mathcal{D}_r)$ ,  $Err(\mathcal{D}_f)$  and  $Err(\mathcal{D}_t)$ , respectively.
Metrics for Unlearning for Resolving Confusion (RC) We now define the class confusion metrics inspired by Goel et al. [2022]. Specifically, we explore a scenario where the forget set has confused labels (e.g. for two classes A and B, examples of A are labelled as B, and vice versa). The idea here is that, because mislabelled examples are only present in the forget set, successful unlearning (removing the influence of the forget set) would lead to a model that is not at all confused between classes A and B.
In more detail, the setup we follow is: 1) We first mislabel some portion of the training dataset (we mislabelled examples between classes 0 and 1 of each of CIFAR-5 and Lacuna-5 in our experiments), 2) train the 'original model' on the (partly mislabelled) training dataset (it has mislabelled examples for classes 0 and 1 but correct labels for the remaining classes), 3) perform unlearning where the forget set contains all and only the confused examples. Given this, the goal for the unlearning algorithm is to resolve the confusion of the original model.
We consider the following metrics (using terminology consistent with Goel et al. [2022]). They are presented in order of decreasing generality, and increasing focus on measuring degrees of confusion between the two classes considered.
- Error (e.g. test error, retain error, forget error). This counts all mistakes, so anytime that an example of some class is predicted to be in any other class, it will be counted. These are the same metrics that we use for the rest of the paper (see Equation 5). For test and retain error, lower is better, whereas for forget error, higher is better.
- Interclass Confusion IC-ERR (e.g. IC test error, IC retain error). This counts only mistakes that involve examples from the confused classes A and B. Specifically, it counts instances of any example of class A being predicted to be in any other class, and similarly for class B. Compared to Error, this metric is more focused towards understanding the result of the introduced confusion, since it only considers cases that relate to the confused classes. A successful unlearning method would make no such errors, so lower is better for each of IC test error and IC retain error.
- FGT-ERR (e.g. Fgt test error, Fgt retain error). This metric counts only misclassification between the confused classes A and B. Here, a mistake of an example of class A (or B) being predicted to be in class other than A or B will not be counted. Only mistakes of an example of class A being predicted to be in class B, and vice versa, are counted. This is the most focused metric that directly measures the amount of remaining confusion between the two classes in question. A successful unlearning method would make no such errors, so lower is better for each of Fgt test and Fgt retain.
More formally, Error is the same as defined in Equation 5. Let us now mathematically define IC-ERR and FGT-ERR. We denote by  $C^{w,\mathcal{D}}$  the confusion matrix for model parameterized by  $w$  on the dataset  $\mathcal{D}$ , and let  $\mathcal{D}_A$  denote the part of the dataset  $\mathcal{D}$  that belongs to class  $A$ . So, for example  $\mathcal{D}_{r_A}$  denotes the part of the retain set  $\mathcal{D}_r$  that belongs to class  $A$ , and the entry  $C_{A,B}^{w,\mathcal{D}}$  of the confusion matrix stores the number of times that a sample belonging to class  $A$  was (mis)classified as belonging to class  $B$  in the dataset  $\mathcal{D}$  by the model parameterized by  $w$ . Then, we have:
$$
\operatorname {I C - E R R} (\mathcal {D}, A, B; w) = \frac {\sum_ {k} C _ {A , k} ^ {w , \mathcal {D}} + \sum_ {k ^ {\prime}} C _ {B , k ^ {\prime}} ^ {w , \mathcal {D}}}{| \mathcal {D} _ {A} | + | \mathcal {D} _ {B} |} \tag {6}
$$
where  $k\neq A,k^{\prime}\neq B$
So, for example, the 'IC test error' column in our tables is computed via IC-ERR  $(\mathcal{D}_t,0,1;w)$ , where  $\mathcal{D}_t$  denotes the test set, and 0 and 1 are the two classes confused in our experiments. Analogously, 'IC retain error' is computed as IC-ERR  $(\mathcal{D}_r,0,1;w)$
$$
\operatorname {F G T - E R R} (\mathcal {D}, A, B; w) = C _ {A, B} ^ {w, \mathcal {D}} + C _ {B, A} ^ {w, \mathcal {D}} \tag {7}
$$
That is, FGT-ERR only measures the misclassification between the two confused classes A and B. So, for example, the 'Fgt test error' in our tables is computed as FGT-ERR  $(\mathcal{D}_t,0,1;w)$  and analogously 'Fgt retain error' is computed as FGT-ERR  $(\mathcal{D}_r,0,1;w)$ .
User Privacy (UP) Metrics Please see the next section for full details for each of the two Membership Inference Attacks (MIAs) that we use and experimental results.
11 Membership Inference Attacks: Description and Additional Findings
As mentioned in our paper, we utilize two different MIAs: 1) a 'Basic MIA' that is similar to the ones typically used in unlearning papers (but far from the state-of-the-art of MIAs used by privacy and security colleagues), and 2) the first, to our knowledge, adaptation of the state-of-the-art LiRA attack [Carlini et al., 2022] to the framework of unlearning ('LiRA-for-unlearning' MIA).
In this section, we use the term 'target model' to refer to the model that is being attacked and the term 'target example' to refer to an example whose membership status ('in' or 'out') the attacker tries to predict, based on the 'behaviour' (e.g loss value) of that example under the 'target model'. In both attacks that we consider, the target model is the unlearned model, and target examples are either forget set ('in') or test set ('out') examples. The unlearning algorithm successfully defends an MIA if the attacker can't tell apart examples that were unlearned (forget set examples) from examples that were truly never seen.
11.1 Basic MIA
Returning to our previous notation, let  $l(f(x; w^u), y)$  denote the cross-entropy loss of the unlearned model (a deep network  $f$  with weights  $w^u$ ) on example  $x$  with label  $y$ . We abbreviate this as  $l(x, y)$  from now on; dropping the dependence on  $f$  and  $w^u$ .