SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinghan Jia $^{1,\star}$  Jiancheng Liu $^{1,\star}$  Parikshit Ram $^{2}$  Yuguang Yao $^{1}$  Gaowen Liu $^{3}$
Yang Liu4,5
Sijia Liu<sup>1,2</sup>
<sup>1</sup>Michigan State University, <sup>2</sup>IBM Research, <sup>3</sup>Cisco Research,
<sup>4</sup>University of California, Santa Cruz, <sup>5</sup>ByteDance Research, <sup>6</sup>Carnegie Mellon University *Equal contribution
In response to recent data regulation requirements, machine unlearning (MU) has emerged as a critical process to remove the influence of specific examples from a given model. Although exact unlearning can be achieved through complete model retraining using the remaining dataset, the associated computational costs have driven the development of efficient, approximate unlearning techniques. Moving beyond data-centric MU approaches, our study introduces a novel model-based perspective: model sparsification via weight pruning, which is capable of reducing the gap between exact unlearning and approximate unlearning. We show in both theory and practice that model sparsity can boost the multi-criteria unlearning performance of an approximate unlearner, closing the approximation gap, while continuing to be efficient. This leads to a new MU paradigm, termed prune first, then unlearn, which infuses a sparse model prior into the unlearning process. Building on this insight, we also develop a sparsity-aware unlearning method that utilizes sparsity regularization to enhance the training process of approximate unlearning. Extensive experiments show that our proposals consistently benefit MU in various unlearning scenarios. A notable highlight is the  $77\%$  unlearning efficacy gain of fine-tuning (one of the simplest unlearning methods) when using sparsity-aware unlearning. Furthermore, we demonstrate the practical impact of our proposed MU methods in addressing other machine learning challenges, such as defending against backdoor attacks and enhancing transfer learning. Codes are available at https://github.com/OPTML-Group/Unlearn-Sparse.
Machine unlearning (MU) initiates a reverse learning process to scrub the influence of data points from a trained machine learning (ML) model. It was introduced to avoid information leakage about private data upon completion of training [1-3], particularly in compliance with legislation like 'the right to be forgotten' [4] in General Data Protection Regulation (GDPR) [5]. The direct but optimal unlearning approach is exact unlearning to retrain ML models from scratch using the remaining training set, after removing the data points to be scrubbed. Although retraining yields the ground-truth unlearning strategy, it is the most computationally intensive one. Therefore, the development of approximate but fast unlearning methods has become a major focus in research [6-10].
Despite the computational benefits of approximate unlearning, it often lacks a strong guarantee on the effectiveness of unlearning, resulting in a performance gap with exact unlearning [11]. In particular, we encounter two main challenges. First, the performance of approximate unlearning can heavily rely on the configuration of algorithmic parameters. For example, the Fisher forgetting method [12] needs
to carefully tune the Fisher information regularization parameter in each data-model setup. Second, the effectiveness of an approximate scheme can vary significantly across the different unlearning evaluation criteria, and their trade-offs are not well understood. For example, high 'efficacy' (ability to protect the privacy of the scrubbed data) neither implies nor precludes high 'fidelity' (accuracy on the remaining dataset) [9]. This raises our driving question (Q) below:
(Q) Is there a theoretically-grounded and broadly-applicable method to improve approximate unlearning across different unlearning criteria?
To address (Q), we advance MU through a fresh and novel viewpoint: model sparsification. Our key finding is that model sparsity (achieved by weight pruning) can significantly reduce the gap between approximate unlearning and exact unlearning; see Fig. 1 for the schematic overview of our proposal and highlighted empirical performance.
Model sparsification (or weight pruning) has been extensively studied in the literature [13-19], focusing on the interrelation between model compression and generalization. For example, the notable lottery ticket hypothesis (LTH) [15] demonstrated the existence of a sparse subnetwork (the so-called 'winning ticket') that matches or even exceeds the test accuracy of the original dense model. In addition to generalization, the impact of pruning has also been investigated on model robustness [20-22], fairness [23, 24], interpretability [25, 26], loss landscape [16, 26], and privacy [27, 28]. In particular, the privacy gains from pruning [27, 28] imply connections between data influence and model sparsification.
More recently, a few works [29, 30] attempted to draw insights from pruning for unlearning. In Wang et al. [29], removing channels of a deep neural network (DNN) showed an unlearning ben
eft in federated learning. And in Ye et al. [30], filter pruning was introduced in lifelong learning to detect "pruning identified exemplars" [31] that are easy to forget. However, different from the above literature that customized model pruning for a specific unlearning application, our work systematically and comprehensively explores and exploits the foundational connections between unlearning and pruning. We summarize our contributions below.
Figure 1: Schematic overview of our proposal on model sparsity-driven MU. Evaluation at-a-glance shows the performance of three unlearning methods (retraining-based exact unlearning, finetuning-based approximate unlearning [12], and proposed unlearning on  $95\%$  -sparse model) under five metrics: unlearning accuracy (UA), membership inference attack (MIA)-based unlearning efficacy, accuracy on remaining data (RA), testing accuracy (TA), and run-time efficiency (RTE); see summary in Tab. 1. The unlearning scenario is given by class-wise forgetting, where data points of a single class are scrubbed. Each metric is normalized to  $[0,1]$  based on the best result across unlearning methods for ease of visualization. Results indicate that model sparsity reduces the gap between exact and approximate MU without loss in efficiency.
- First, we provide a holistic understanding of MU across the full training/evaluation stack.
- Second, we draw a tight connection between MU and model pruning and show in theory and practice that model sparsity helps close the gap between approximate unlearning and exact unlearning.
- Third, we develop a new MU paradigm termed 'prune first, then unlearn', and investigate the influence of pruning methods in the performance of unlearning. Additionally, we develop a novel 'sparsity-aware unlearning' framework that leverages a soft sparsity regularization scheme to enhance the approximate unlearning process.
- Finally, we perform extensive experiments across diverse datasets, models, and unlearning scenarios. Our findings consistently highlight the crucial role of model sparsity in enhancing MU.
2 Revisiting Machine Unlearning and Evaluation
Problem setup. MU aims to remove (or scrub) the influence of some targeted training data on a trained ML model [1, 2]. Let  $\mathcal{D} = \{\mathbf{z}_i\}_{i=1}^N$  be a (training) dataset of  $N$  data points, with label
information encoded for supervised learning.  $\mathcal{D}_{\mathrm{f}} \subseteq \mathcal{D}$  represents a subset whose influence we want to scrub, termed the forgetting dataset. Accordingly, the complement of  $\mathcal{D}_{\mathrm{f}}$  is the remaining dataset, i.e.,  $\mathcal{D}_{\mathrm{r}} = \mathcal{D} \setminus \mathcal{D}_{\mathrm{f}}$ . We denote by  $\theta$  the model parameters, and  $\theta_{\mathrm{o}}$  the original model trained on the entire training set  $\mathcal{D}$  using e.g., empirical risk minimization (ERM). Similarly, we denote by  $\theta_{\mathrm{u}}$  an unlearned model, obtained by a scrubbing algorithm, after removing the influence of  $\mathcal{D}_{\mathrm{f}}$  from the trained model  $\theta_{\mathrm{o}}$ . The problem of MU is to find an accurate and efficient scrubbing mechanism to generate  $\theta_{\mathrm{u}}$  from  $\theta_{\mathrm{o}}$ . In existing studies [2, 7, 12], the choice of the forgetting dataset  $\mathcal{D}_{\mathrm{f}}$  specifies different unlearning scenarios. There exist two main categories. First, class-wise forgetting [7, 12] refers to unlearning  $\mathcal{D}_{\mathrm{f}}$  consisting of training data points of an entire class. Second, random data forgetting corresponds to unlearning  $\mathcal{D}_{\mathrm{f}}$  given by a subset of random data drawn from all classes.
Exact and approximate MU methods. The exact unlearning method refers to retraining the model parameters from scratch over the remaining dataset  $\mathcal{D}_{\mathrm{r}}$ . Although retraining from scratch (that we term Retrain) is optimal for MU, it entails a large computational overhead, particularly for DNN training. This problem is alleviated by approximate unlearning, an easy-to-compute proxy for Retrain, which has received growing attention. Yet, the boosted computation efficiency comes at the cost of MU's efficacy. We next review some commonly-used approximate unlearning methods that we improve in the sequel by leveraging sparsity; see a summary in Tab. 1.
$\spadesuit$  Fine-tuning (FT) [6, 12]: Different from Retrain, FT fine-tunes the pre-trained model  $\theta_{\mathrm{o}}$  on  $\mathcal{D}_{\mathrm{r}}$  using a few training epochs to obtain  $\theta_{\mathrm{u}}$ . The rationale is that fine-tuning on  $\mathcal{D}_{\mathrm{r}}$  initiates the catastrophic forgetting in the model over  $\mathcal{D}_{\mathrm{f}}$  as is common in continual learning [33].
$\clubsuit$  Gradient ascent (GA) [7, 8]: GA reverses the model training on  $\mathcal{D}_{\mathrm{f}}$  by adding the corresponding gradients back to  $\theta_{\mathrm{o}}$ , i.e., moving  $\theta_{\mathrm{o}}$  in the direction of increasing loss for data points to be scrubbed.
$\spadesuit$  Fisher forgetting  $(FF)$  [9, 12]: FF adopts an additive Gaussian noise to 'perturb'  $\theta_{\mathrm{o}}$  towards exact unlearning. Here the Gaussian distribution has zero mean and covariance determined by the 4th root of Fisher Information matrix with respect to (w.r.t.)  $\theta_{\mathrm{o}}$  on  $\mathcal{D}_{\mathrm{r}}$ . We note that the computation of the Fisher Information matrix exhibits lower parallel efficiency in contrast to other unlearning methods, resulting in higher computational time when executed on GPUs; see Golatkar et al. [12] for implementation details.
$\spadesuit$  Influence unlearning  $(\mathbf{IU})$  [10, 32]: IU leverages the influence function approach [34] to characterize the change in  $\theta_{\mathrm{o}}$  if a training point is removed from the training loss. IU estimates the change in model parameters from  $\theta_{\mathrm{o}}$  to  $\theta_{\mathrm{u}}$ , i.e.,  $\theta_{\mathrm{u}} - \theta_{\mathrm{o}}$ . IU also relates to an important line of research in MU, known as  $\epsilon-\delta$  forgetting [29, 35, 36]. However, it typically requires additional model and training assumptions [35].
We next take a step further to revisit the IU method and re-derive its formula (Prop. 1), with the aim of enhancing the effectiveness of existing solutions proposed in the previous research.
Proposition 1 Given the weighted ERM training  $\pmb{\theta}(\mathbf{w}) = \arg \min_{\pmb{\theta}} L(\mathbf{w}, \pmb{\theta})$  where  $L(\mathbf{w}, \pmb{\theta}) = \sum_{i=1}^{N} [w_i \ell_i(\pmb{\theta}, \mathbf{z}_i)]$ ,  $w_i \in [0,1]$  is the influence weight associated with the data point  $\mathbf{z}_i$  and  $\mathbf{1}^T \mathbf{w} = 1$ , the model update from  $\pmb{\theta}_{\mathrm{o}}$  to  $\pmb{\theta}(\mathbf{w})$  yields
$$
\Delta (\mathbf {w}) := \boldsymbol {\theta} (\mathbf {w}) - \boldsymbol {\theta} _ {\mathrm {o}} \approx \mathbf {H} ^ {- 1} \nabla_ {\boldsymbol {\theta}} L (\mathbf {1} / N - \mathbf {w}, \boldsymbol {\theta} _ {\mathrm {o}}), \tag {1}
$$
where  $\mathbf{1}$  is the  $N$ -dimensional vector of all ones,  $\mathbf{w} = \mathbf{1} / N$  signifies the uniform weights used by  $ERM$ ,  $\mathbf{H}^{-1}$  is the inverse of the Hessian  $\nabla_{\pmb{\theta},\pmb{\theta}}^2 L(\mathbf{1} / N,\pmb{\theta}_{\mathrm{o}})$  evaluated at  $\pmb{\theta}_{\mathrm{o}}$ , and  $\nabla_{\pmb{\theta}}L$  is the gradient of  $L$ . When scrubbing  $\mathcal{D}_{\mathrm{f}}$ , the unlearned model is given by  $\pmb{\theta}_{\mathrm{u}} = \pmb{\theta}_{\mathrm{o}} + \Delta (\mathbf{w}_{\mathrm{MU}})$ . Here  $\mathbf{w}_{\mathrm{MU}} \in [0,1]^N$  with entries  $w_{\mathrm{MU},i} = \mathbb{I}_{\mathcal{D}_{\mathrm{r}}}(i) / |\mathcal{D}_{\mathrm{r}}|$  signifying the data influence weights for  $MU$ ,  $\mathbb{I}_{\mathcal{D}_{\mathrm{r}}}(i)$  is the indicator function with value 1 if  $i \in \mathcal{D}_{\mathrm{r}}$  and 0 otherwise, and  $|\mathcal{D}_{\mathrm{r}}|$  is the cardinality of  $\mathcal{D}_{\mathrm{r}}$ .
Proof: We derive (1) using an implicit gradient approach; see Appendix A.
It is worth noting that we have taken into consideration the weight normalization effect  $\mathbf{1}^T\mathbf{w} = 1$  in (1). This is different from existing work like Izzo et al. [10, Sec. 3] using Boolean or unbounded
Table 1: Summary of approximate unlearning methods considered in this work. The marker  $\checkmark$  denotes the metric used in previous research. The number in RTE is the run-time cost reduction compared to the cost of Retrain, based on our empirical studies in Sec. 5 on (CIFAR-10, ResNet-18). Note that GA seems better than ours in terms of RTE, but it is less effective in unlearning.
Unlearning Methods	Evaluation metrics	Representative work
UA	MIA-Efficacy	RA	TA	RTE
FT	✓	✓	✓	0.06×	[6, 12]
GA	✓	✓	✓	✓	0.02×	[7, 8]
FF	✓	✓	✓	0.9×	[9, 12]
IU	✓	✓	0.08×	[10, 32]
Ours	✓	✓	✓	✓	0.07×	This work
weights. In practice, we found that IU with weight normalization can improve the unlearning performance. Furthermore, to update the model influence given by (1), one needs to acquire the second-order information in the form of inverse-Hessian gradient product. Yet, the exact computation is prohibitively expensive. To overcome this issue, we use the first-order WoodFisher approximation [37] to estimate the inverse-Hessian gradient product.
Towards a 'full-stack' MU evaluation. Existing work has assessed MU performance from different aspects [7, 8, 12]. Yet, a single performance metric may provide a limited view of MU [11]. By carefully reviewing the prior art, we focus on the following empirical metrics (summarized in Tab. 1).
$\spadesuit$  Unlearning accuracy  $(UA)$ : We define  $UA(\theta_{u}) = 1 - \mathrm{Acc}_{\mathcal{D}_{f}}(\theta_{u})$  to characterize the efficacy of MU in the accuracy dimension, where  $\mathrm{Acc}_{\mathcal{D}_{f}}(\theta_{u})$  is the accuracy of  $\theta_{u}$  on the forgetting dataset  $\mathcal{D}_{\mathrm{f}}$  [7, 12]. It is important to note that a more favorable UA for an approximate unlearning method should reduce its performance disparity with the gold-standard retrained model (Retrain); a higher value is not necessarily better. This principle also extends to other evaluation metrics.
$\spadesuit$  Membership inference attack (MIA) on  $\mathcal{D}_{\mathrm{f}}$  (MIA-Efficacy): This is another metric to assess the efficacy of unlearning. It is achieved by applying the confidence-based MIA predictor [38, 39] to the unlearned model  $(\theta_{\mathrm{u}})$  on the forgetting dataset  $(\mathcal{D}_{\mathrm{f}})$ . The MIA success rate can then indicate how many samples in  $\mathcal{D}_{\mathrm{f}}$  can be correctly predicted as forgetting (i.e., non-training) samples of  $\theta_{\mathrm{u}}$ . A higher MIA-Efficacy implies less information about  $\mathcal{D}_{\mathrm{f}}$  in  $\theta_{\mathrm{u}}$ ; see Appendix C.3 for more details.
$\spadesuit$  Remaining accuracy (RA): This refers to the accuracy of  $\theta_{\mathrm{u}}$  on  $\mathcal{D}_{\mathrm{r}}$ , which reflects the fidelity of MU [9], i.e., training data information should be preserved from  $\theta_{\mathrm{o}}$  to  $\theta_{\mathrm{u}}$ .
$\spadesuit$  Testing accuracy (TA): This measures the generalization ability of  $\theta_{\mathrm{u}}$  on a testing dataset rather than  $\mathcal{D}_{\mathrm{f}}$  and  $\mathcal{D}_{\mathrm{r}}$ . TA is evaluated on the whole test dataset, except for class-wise forgetting, in which testing data points belonging to the forgetting class are not in the testing scope.
$\spadesuit$  Run-time efficiency (RTE): This measures the computation efficiency of an MU method. For example, if we regard the run-time cost of Retrain as the baseline, the computation acceleration gained by different approximate unlearning methods is summarized in Tab. 1.
3 Model Sparsity: A Missing Factor Influencing Machine Unlearning
Model sparsification via weight pruning. Model sparsification could not only facilitate a model's training, inference, and deployment but also benefit model's performance. For example, LTH (lottery ticket hypothesis) [15] stated that a trainable sparse sub-model could be identified from the original dense model, with test accuracy on par or even better than the original model. Fig. 2 shows an example of the pruned model's generalization vs. its sparsity ratio. Here one-shot magnitude pruning (OMP) [17] is adopted to obtain sparse models. OMP is computationally the lightest pruning method, which directly prunes the model weights to the target sparsity ratio based on their magnitudes. As we can see, there exists a graceful sparse regime with lossless testing accuracy.
Gains of MU from sparsity. We first analyze the impact of model sparsity on MU through a lens of unrolling stochastic gradient descent (SGD) [8]. The specified SGD method allows us to derive the unlearning error (given by the weight difference between the approximately unlearned model and the gold-standard retrained model) when scrubbing a single data point. However, different from Thudi et al. [8], we will infuse the model sparsity into SGD unrolling.
Let us assume a binary mask  $\mathbf{m}$  associated with the model parameters  $\theta$ , where  $m_{i} = 0$  signifies that the  $i$ th parameter  $\theta_{i}$  is pruned
to zero and  $m_{i} = 1$  represents the unmasked  $\theta_{i}$ . This sparse pattern  $\mathbf{m}$  could be obtained by a weight pruning method, like OMP. Given  $\mathbf{m}$ , the sparse model is  $\mathbf{m} \odot \boldsymbol{\theta}$ , where  $\odot$  denotes the element-wise multiplication. Thudi et al. [8] showed that if GA is adopted to scrub a single data point for the original (dense) model  $\boldsymbol{\theta}$  (i.e.,  $\mathbf{m} = \mathbf{1}$ ), then the gap between GA and Retrain can be approximately bounded in the weight space. Prop. 2 extends the existing unlearning error analysis to a sparse model.
Figure 2: Testing accuracy of OMP-based sparse ResNet-18 vs. the dense model on CIFAR-10.
Figure 3: Performance of approximate unlearning (FT, GA, FF, IU) and exact unlearning (Retrain) in efficacy (UA and MIA-Efficacy), fidelity (RA), and generalization (TA) vs. model sparsity (achieved by OMP) in the data-model setup (CIFAR-10, ResNet-18). The unlearning scenario is class-wise forgetting, and the average unlearning performance over 10 classes is reported. We remark that being closer to Retrain performance is better for approximate MU schemes.
Proposition 2 Given the model sparse pattern  $\mathbf{m}$  and the SGD-based training, the unlearning error of GA, denoted by  $e(\mathbf{m})$ , can be characterized by the weight distance between the GA-unlearned model and the gold-standard retrained model. This leads to the error bound
$$
e (\mathbf {m}) = \mathcal {O} \left(\eta^ {2} t \| \mathbf {m} \odot \left(\boldsymbol {\theta} _ {t} - \boldsymbol {\theta} _ {0}\right) \| _ {2} \sigma (\mathbf {m})\right) \tag {2}
$$
where  $\mathcal{O}$  is the big- $O$  notation,  $\eta$  is the learning rate,  $t$  is the number of training iterations,  $(\pmb{\theta}_t - \pmb{\theta}_0)$  denotes the weight difference at iteration  $t$  from its initialization  $\pmb{\theta}_0$ , and  $\sigma(\mathbf{m})$  is the largest singular value  $(\sigma)$  of the Hessian  $\nabla_{\pmb{\theta},\pmb{\theta}}^2\ell$  (for a training loss  $\ell$ ) among the unmasked parameter dimensions, i.e.,  $\sigma(\mathbf{m}) := \max_j\{\sigma_j(\nabla_{\pmb{\theta},\pmb{\theta}}^2\ell), if m_j \neq 0\}$ .
Proof: See Appendix B.
We next draw some key insights from Prop. 2. First, it is clear from (2) that the unlearning error reduces as the model sparsity in  $\mathbf{m}$  increases. By contrast, the unlearning error derived in Thudi et al. [8] for a dense model (i.e.,  $\mathbf{m} = \mathbf{1}$ ) is proportional to the dense model distance  $\|\pmb{\theta}_t - \pmb{\theta}_0\|_2$ . Thus, model sparsity is beneficial to reducing the gap between (GA-based) approximate and exact unlearning. Second, the error bound (2) enables us to relate MU to the spectrum of the Hessian of the loss landscape. The number of active singular values (corresponding to nonzero dimensions in  $\mathbf{m}$ ) decreases when the sparsity grows. However, it is important to note that in a high-sparsity regime, the model's generalization could decrease. Consequently, it is crucial to select the model sparsity to strike a balance between generalization and unlearning performance.
Inspired by Prop. 2, we ask: Does the above benefit of model sparsification in MU apply to other approximate unlearning methods besides GA? This drives us to investigate the performance of approximate unlearning across the entire spectrum as depicted in Tab. 1. Therefore, Fig. 3 shows the unlearning efficacy (UA and MIA-Efficacy), fidelity (RA), and generalization (TA) of different approximate unlearning methods in the sparse model regime. Here class-wise forgetting is considered for MU and OMP is used for weight pruning. As we can see, the efficacy of approximate unlearning is significantly improved as the model sparsity increases, e.g., UA and MIA-Efficacy of using FT over  $90\%$  sparsity. By contrast, FT over the dense model ( $0\%$  sparsity) is the least effective for MU. Also, the efficacy gap between exact unlearning (Retrain) and approximate unlearning reduces on sparse models. Further, through the fidelity and generalization lenses, FT and FF yield the RA and TA performance closest to Retrain, compared to other unlearning methods. In the regime of ultra-high sparsity ( $99\%$ ), the efficacy of unlearning exhibits a tradeoff with RA and TA to some extent.
4 Sparsity-Aided Machine Unlearning
Our study in Sec. 3 suggests the new MU paradigm 'prune first, then unlearn', which leverages the fact that (approximate) unlearning on a sparse model yields a smaller unlearning error (Prop. 2) and improves the efficacy of MU (Fig. 3). This promising finding, however, raises some new questions. First, it remains elusive how the choice of a weight pruning method impacts the unlearning performance. Second, it leaves room for developing sparsity-aware MU methods that can directly scrub data influence from a dense model.
Prune first, then unlearn: Choice of pruning methods. There exist many ways to find the desired sparse model in addition to OMP. Examples include pruning at random initialization before training [40, 41] and simultaneous pruning-training iterative magnitude pruning (IMP) [15]. Thus, the problem of pruning method selection arises for MU. From the viewpoint of MU, the unlearner would
prioritize a pruning method that satisfies the following criteria: ① least dependence on the forgetting dataset  $(\mathcal{D}_{\mathrm{f}})$ , ② lossless generalization when pruning, and ③ pruning efficiency. The rationale behind ① is that it is desirable not to incorporate information of  $\mathcal{D}_{\mathrm{f}}$  when seeking a sparse model prior to unlearning. And the criteria ② and ③ ensure that sparsity cannot hamper TA (testing accuracy) and RTE (run-time efficiency). Based on ①-③, we propose to use two pruning methods.