SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\spadesuit$  SynFlow (synaptic flow pruning) [40]: SynFlow provides a (training-free) pruning method at initialization, even without accessing the dataset. Thus, it is uniquely suited for MU to meet the criterion ①. And SynFlow is easy to compute and yields a generalization improvement over many other pruning-at-initialization methods; see justifications in [40].
$\spadesuit$  OMP (one-shot magnitude pruning) [17]: Different from SynFlow, OMP, which we focused on in Sec. 3, is performed over the original model  $(\theta_{\mathrm{o}})$ . It may depend on the forgetting dataset  $(\mathcal{D}_{\mathrm{f}})$ , but has a much weaker dependence compared to IMP-based methods. Moreover, OMP is computationally lightest (i.e. best for  $\Theta$ ) and can yield better generalization than SynFlow [18].
Furthermore, it is important to clarify that IMP (iterative magnitude pruning) is not suitable for MU, despite being widely used to find the most accurate sparse models (i.e., best for criterion 2). Compared with the proposed pruning methods, IMP has the largest computation overhead and the strongest correlation with the training dataset (including  $\mathcal{D}_{\mathrm{f}}$ ), thereby deviating from 1 and 3. In Fig. 4, we show the efficacy of FT-based unlearning on sparse models generated using different pruning methods (SynFlow, OMP, and IMP). As we can see, unlearning on SynFlow or OMP-generated sparse
Figure 4: Influence of different pruning methods (SynFlow, OMP, and IMP) in unlearning efficacy (UA and MIA-Efficacy) and generalization (TA) on (CIFAR-10, ResNet-18). Left: UA vs. TA. Right: MIA-Efficacy vs. TA. Each point is a FT-based unlearned dense or sparse model (75% or 95% sparsity), or a retrained dense model.
models yields improved UA and MIA-Efficacy over that on the original dense model and IMP-generated sparse models. This unlearning improvement over the dense model is consistent with Fig. 3. More interestingly, we find that IMP cannot benefit the unlearning efficacy, although it leads to the best TA. This is because IMP heavily relies on the training set including forgetting data points, which is revealed by the empirical results – the unlearning metrics get worse for IMP with increasing sparsity. Furthermore, when examining the performance of SynFlow and OMP, we observe that the latter generally outperforms the former, exhibiting results that are closer to those of Retrain. Thus, OMP is the pruning method we will use by default.
Sparsity-aware unlearning. We next study if pruning and unlearning can be carried out simultaneously, without requiring prior knowledge of model sparsity. Let  $L_{\mathrm{u}}(\pmb{\theta}; \pmb{\theta}_{\mathrm{o}}, \mathcal{D}_{\mathrm{r}})$  denote the unlearning objective function of model parameters  $\pmb{\theta}$ , given the pre-trained state  $\pmb{\theta}_{\mathrm{o}}$ , and the remaining training dataset  $\mathcal{D}_{\mathrm{r}}$ . Inspired by sparsity-inducing optimization [42], we integrate an  $\ell_1$  norm-based sparse penalty into  $L_{\mathrm{u}}$ . This leads to the problem of '  $\ell_1$ -sparse MU':
$$
\boldsymbol {\theta} _ {\mathrm {u}} = \underset {\boldsymbol {\theta}} {\arg \min } L _ {\mathrm {u}} (\boldsymbol {\theta}; \boldsymbol {\theta} _ {\mathrm {o}}, \mathcal {D} _ {\mathrm {r}}) + \gamma \| \boldsymbol {\theta} \| _ {1}, \tag {3}
$$
where we specify  $L_{\mathrm{u}}$  by the fine-tuning objective, and  $\gamma > 0$  is a regularization parameter that controls the penalty level of the  $\ell_1$  norm, thereby reducing the magnitudes of 'unimportant' weights.
Table 2: MU performance comparison of using  $\ell_1$ -sparse MU with different sparsity schedulers of  $\gamma$  in (3) and using Retrain. The unlearning scenario is given by random data forgetting (10% data points across all classes) on (ResNet-18, CIFAR-10). A performance gap against Retrain is provided in  $(\bullet)$ .
MU	UA	MIA-Efficacy	RA	TA	RTE (min)
Retrain	5.41	13.12	100.00	94.42	42.15
\( \ell_1 \)-sparse MU + constant \( \gamma \)	6.60 (1.19)	14.64 (1.52)	96.51 (3.49)	87.30 (7.12)	2.53
\( \ell_1 \)-sparse MU + linear growing \( \gamma \)	3.80 (1.61)	8.75 (4.37)	97.13 (2.87)	90.63 (3.79)	2.53
\( \ell_1 \)-sparse MU + linear decaying \( \gamma \)	5.35 (0.06)	12.71 (0.41)	97.39 (2.61)	91.26 (3.16)	2.53
In practice, the unlearning performance could be sensitive to the choice of the sparse regularization parameter  $\gamma$ . To address this limitation, we propose the design of a sparse regularization scheduler. Specifically, we explore three schemes: (1) constant  $\gamma$ , (2) linearly growing  $\gamma$  and (3) linearly decaying  $\gamma$ ; see Sec. 5.1 for detailed implementations. Our empirical evaluation presented in Tab. 2
shows that the use of a linearly decreasing  $\gamma$  scheduler outperforms other schemes. This scheduler not only minimizes the gap in unlearning efficacy compared to Retrain, but also improves the preservation of RA and TA after unlearning. These findings suggest that it is advantageous to prioritize promoting sparsity during the early stages of unlearning and then gradually shift the focus towards enhancing fine-tuning accuracy on the remaining dataset  $\mathcal{D}_{\mathrm{r}}$ .
5.1 Experiment setups
Datasets and models. Unless specified otherwise, our experiments will focus on image classification under CIFAR-10 [43] using ResNet-18 [44]. Yet, experiments on additional datasets (CIFAR-100 [43], SVHN [45], and ImageNet [46]) and an alternative model architecture (VGG-16 [47]) can be found in Appendix C.4. Across all the aforementioned datasets and model architectures, our experiments consistently show that model sparsification can effectively reduce the gap between approximate unlearning and exact unlearning.
Unlearning and pruning setups. We focus on two unlearning scenarios mentioned in Sec. 2, class-wise forgetting and random data forgetting (10% of the whole training dataset together with 10 random trials). In the 'prune first, then unlearn' paradigm, we focus on unlearning methods (FT, GA, FF, and IU) shown in Tab. 1 when applying to sparse models. We implement these methods following their official repositories. However, it is worth noting that the implementation of FF in Golatkar et al. [12] modifies the model architecture in class-wise forgetting, i.e., removes the prediction head of the class to be scrubbed. By contrast, other methods keep the model architecture intact during unlearning. Also, we choose OMP as the default pruning method, as justified in Fig. 4. In the 'sparsity-aware unlearning' paradigm, the sparsity-promoting regularization parameter  $\gamma$  in (3) is determined through the line search in the interval  $[10^{-5}, 10^{-1}]$ , with consideration for the trade-off between testing accuracy and unlearning accuracy. For all schedulers,  $\gamma$  is set around to  $5 \times 10^{-4}$ . The linearly increasing and decaying schedulers are implemented as  $\gamma_{t} = \frac{2t}{T}\gamma$  and  $\gamma_{t} = (2 - \frac{2t}{T})\gamma$  respectively, where  $t$  is the epoch index and  $T$  is the total number of epochs. We refer readers to Appendix C.2 for more details.
Evaluation metrics. We evaluate the unlearning performance following Tab. 1. Recall that UA and MIA-Efficacy depict the efficacy of MU, RA reflects the fidelity of MU, and TA and RTE characterize the generalization ability and the computation efficiency of an unlearning method. We implement MIA (membership inference attack) using the prediction confidence-based attack method [38, 39], whose effectiveness has been justified in Song and Mittal [48] compared to other methods. We refer readers to Appendix C.3 for more implementation details. To more precisely gauge the proximity of each approximate MU to Retrain, we introduce a metric termed 'Disparity Average'. This metric quantifies the mean performance gap between each unlearning method and Retrain across all considered metrics. A lower value indicates closer performance to Retrain.
Table 3: Performance overview of various MU methods on dense and  $95\%$  -sparse models considering different unlearning scenarios: class-wise forgetting, and random data forgetting. The forgetting data of random data forgetting ratio is  $10\%$  of the whole training dataset, the sparse models are obtained using OMP [17], and the unlearning methods and evaluation metrics are summarized in Tab. 1. Class-wise forgetting is conducted class-wise. The performance is reported in the form  $a_{\pm b}$ , with mean  $a$  and standard deviation  $b$  computed over 10 independent trials. A performance gap against Retrain is provided in  $(\bullet)$ . Note that the better performance of approximate unlearning corresponds to the smaller performance gap with the gold-standard retrained model. 'Disparity Ave.' represents the average unlearning gaps across diverse metrics.
MU	UA	MIA-Efficacy	RA	TA	Disparity Ave. ↓	RTE
DENSE	95% Sparsity	DENSE	95% Sparsity	DENSE	95% Sparsity	DENSE	95% Sparsity	DENSE	95% Sparsity	(min)
Class-wise forgetting
Retrain	100.00±0.00	100.00±0.00	100.00±0.00	100.00±0.00	100.00±0.00	99.99±0.01	94.83±0.11	91.80±0.80	0.00	0.00	43.23
FT	22.53±8.16 (77.47)	73.64±9.46 (26.36)	75.00±14.68 (25.00)	83.02±16.33 (16.98)	99.87±0.04 (0.13)	99.87±0.05 (0.12)	94.31±0.19 (0.52)	94.32±0.12 (2.52)	25.78	11.50	2.52
GA	93.08±2.29 (6.92)	98.09±1.11 (1.91)	94.03±3.27 (5.97)	97.74±2.24 (2.26)	92.60±0.25 (7.40)	87.74±0.27 (12.25)	86.64±0.28 (8.19)	82.58±0.27 (9.22)	7.12	6.41	0.33
FF	79.93±8.92 (20.07)	94.83±4.29 (5.17)	100.00±0.00 (0.00)	100.00±0.00 (0.00)	99.45±0.24 (5.55)	99.45±0.33 (0.51)	94.18±0.08 (0.65)	94.04±0.10 (2.24)	5.32	1.98	38.91
IU	87.82±2.15 (12.18)	99.47±0.15 (0.53)	95.96±0.21 (4.04)	99.93±0.04 (0.07)	97.98±0.21 (2.02)	97.24±0.13 (2.75)	91.42±0.21 (3.41)	90.76±0.18 (1.04)	5.41	1.10	3.25
Random data forgetting
Retrain	5.41±0.11	6.77±0.23	13.12±0.14	14.17±0.18	100.00±0.00	100.00±0.00	94.42±0.09	93.33±0.12	0.00	0.00	42.15
FT	6.83±0.51 (1.42)	5.97±0.57 (0.80)	14.97±0.62 (1.85)	13.30±0.59 (0.81)	96.61±0.25 (3.39)	96.99±0.31 (3.01)	90.13±0.26 (4.29)	90.29±0.31 (3.04)	2.74	1.92	2.33
GA	7.54±0.29 (2.13)	5.62±0.46 (1.15)	10.04±0.31 (3.08)	11.70±0.32 (2.41)	93.31±0.04 (6.69)	95.44±0.11 (4.56)	89.28±0.07 (5.14)	89.26±0.15 (4.07)	4.26	3.05	0.31
FF	7.84±0.71 (2.43)	8.10±0.67 (1.39)	9.52±0.43 (3.60)	10.80±0.37 (3.37)	92.05±0.16 (7.95)	92.29±0.24 (7.71)	88.10±0.19 (6.32)	87.79±0.23 (5.54)	5.08	4.50	38.24
IU	2.03±0.43 (3.38)	6.51±0.52 (0.26)	5.07±0.74 (8.05)	11.93±0.68 (2.24)	98.26±0.29 (1.74)	94.94±0.31 (5.06)	91.33±0.22 (3.09)	88.74±0.42 (4.59)	4.07	3.08	3.22
Model sparsity improves approximate unlearning. In Tab.3, we study the impact of model sparsity on the performance of various MU methods in the 'prune first, then unlearn' paradigm. The performance of the exact unlearning method (Retrain) is also provided for comparison. Note that the better performance of approximate unlearning corresponds to the smaller performance gap with the gold-standard retrained model.
First, given an approximate unlearning method (FT, GA, FF, or IU), we consistently observe that model sparsity improves UA and MIA-Efficacy (i.e., the efficacy of approximate unlearning) without much performance loss in RA (i.e., fidelity). In particular, the performance gap between each approximate unlearning method and Retrain reduces as the model becomes sparser (see the '95% sparsity' column vs. the 'dense' column). Note that the performance gap against Retrain is highlighted in  $(\cdot)$  for each approximate unlearning. We also observe that Retrain on the 95%-sparsity model encounters a 3% TA drop. Yet, from the perspective of approximate unlearning, this drop brings in a more significant improvement in UA and MIA-Efficacy when model sparsity is promoted. Let us take FT (the simplest unlearning method) for class-wise forgetting as an example. As the model sparsity reaches 95%, we obtain 51% UA improvement and 8% MIA-Efficacy improvement. Furthermore, FT and IU on the 95%-sparsity model can better preserve TA compared to other methods. Table 3 further indicates that sparsity reduces average disparity compared to a dense model across various approximate MU methods and unlearning scenarios.
Second, existing approximate unlearning methods have different pros and cons. Let us focus on the regime of  $95\%$  sparsity. We observe that FT typically yields the best RA and TA, which has a tradeoff with its unlearning efficacy (UA and MIA-Efficacy). Moreover, GA yields the worst RA since it is most loosely connected with the remaining dataset  $\mathcal{D}_{\mathrm{r}}$ . FF becomes ineffective when scrubbing random data points compared to its class-wise unlearning performance. Furthermore, IU causes a TA drop but yields the smallest gap with exact unlearning across diverse metrics under the  $95\%$  model sparsity. In Appendix C.4, we provide additional results on CIFAR-100 and SVHN datasets, as shown in Tab. A3, as well as on the ImageNet dataset, depicted in Tab. A5. Other results pertaining to the VGG-16 architecture are provided in Tab. A4.
Figure 5: Performance of sparsity-aware unlearning vs. FT and Retrain on class-wise forgetting and random data forgetting under (CIFAR10, ResNet-18). Each metric is normalized to [0, 1] based on the best result across unlearning methods for ease of visualization, while the actual best value is provided (e.g., 2.52 is the least computation time for class-wise forgetting).
Effectiveness of sparsity-aware unlearning. In Fig. 5, computation time for class-wise forgetting). we showcase the effectiveness of the proposed sparsity-aware unlearning method, i.e.,  $\ell_1$ -sparse MU. For ease of presentation, we focus on the comparison with FT and the optimal Retrain strategy in both class-wise forgetting and random data forgetting scenarios under (CIFAR-10, ResNet-18). As we can see,  $\ell_1$ -sparse MU outperforms FT in the unlearning efficacy (UA and MIA-Efficacy), and closes the performance gap with Retrain without losing the computation advantage of approximate unlearning. We refer readers to Appendix C.4 and Fig. A2 for further exploration of  $\ell_1$ -sparse MU on additional datasets.
Application: MU for Trojan model cleanse. We next present an application of MU to remove the influence of poisoned backdoor data from a learned model, following the backdoor attack setup [49], where an adversary manipulates a small portion of training data (a.k.a. poisoning ratio) by injecting a backdoor trigger (e.g., a small image patch) and modifying data labels towards a targeted incorrect label. The trained model is called Trojan model, yielding the backdoor-designated incorrect prediction if the trigger is present at testing.
Figure 6: Performance of Trojan model cleansse via proposed unlearning vs. model sparsity, where 'Original' refers to the original Trojan model. Left: ASR vs. model sparsity. Right: SA vs. model sparsity.
Otherwise, it behaves normally.
We then regard MU as a defensive method to scrub the harmful influence of poisoned training data in the model's prediction, with a similar motivation as Liu et al. [50]. We evaluate the performance of the unlearned model from two perspectives, backdoor attack success rate (ASR) and standard accuracy (SA). Fig. 6 shows ASR and SA of the Trojan model (with poisoning ratio  $10\%$ ) and its unlearned version using the simplest FT method against model sparsity. Fig. 6 also includes the  $\ell_1$ -sparse MU to demonstrate its effectiveness on model clease. Since it is applied to a dense model (without using hard thresholding to force weight sparsity), it contributes just a single data point at the sparsity level  $0\%$ . As we can see, the original Trojan model maintains  $100\%$  ASR and a similar SA across different model sparsity levels. By contrast, FT-based unlearning can reduce ASR without inducing much SA loss. Such a defensive advantage becomes more significant when sparsity reaches  $90\%$ . Besides,  $\ell_1$ -sparse MU can also effectively remove the backdoor effect while largely preserving the model's generalization. Thus, our proposed unlearning shows promise in application of backdoor attack defense.
Application: MU to improve transfer learning. Further, we utilize the  $\ell_1$ -sparse MU method to mitigate the impact of harmful data classes of ImageNet on transfer learning. This approach is inspired by Jain et al. [51], which shows that removing specific negatively-influenced ImageNet classes and retraining a source model can enhance its transfer learning accuracy on downstream datasets after finetuning. However, retraining the source model introduces additional computational overhead. MU naturally addresses this limitation and offers a solution.
Tab. 4 illustrates the transfer learning accuracy of the unlearned or retrained source model (ResNet-18) on ImageNet, with  $n$  classes removed. The downstream target datasets used for evaluation are SUN397 [52] and OxfordPets [53]. The employed finetuning approach is linear probing, which finetunes the classification head of the source model on target datasets while keeping the feature extraction network of the source model intact. As we can see, removing data classes from the source ImageNet dataset can lead to improved transfer learning accuracy compared to the conventional method of using the pre-trained model on the full ImageNet (i.e.,  $n = 0$ ). Moreover, our proposed
$\ell_{1}$ -sparse MU method achieves comparable or even slightly better transfer learning accuracy than the retraining-based approach [51]. Importantly,  $\ell_{1}$ -sparse MU offers the advantage of computational efficiency  $2\times$  speed up over previous method [51] across all cases, making it an appealing choice for transfer learning using large-scale models. Here we remark that in order to align with previous method [51], we employed a fast-forward computer vision training pipeline (FFCV) [54] to accelerate our ImageNet training on GPUs.
Additional results. We found that model sparsity also enhances the privacy of the unlearned model, as evidenced by a lower MIA-Privacy. Refer to Appendix C.4 and Fig. A1 for more results. In addition,
Table 4: Transfer learning accuracy (Acc) and computation time (mins) of the unlearned ImageNet model with  $n \in  \{ {100},{200},{300}\}$  classes removed, where SUN397 and OxfordPets are downstream target datasets on linear probing transfer learning setting. When  $n = 0$  ,transfer learning is performed using the pretrained model on the full ImageNet,serving as a baseline,together with the method in [51] for comparison.
Forgetting class #	0 Acc	100 Acc	Time	200 Acc	Time	300 Acc	Time
OxfordPets
Method [51]	85.70	85.79	71.84	86.10	61.53	86.32	54.53
\( \ell_1 \)-sparse MU	85.83	35.47	86.12	30.19	86.26	26.49
SUN397
Method [51]	46.55	46.97	73.26	47.14	61.43	47.31	55.24
\( \ell_1 \)-sparse MU	47.20	36.69	47.25	30.96	47.37	27.12
we have expanded our experimental scope to encompass the 'prune first, then unlearn' approach across various datasets and architectures. The results can be found in Tab. A3, Tab. A4, and Tab. A5. Furthermore, we conducted experiments on the  $\ell_1$ -sparse MU across different datasets, the Swin-Transformer architecture, and varying model sizes within the ResNet family. The corresponding findings are presented in Fig. A2 and Tab. A6, A7, A8 and A9.
While Sec. 2 provides a summary of related works concerning exact and approximate unlearning methods and metrics, a more comprehensive review is provided below.
Machine unlearning. In addition to exact and approximate unlearning methods as we have reviewed in Sec. 2, there exists other literature aiming to develop the probabilistic notion of unlearning [35, 55-58], in particular through the lens of differential privacy (DP) [59]. Although DP enables unlearning with provable error guarantees, they typically require strong model and algorithmic assumptions and could lack effectiveness when facing practical adversaries, e.g., membership inference attacks. Indeed, evaluating MU is far from trivial [8, 9, 11]. Furthermore, the attention on MU has also been raised
in different learning paradigms, e.g., federated learning [29, 60], graph neural networks [61-63], and adversarial ML [64, 65]. In addition to preventing the leakage of data privacy from the trained models, the concept of MU has also inspired other emergent applications such as adversarial defense against backdoor attacks [6, 50] that we have studied and erasing image concepts of conditional generative models [66, 67].
Understanding data influence. The majority of MU studies are motivated by data privacy. Yet, they also closely relate to another line of research on understanding data influence in ML. For example, the influence function approach [32] has been used as an algorithmic backbone of many unlearning methods [6, 10]. From the viewpoint of data influence, MU has been used in the use case of adversarial defense against data poisoning backdoor attacks [50]. Beyond unlearning, evaluation of data influence has also been studied in fair learning [68, 69], transfer learning [51], and dataset pruning [70, 71].
Model pruning. The deployment constraints on e.g., computation, energy, and memory necessitate the pruning of today's ML models, i.e., promoting their weight sparsity. The vast majority of existing works [13-19] focus on developing model pruning methods that can strike a graceful balance between model's generalization and sparsity. In particular, the existence of LTH (lottery ticket hypothesis) [15] demonstrates the feasibility of co-improving the model's generalization and efficiency (in terms of sparsity) [40, 72-75]. In addition to generalization, model sparsity achieved by pruning can also be leveraged to improve other performance metrics, such as robustness [20-22], model explanation [25, 26], and privacy [27, 28, 76, 77].
In this work, we advance the method of machine unlearning through a novel viewpoint: model sparsification, achieved by weight pruning. We show in both theory and practice that model sparsity plays a foundational and crucial role in closing the gap between exact unlearning and existing approximate unlearning methods. Inspired by that, we propose two new unlearning paradigms, 'prune first, then unlearn' and 'sparsity-aware unlearn', which can significantly improve the efficacy of approximate unlearning. We demonstrate the effectiveness of our findings and proposals in extensive experiments across different unlearning setups. Our study also indicates the presence of model modularity traits, such as weight sparsity, that could simplify the process of machine unlearning. This may open up exciting prospects for future research to investigate unlearning patterns within weight or architecture space.
The work of J. Jia, J. Liu, Y. Yao, and S. Liu were supported by the Cisco Research Award and partially supported by the NSF Grant IIS-2207052, and the ARO Award W911NF2310343. Y. Liu was partially supported by NSF Grant IIS-2143895 and IIS-2040800.
[1] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pages 463-480. IEEE, 2015.
[2] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159. IEEE, 2021.
[3] Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint arXiv:2209.02299, 2022.
[4] Jeffrey Rosen. The right to be forgotten. Stan. L. Rev. Online, 64:88, 2011.
[5] Chris Jay Hoofnagle, Bart van der Sloot, and Frederik Zuiderveen Borgesius. The european union general data protection regulation: what it is and what it means. Information & Communications Technology Law, 28(1):65-98, 2019.
Recap the definition of model update  $\Delta (\mathbf{w})$  in (1) and  $\theta_{\mathrm{o}} = \theta (\mathbf{1} / N)$ , we approximate  $\Delta (\mathbf{w})$  by the first-order Taylor expansion of  $\theta (\mathbf{w})$  at  $\mathbf{w} = \mathbf{1} / N$ . This leads to