Model Sparsity Can Simplify Machine Unlearning
Authors: Jinghan Jia
Jiancheng Liu
Parikshit Ram
Yuguang Yao
Gaowen Liu
Yang Liu
Pranay Sharma
Sijia Liu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Machine unlearning
model pruning

Abstract: In response to recent data regulation requirements, machine unlearning (MU) has emerged as a critical process to remove the influence of specific examples from a given model. Although exact unlearning can be achieved through complete model retraining using the remaining dataset, the associated computational costs have driven the development of efficient, approximate unlearning techniques. Moving beyond data-centric MU approaches, our study introduces a novel model-based perspective: model sparsification via weight pruning, which is capable of reducing the gap between exact unlearning and approximate unlearning. We show in both theory and practice that model sparsity can boost the multi-criteria unlearning performance of an approximate unlearner, closing the approximation gap, while continuing to be efficient. This leads to a new MU paradigm,    termed prune first, then unlearn, which infuses a sparse prior to the unlearning process. Building on this insight, we also develop a sparsity-aware unlearning method that utilizes sparsity regularization to enhance the training process of approximate unlearning. Extensive experiments show that our proposals consistently benefit MU in various unlearning scenarios. A notable highlight is the 77% unlearning efficacy gain of fine-tuning (one of the simplest approximate unlearning methods) when using our proposed sparsity-aware unlearning method. Furthermore, we showcase the practical impact of our proposed MU methods through two specific use cases: defending against backdoor attacks, and enhancing transfer learning through source class removal. These applications demonstrate the versatility and effectiveness of our approaches in addressing a variety of machine learning challenges beyond unlearning for data privacy. Codes are available at https://github.com/OPTML-Group/Unlearn-Sparse.

Model Sparsity Can Simplify Machine Unlearning
Jinghan Jia
Jiancheng Liu
Parikshit Ram
Yuguang Yao
Gaowen Liu
Yang Liu
Pranay Sharma
Sijia Liu
Michigan State University, 2BB Research, &Cisco Research, University of California, Santa Cruz, Bayteance Research, CCrnegie Mellon University *Equal contribution
Abstract
response to recent data regulation requirements, machine unlearning (MU) has emerged as critical process to remove the influence of specific examples from a given model. Although exact unlearning can be achieved through complete model retraining using the remaining dataset, the associated computational costs have driven the development of efficient, approximate unlearning techniques. Moving beyond data-centric MU approaches, our study introduces novel model-based perspective: model sparsification via weight pruning, which is capable of reducing the gap between exact unlearning and approximate unlearning. We show in both theory and practice that model sparsity can boost the multi-criteria unlearning performance of approximate unlearner, closing the approximation gap, while continuing to be efficient. This leads to a new MU paradigm, termed prune first, then unlearn, which infuses sparse model prior into the unlearning process. Building on this insight, we also develop a sparsity-aware unlearning method that utilizes sparsity regularization enhance the training process of approximate unlearning. Extensive experiments show that our proposals consistently benefit MU in various unlearning scenarios. notable highlight is the 77% unlearning efficacy gain of fine-tuning (one of the simplest unlearning methods) when using sparsity- unlearning. Furthermore, we demonstrate the practical impact of our proposed MU methods in addressing other machine learning challenges, such as defending against backdoor attacks and enhancing transfer learning. Codes are available at https::/// / thub. com/ Coom/OPTML--T http///OMGGoooo////eeee Caroou//nneeannSS51 -Sarsse.
1 Introduction
Machine unlearning (MU) initiates a reverse learning process to scrub the influence of data points from trained machine learning (ML) model. was introduced to avoid information leakage about private data upon completion of training [1-3], particularly in compliance with legislation like 'the right be forgotten' [4] in General Data Protection Regulation (GDPR) [5]. The direct but optimal unlearning approach is exact unlearning to retrain ML models from scratch using the remaining training set, after removing the data points be scrubbed. Although retraining yields the ground-trath unlearning strategy, it is the most computationally intensive one. Therefore, the development of approximate but fast unlearning methods has become major focus research [6-10].
Despite the computational benefits of approximate unlearning, it often lacks strong guarantee on the effectiveness of unlearning, resulting in performance gap with exact unlearning [11]. In particular, we encounter two main challenges. First, the performance of approximate unlearning can heavily rely on the configuration of algorithmic parameters. For example, the Fisher forgetting method [12] needs
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

to carefully tune the Fisher information regularization parameter in each data-model setup. Second, the effectiveness of an approximate scheme can vary significantly across the different unlearning evaluation criteria, and their trade-offss are not well understood. For example, high 'efficacy' (ability to protect the privacy of the scrubbed data) neither implies nor precludes high 'fidelity" (aceuracy on the remaining dataset) [9]. This raises our driving question (Q) below:
(Q) Is there a theoreticallygroumdedd and aepadly-applicatll method to improve approximate unlearning across different unlearning criteria?
To address (Q), we advance MU through a fresh and novel viewpoint: model sparsification. Our key finding is that model sparsity (achieved by weight pruning) can significantly reduce the gap between approximate unlearning and exact unlearning; see Fig. for the schematic overview of our proposal and highlighted empirical performance.
Model sparsification (or weight prun- ing) has been extensively studied in the literature [13-19], focusing on the in- terrelation between model compression and generalization. For example, the notable lottery ticket hypothesis (LTH) [15] demonstrated the existence of a sparse subnetwork (the so-called 'win- ning ticket') that matches or even ex- ceeds the test accuracy of the original dense model. In addition to general- ization, the impact of pruning has also been investigated on model robustness [20-22], fairness [23, 24], interpretabil- ity [25, 26), loss landscape [16, 26), and privacy [27, 28]. In particular, the pri- vacy gains from pruning [27, 28] imply connections between data influence and model sparsification.
Data
Machine Unlearning with Sparsity
MIA (%)
RIE miin)
(A
Uniaamn
Daaet Uakerrs
Approciman Unkera (Dmun) Approcinane Unkera (9%% Spares)
Algorithm
Evaluation
Figure Schematic overview our proposal on model sparsity- driven MU. Evaluation at-a-glance shows the performance of three unlearning methods (retraining -ased exact unlearning, finetuning -basd approximate unlearning [12]. and proposed unlearning on 95%-sparse model) under five metries: unlearn- ing accuracy (UA), membership inference attack (MIA)-based unlcarning efficacy, accuracy on remaining data (RA), testing accuracy (TA), and run-time efficiency (RTE); see summary in Tab. 1. The unlearning scenario given class-wise forgetting, where data points a single class are scrubbed. Each metric normalized to [0, 1] based on the best result across unlearning methods for case of visualization. Results indicate that model sparsity reduces the gap between exact and approximate MU without loss in efficiency.
More recently, few works [29, 30] at- tempted to draw insights from pruning for unlearning. In Wang et al. [29], re- moving channels of a deep neural net- work (DNN) showed an unlearning ben-
efit in federated learning. And in Ye et al. [30], filter pruning was introduced in lifelong learning to detect "pruning identified exemplars" [31] that are easy to forget. However, different from the above literature that customized model pruning for a specific unlearning application, our work systematically and comprehensively explores and exploits the foundational connections between unlearning and pruning. We summarize our contributions below.
First, we provide a holistic understanding MU across the full training/evaluation stack.
Second, we draw a tight connection between MU and model pruning and show in theory and practice that model sparsity helps close the gap between approximate unlearning and exact unlearning.
Third, we develop a new MU paradigm termed 'prune first, then unlearn', and investigate the influence of pruning methods in the performance of unlearning. Additionally, we develop novel sparsity--ware unlearning' framework that leverages a soft sparsity regularization scheme to enhance the approximate unlearning process.
Finally, we perform extensive experiments across diverse datasets, models, and unlearning scenarios. Our findings consistently highlight the crucial role of model sparsity in enhancing MU.
2 Revisiting Machine Unlearning and Evaluation
Problem setup. MU aims to remove (or scrub) the influence of some targeted training data on trained ML model [1. 2]. Let D J.N be (training) dataset of N data points, with label

---

information encoded for supervised learning. D; S D represents subset whose influence we want to scrub, termed the forgetting dataset. Accordingly, the complement of D; is the remaining dataset, Le.. D. = D Dr. We denote by 0 the model parameters, and O. the original model trained on the entire training set 'D using e.g., empirical risk minimization (ERM). Similarly, we denote by 0, an unlearned model, obtained by scrubbing algorithm, after removing the influence of 'Dr from the trained model 0.. The problem of MU is to find an accurate and efficient scrubbing mechanism to generate On from Oo. In existing studies [2, 7, 12], the choice of the forgetting dataset Dj specifies different unlearning scenarios. There exist two main categories. First, class-wise fuass-iiseoorgettnnn [7, 12] refers to unlearning Dj consisting of training data points of an entire class. Second, random data forgetting corresponds to unlearning Dr given by a subset of random data drawn from all classes.
Exact and approximate MU methods. The exact unlearning method refers to retraining the model parameters from scratch over the remaining dataset D, Although retraining from scratch (that we term Retrain) is optimal for MU, it entails large computational overhead, particularly for DNN training. This problem alleviated by approximate unlearming, an easy-to-comppe proxy for Retrain, which has received growing attention. Yet, the boosted computation efficiency comes at the cost of MU's efficacy. We next review some commonly-used approximate unlearning methods that we improve in the sequel by leveraging sparsity; see summary in Tab. 1.
Different from Retrain, FT fine-tunes the pre-trained model 0, on D, using few training epochs to obtain 0n.
Fine-tuning (FT) [6, 12]:
Table I: Summary of approximate unlcarning methods considered in this work. The marker 'V" denotes the met- ric used in previous rescarch. The number in RTE is the run-time cost reduction compared to the cost of Retrain, based on our empirical studies in Sec. 5 (CIFAR-1O. ResNet- 18). Note that GA seems better than ours in terms of RTE, but it is less effective in unlearning.
The rationale is that fine-tuning on DD initiates the catastrophic forgetting in the model over Dr as is common in continual learning [33].
Gradient ascent (GA) [7, 8]: GA reverses the model training on Dr by adding the correspond- ing gradients back 0,, i.e., moving O, in the direction of increasing loss for data points to be scrubbed.
Unkanning Methods
UA
MIA-Efficacy RA
Evalaatio metrics
TA
RTE
Representative work
FT
/
0.6xx
16, 12]
GA
/
0.00
[7.8]
FF
/
0. x
[9. 12]
IL
0.08%
[10, 32]
Ours
0.00
This work
Fisher forgetting (FF) [9, 12]: FF adopts an additive Gaussian noise to 'perturb' 0. towards exact unlearning. Here the Gaussian distribution has zero mean and covariance determined by the 4th root of Fisher Information matrix with respect to (w.r.t)) A. on 'D,- We note that the computation of the Fisher Information matrix exhibits lower parallel efficiency in contrast other unlearning methods, resulting in higher computational time when executed on GPUs; see Golatkar et al. [12] for implementation details.
Influence unlearning (IU) [10, 32]: IU leverages the influence function approach [34] to character- ize the change in 0 if training point is removed from the training loss. IU estimates the change in model parameters from 0, to o.. i.e. 0, - 0.. IU also relates to an important line of research in MU, known e-ô forgetting [29, 35, 36). However, it typically requires additional model and training assumptions [35].
We next take step further to revisit the IU method and re-derive its formula (Prop. 1), with the aim of enhancing the effectiveness of existing solutions proposed in the previous research.
Proposition 1 Given the weighted ERM training 0(w) =
arg ming L(w, O) where L(w, O) =
[w,e: 2.)l. w; E [0, 1] is the influence weight associated with the data point z; and 1" w 1,
the model update from Og to 0 (w) yields
A(w) = 0(w)- - O. ~ H H
(1)
where is the N-dimensional vector of all ones, w = 1 I/N signifies the uniform weights used by ERM, H-1 is the inverse of the Hessian Vo.0 T. O.) evaluated at O., and V is the gradient of L.
When scrubbing D:, the unlearned model is given by e, = 6, + A A(waau). Here WMU E [0, 1 with
entries WMU, = ID. (D) signifying the data influence weights for MU, Lp, (1) is the indicator function with value 1 if E D, and 0 otherwise. and [Dr] is the cardinality of Dr,
Proof:
We derive (1) using implicit gradient approach; see Appendix A.
is worth noting that we have taken into consideration the weight normalization effect 1 w = in (1). This is different from existing work like Izzo et al. [10, Sec. 3] using Boolean or unbounded
3

---

weights. In practice, we found that IU with weight normalization can improve the unlearning performance. Furthermore, to update the model influence given by (1), one needs acquire the second-order information in the form of inverse-Hessian gradient product. Yet, the exact computation prohibitively expensive. To overcome this issue, we use the first-order WoodFisher approximation [37] to estimate the inverse-Hessian gradient product.
Towards 'full-stack' MU evaluation. Existing work has assessed MU performance from different aspects [7, 8, 12]. Yet, a single performance metric may provide a limited view of MU [1]]. By carefully reviewing the prior art, we focus on the following empirical metrics (summarized in Tab. 1).
Unlearning accuracy (UA): We define UA(O.) = 1 Accp, (0.) to characterize the efficacy of MU in the accuracy dimension, where Accp, (0.) is the accuracy of 0u on the forgetting dataset Dr [7, 12]. It is important to note that a more favorable UA for an approximate unlearning method should reduce its performance disparity with the gold-standard retrained model (Retrain); higher value is not necessarily better. This principle also extends to other evaluation metrics.
Membership inference attack (MIA) on Dj (MIA-Efficacy); This is another metric to assess the efficacy of unlearning. It is achieved by applying the confidence-basee MIA predictor [38, 39] to the unlearned model (0.)) on the forgetting dataset (Dt). The MIA success rate can then indicate how many samples in Dj can be correctly predicted as forgetting (i.e., non-training) samples of 0,. A higher MIA-Efficacy implies less information about D: in ou see Appendix C.3 for more details.
Remaining accuracy (RA): This refers to the accuracy 0n on D,, which reflects the fidelity of MU [9], ie., training data information should be preserved from 0, to 0,.
Testing accuracy )TA): This measures the generalization ability of 0, on a testing dataset rather than Dj and D.. TA is evaluated on the whole test dataset, except for class-wise forgetting, in which lesting data points belonging to the forgetting class are not in the testing scope.
Run-time efficiency (RTE): This measures the computation efficiency of an MU method. For example, if regard the run-time cost of Retrain as the baseline, the computation acceleration gained by different approximate unlearning methods is summarized in Tab. 1.
3 Model Sparsity: A Missing Factor Influencing Machine Unlearning
Model sparsification via weight pruning. Model sparsification could not only facilitate a model's training. inference, and deployment but also benefit model's 's performance. For example, LTH (lottery ticket hypothesis) [15] stated that a trainable sparse sub-model could be identified from the original dense model, with test accuracy on par even better than the original model. Fig. 2 shows an example of the pruned model's s generalization vs. its sparsity ratio. Here one-shot magnitude pruning (OMP) [17] is adopted to obtain sparse models. OMP is computationally the lightest pruning method, which directly prunes the model weights to the target sparsity ratio based on their magnitudes. As we can see, there exists graceful sparse regime with lossless testing accuracy.
Gains of MU from sparsity. We first analyze the impact of model sparsity on MU through a lens of unrolling stochastic gradient descent (SGD) [8]. The specified SGD method allows us to derive the unlearning error (given by the weight difference between the approximately unlearned model and the gold-standard retrained model) when scrubbing single data point. However, different from Thudi et al. [8], we will infuse the model sparsity into SGD unrolling.
95.0
Terting Accuracy (%6)
94.7
94.5
Dense Model OMP
94.24
94.0g 1
80
Spesity (5)
Testing accuracy of OMP- based sparse ResNet-18 vs. the dense model on C1FAR-10.
Figure 2:
Let us assume a binary mask m associated with the model param- eters 0, where m. 0 signifies that the ith parameter 0; is pruned to zero and m, = represents the unmasked 0,. This sparse pattern m could be obtained by a weight pruning method, like OMP. Given m, the sparse model is m () 0, where ( denotes the element-wise multiplication. Thudi et al. [8] showed that if GA is adopted to scrub a single data point for the original (dense) model 0 (i.e., m = 1), then the gap between GA and Retrain can be approximately bounded in the weight space. Prop. 2 extends the existing unlearning error analysis to a sparse model.

---

03
Uatcmmin
00
5
MIN
15
23
Spaniy Figure 3: Performance approximate unlearning (FT, GA, FF, IU) exact unlearning (Retrain) efficacy (UA and MIA-Efficacy), fidelity (RA), and generalization (TA) vs. model sparsity (achieved by OMP) in the data-model setup (CIFAR-IK) 10, ResNet-18). The unlearning scenario is class-wise forgetting, and the average unlearning performance over 10 classes is reported. We remark that being closer Retrain performance better for approximate MU schemes.
Spandry
Proposition 2 Given the model sparse pattern m and the SGD-based training, the unlearning error of GA, denoted by e(m), can be characterized by the weight distance between the GÅ iA-uuleareed model and the gold-standard retrained model. This leads to the error bound
e(m) (0n/ttllm
(000)ll)2((m)))
(2)
where O is the big-O notation, y) is the learning rate, t is the number of training iterations, (0; Ob) denotes the weight difference at iteration from its initialization 0,, and o (m)) is the largest singular value (a) of the Hessian V². ol (for a training loss {) among the unmasked parameter dimensions, Le., o (m) == max, {o (V )) if m, # #0}.
Proof: See Appendix B.
We next draw some key insights from Prop. 2. First, it is clear from (2) that the unlearning error reduces the model sparsity in increases. By contrast, the unlearning error derived in Thudi et al. [8] for a dense model (ie., m = 1) is proportional to the dense model distance ||0 - 0 2 Thus, model sparsity is beneficial reducing the gap between (GA-based) approximate and exact unlearning. Second, the error bound (2) enables us to relate MU to the spectrum of the Hessian the loss landscape. The number of active singular values (corresponding to nonzero dimensions in m) decreases when the sparsity grows. However, is important to note that in a high-sparsity regime, the model's generalization could decrease. Consequently, it is crucial select the model sparsity to strike a balance between generalization and unlearning performance.
Inspired by Prop. 2, we ask: Does the above benefit of model sparsification in MU apply other approximate unlearning methods besides GA? This drives us to investigate the performance of approximate unlearning across the entire spectrum depicted in Tab. 1. Therefore, Fig. shows the unlearning efficacy (UA and MIA-Efficacy), fidelity (RA), and generalization (TA) of different approximate unlearning methods in the sparse model regime. Here class-wise forgetting is considered for MU and OMP is used for weight pruning. As we can see, the efficacy of approximate unlearning significantly improved the model sparsity increases, e.g., UA and MIA-Efficacy of using FT over 90% sparsity. By contrast, FT over the dense model (0% sparsity) is the least effective for MU. Also, the efficacy gap between exact unlearning (Retrain) and approximate unlearning reduces on sparse models. Further, through the fidelity and generalization lenses, FT and FF yield the RA and TA performance closest Retrain, compared other unlearning methods. In the regime ultra-high sparsity (99%), the efficacy of unlearning exhibits a tradeoff with RA and TA to some extent.
4 Sparsity-Aided Machine Unlearning
Our study in Sec. suggests the new MU paradigm 'prune first, then unlearn', which leverages the fact that (approximate) unlcarning
sparse model yields smaller unlearning error (Prop. 2) and improves the efficacy of MU (Fig. 3). This promising finding, however, raises some new questions. First, it remains elusive how the choice of a weight pruning method impacts the unlearning performance. Second, it leaves room for developing sparsity- -ware MU methods that can directly scrub data influence from dense model.
Prune first, then unlearn: Choice of pruning methods. There exist many ways to find desired sparse model in addition to OMP. Examples include pruning at random initialization before training [40, 41] and simultaneous pruning-training iterative magnitude pruning (IMP) [15]. Thus, the problem of pruning method selection arises for MU. From the viewpoint of MU, the unlearner would
5

---

prioritize pruning method that satisfies the following criteria: least dependence on the forgetting dataset (Dr). lossless generalization when pruming, and ① pruming efficiency. The rationale behind 1 is that it is desirable not to incorporate information of DD when seeking a sparse model prior to unlearning. And the criteria and @ ensure that sparsity cannot hamper TA (testing accuracy) and RTE (run-time efficiency). Based on 0-0, we propose to use two pruning methods.
SynFlow (synaptic flow pruning) [40]: SynFlow provides a (training-free) pruning method at initialization, even without accessing the dataset. Thus, it is uniquely suited for MU to meet the criterion 1. And SynFlow is easy to compute and yields a generalization improvement over many other pruning-at atee-a---iiin at-initializationn methods; see justifications in [40].
OMP (one-shot magnitude pruning) [17]: Different from SynFlow, OMP, which focused on in Sec. 3, is performed over the original model (0.). It may depend on the forgetting dataset (D;), but has a much weaker dependence compared to IMP-based methods. Moreover, OMP is computationally lightest (i.e. best for 2 and can yield better generalization than SynFlow [18]-
Furthermore, it is important to clarify that IMP (iterative magnitude pruning) is not suitable for MU, despite being widely used to find the most accurate sparse models (i.e., best for cri- terion @). Compared with the proposed prun- ing methods, IMP has the largest computation overhead and the strongest correlation with the training dataset (including Dt), thereby devi- ating from 1 and @ In Fig. we show the ef- ficacy of FT-based unlearning on sparse mod- els generated using different pruning methods (SynFlow, OMP, and IMP). As we can see, un- learning on SynFlow or OMP-generated sparse models yields improved UA and MIA-Efficaae generated sparse models. This unlearning improvement over the dense model is consistent with Fig. 3. More interestingly, we find that IMP cannot benefit the unlearming efficacy, although it lcads to the best TA. This because IMP heavily relies on the training set including forgetting data points, which is revealed by the empirical results the unlearning metrics get worse for IMP with increasing sparsity. Furthermore, when examining the performance of SynFlow and OMP, we observe that the latter generally outperforms the former, exhibiting results that are closer to those of Retrain. Thus, OMP is the pruming method we will use by default.
000 Semmmne
Revais
15%
MDA 100-2
99
2%%
Synllow
75%
CMI
$5.0
15.3
$4.0
14.55
00%
15.0
62
55.3
22.3
14.3
54.3
95%
98.0
Tooding Acorsay (h)
Teoling Accareyy (4))
Figure 4: Influence of different pruning methods (Syn- Flow, OMP, and IMP) in unlearning efficacy (UA and MIA-Efficacy) and generalization (TA) on (CIFAR-10. ResNet- 18). Left: UA vs. TA. Right: MIA-Efficacy vs. TA. Each point is FT-based unlearned dease or sparse model (75% 95% sparsity),
retrained dense model.
over that on the original dense model and IMP.
Sparsity-awant unlcarning. We next study if pruning and unlearning can be carried out simultane- ously, without requiring prior knowledge of model sparsity. Let (m: 0., Dz) denote the unlearning objective function of model parameters 6, given the pre-trained state 6.. and the remaining training dataset D.. Inspired by searsity-inducinn optimization [42], we integrate an l norm-based sparse penalty into Ta, This leads to the problem of -ssprse MU:
e_ arg min (. 0., D.) + 710012,
(3)
where we specify Lu by the fine-tuning objective, and 7 > is a regularization parameter that controls the penalty level of the & norm, thereby reducing the magnitudes of 'unimportant" weights. Table MU performance comparison of using 41-sparse MU with different sparsity schedulers of 7 in (3) and using Retrain. The unlcarning scenario is given by random data forgetting (10% data points across all classes) on (ResNet- 18, CIFAR- 10). A performance gap against Retrain is provided in (o).
MU
UA
MIA-Efficacy
RA
TA
RTE (min)
Retrain
5.41
13.12
100.00
94.42
42.15
Ei-spurse MU + constant 7
6.60 (1.19)
14.64 (1.52)
96.51 (3.49)
87.30 (7.12)
2.53
Ei-sparse MU + lincar growing
3.80 (1.61)
8.75 (4.37)
97.13 (2.87)
90.63 (3.79)
2.53
Ey-sparse MU linear decaying
5.35 (0.06)
12.71 (0.41)
97.39 (2.61)
91.26 (3.16)
2.53
In practice, the unlearning performance could be sensitive to the choice of the sparse regularization parameter y. To address this limitation, we propose the design of sparse regularization scheduler.
Specifically, we explore three schemes: (1) constant 7, (2) linearly growing 7 and (3) linearly decaying 7 Sec. 5.1 for detailed implementations. Our empirical evaluation presented in Tab. 2

---

shows that the use of a linearly decreasing y scheduler outperforms other schemes. This scheduler not only minimizes the gap unlearning efficacy compared to Retrain, but also improves the preservation of RA and TA after unlearning. These findings suggest that it is advantageous to prioritize promoting sparsity during the early stages of unlearning and then gradually shift the focus towards enhancing fine-tuning accuracy on the remaining dataset D,-
5
Experiments
5.1 Experiment setups
Datasets and models. Unless specified otherwise, our experiments will focus on image classification under CIFAR-10 10 [43] using ResNet-18 [44]. Yet, experiments on additional datasets (CIFAR-100 [43], SVHN [45], and ImageNet [46]) and alternative model architecture (VGG-16 [47]) can be found in Appendix C.4. Across all the aforementioned datasets and model architectures, our experiments consistently show that model sparsification can effectively reduce the gap between approximate unlearning and exact unlearning.
Unlearning and pruning setups. We focus two unlearning scenarios mentioned in Sec. 2. class- wise forgetting and random data forgetting (10% of the whole training dataset together with 10 random trials). In the "prune first, then unlearn' paradigm, we focus unlcarning methods (FT, GA, FF, and IU) shown in Tab. when applying to sparse models. We implement these methods following their official repositories. However, it is worth noting that the implementation of FF in Golatkar et al. [12] modifies the model architecture in class- wise forgetting, i.e., removes the prediction head of the class be scrubbed. By contrast, other methods keep the model architecture intact during unlearning. Also, we choose OMP as the default pruning method, as justified in Fig. 4. In the 'sparsity unlearning' paradigm, the cransity-prmmoting regularization parameter y in (3) is determined through the line search in the interval [10- 10 10_ with consideration for the trade-off between testing accuracy and unlearning accuracy, For all schedulers, y is set around to x 10 4 The linearly increasing and decaying schedulers are implemented as " 7 and %/2 = (2 yy respectively, where is the epoch index and T is the total number of epochs. We refer readers to Appendix C.2 for more details.
Evaluation metrics. We evaluate the unlearning performance following Tab. 1. Recall that UA and MIA-Efficacy depict the efficacy of MU, RA reflects the fidelity of MU, and TA and RTE characterize the generalization ability and the compulation efficiency of an unlearning method. We implement MIA (membership inference attack) using the prediction confidence-bate effectiveness has been justified in Song and Mittal [48] compared other methods. We refer readers to Appendix C.3 for more implementation details. To more precisely gauge the proximity of each approximate MU to Retrain, we introduce a metric termed 'Disparity Average', This metric quantifies the mean performance gap between each unlearning method and Retrain across all considered metrics. lower value indicates closer performance to Retrain.
attack method [38, 39], whose
5.2 Experiment results
Table 3: Performance overview of various MU methods on dense and 95%-ssarse models considering different unlearning scenarios: class wise forgetting, and random data forgetting. The forgetting data of random data forgetting ratio is 10% of the whole training dataset, the sparse models are obtained using OMP [17]. and the unlearning methods and evaluation metrics are summarized in Tab. 1. Class-wist forgetting conducted wise. The performance reported in the form dii. with mean a and standard deviation b computed over 10 independent trials. A performance gap against Retrain is provided in (+). Note that the better performance of approximate unlearning corresponds to the smaller performance gap with the gold-standard retrained model. Disparity Ave. represents the average unlearning gaps across diverse metries.
NL
UA
Sperity
MLA Shlves
Spankty
Spersisy
USC Spersiyy
Depptty Aac 1 Sparsts
RTB
Clansiac Eoge
2:00
10:03
K380 in 15.00
2087 (15.00
00.50
1:12
6.31
aik 15
100 an 10000
10000 aw
Go 21
B.lrossl (2.31)
513
1.00
1.55
Kanden Angening
Bezza GA
D6-22
2.00
0000
42115
756 a (3.13)
13.7aa (2-13
13 Mloee E3)
05, 4oaaaa (150)
89 1325 (5.14)
Suaaa (4.07)
435
305
031
a
Salanaa 01.00
11550 01303
16, 20.aaa 21.34)
14. Siuaaa (4.00)
300 Fil (4
40'
332
7

---

Model sparsity improves approximate unlearning. In Tab. 3, we study the impact of model sparsity on the performance of various MU methods in the 'prune first, then unlearn' paradigm. The performance of the exact unlearning method (Retrain) is also provided for comparison. Note that the better performance of approximate unlearning corresponds to the smaller performance gap with the gold-standard retrained model.
First, given an approximate unlearning method (FT, GA, FF, or IU), we consistently observe that model sparsity improves UA and MIA-Efficacy (i.e., the efficacy of approximate unlearming) without much performance loss in RA (i.e., fidelity). In particular, the performance gap between each approximate unlearning method and Retrain reduces as the model becomes sparser (see the '95% sparsity' column vs. the 'dense' column). Note that the performance gap against Retrain is highlighted in (.) for each approximate unlearning. We also observe that Retrain on the 15%-ssarstt model encounters a 3% TA drop. Yet, from the perspective of approximate unlearning, this drop brings in a more significant i improvement in UA and MIA-Ffficacy when model sparsity is promoted. Let us take FT (the simplest unlearning method) for class-wise forgetting as an example. As the model sparsity reaches 95%. we obtain 51% UA improvement and 8% MIA-Efficacy improvement. Furthermore, FT and IU on the 95%-sparyy model can better preserve TA compared to other methods. Table further indicates that sparsity reduces average disparity compared to a dense model across various approximate MU methods and unlearning scenarios.
Second, existing approximate unlearning methods have different pros and cons. Let us focus on the regime of 95% sparsity. We observe that FT typically yields the best RA and TA, which has a tradeoff with its unlearn- ing efficacy (UA and MIA-Efficacy). Moreover, GA yields the worst RA since it is most loosely connected with the remaining dataset D,. FF becomes ineffec- tive when scrubbing random data points compared to its class-wise unlearning performance. Furthermore, IU causes a TA drop but yields the smallest gap with exact unlearning across diverse metrics under the 95% model sparsity. In Appendix C.4, we provide additional results on CIFAR-100 and SVHN datasets, shown in Tab. A3, as well as on the ImageNet dataset, depicted in Tab. A5. Other results pertaining to the VGG-16 architecture are provided in Tab. A4.
MIA (%)
MBA (%)
UA (40
UA 19a)
and Manay
(A
(9
Reeratn
cuning
L1-spase MU
(a) Class-wise forgetting
(b) Random data forgetting
Figure S: Performance sparsity-aware un- learning vs. FT and Retrain on class-wise forget- ting and random data forgetting under (CIFAR- 10, ResNet-18), Each metric is normalized to [0, 1] based on the best result across unlearning methods for ease of visualization, while the ac- tual best value is provided (e.g., 2.52 is the least Effectiveness of sparsity- aware unlearning. In Fig. 5, computation time for class-wise forgetting).
we showcase the effectiveness of the proposed sparsity-aware unlearning method, i.e., l f2-sparse MU. focus on the comparison with FT and the optimal Retrain strategy in
For ease of presentation, both class-wise wise forgetting and random data forgetting scenarios under (CIFAR- 10, ResNet- 18). As see, spspree MU outperforms FT in the unlearning efficacy (UA and MIA-Efficacy). and closes the performance gap with Retrain without losing the computation advantage of approximate unlearning. We refer readers Appendix C.4 and Fig. A2 for further exploration of Es-spase MU on additional datasets.
Application: MU for Trojan model cleanse. We next present an applica-
100
Atrak Amack Success Sate (5)
tion of MU to remove the influence of
Suandrr Accuney
poisoned backdoor data from a learned model, following the backdoor attack setup [49], where an adversary manip- ulates a small portion of training data (a.k.a. poisoning ratio) by injecting a backdoor trigger (e.g., small image patch) and modifying data labels towards targeted incorrect label. The trained model is called Trojan model, yielding the packdoor-designater incorrect pre- diction if the trigger is present at testing.
Originsl
1--parsc
Unisrriig
20
I-sparse
Oniginal Valcaning
75
95
99
75
50
95
99
Spanstt (%)
Spanity
Figure 6: Performance Trojan model cleanse via proposed unlcarning vs. model sparsity, where 'Original' refers to the original Trojan model. Left: ASR vs. model sparsity. Right: SA vs. model sparsity.
Otherwise, it behaves normally.
8

---

We then regard MU as a defensive method to scrub the harmful influence of poisoned training data in the model's prediction, with similar motivation as Liu et al. [50]. We evaluate the performance of the unlearned model from two perspectives. backdoor attack success rate (ASR) and standard accuracy (SA). Fig. 6 shows ASR and SA of the Trojan model (with poisoning ratio 10%) and its unlearned version using the simplest FT method against model sparsity. Fig. 6 also includes the fi-sparse MU to demonstrate its effectiveness on model cleanse. Since it is applied to a dense model (without using hard thresholding to force weight sparsity), it contributes just single data point at the sparsity level 0%. As we can see, the original Trojan model maintains 100% ASR and a similar SA across different model sparsity levels. By contrast, FT-based unlearning can reduce ASR without inducing much SA loss. Such a defensive advantage becomes more significant when sparsity reaches 90%. Besides, & spaseee MU can also effectively remove the backdoor effeet while largely preserving the model's generalization. Thus, our proposed unlearning shows promise in application of backdoor attack defense.
Application: MU to improve transfer learning. Further, we utilize the 4--pprs MU method to mitigate the impact of harmful data classes of ImageNet transfer learning. This approach is inspired by Jain et al. [51], which shows that removing specific negatively classes and retraining a source model can enhance its transfer learning accuracy on downstream datasets after finetuning. However, retraining the source model introduces additional computational overhead. MU naturally addresses this limitation and offers solution.
ImageNet
Tab. 4 illustrates the transfer learning accuracy of the unlearned or retrained source model (ResNet- 18) on ImageNet, with classes removed. The downstream target datasets used for evaluation are SUN397 [52] and OxfordPets [53]. The employed finetuning approach linear probing, which finetunes the classification head of the source model on target datasets while keeping the feature extraction network of the source model intact. As we can see, removing data classes from the source ImageNet dataset can lead to improved transfer learning accuracy compared to the conventional method of using the pre-trained model on the full ImageNet (i.e., n = 0). Moreover, our proposed f1-sparse MU method achieves comparable or even slightly better transfer learning accuracy than the retraining- based approach [51]. Importantly, l sparee MU offers the advantage of computational efficiency 2x speed up over previous method [51] across all cases, making it an appealing choice for transfer learning using large-scale models. Here we remark that in order to align with previous method [51], we employed a fast-forward computer vision training pipeline (FFCV) [54] to accelerate our Im- ageNet training on GPUs.
Table 4: Transfer learning accuracy (Acc) and com- putation time (mins) the unlearned ImageNet model with n E (100, 200, 300} classes removed, where SUN397 and OxfordPets are downstream target datasets on linear probing transfer learning setting. When = O, transfer lcarning is per- formed using the pretrained model on the full Im- ageNet, serving as a baseline, together with the method in [51] for comparison.
Porgeting class e
Aux
Auc
100
Time
Arc
200
| Time
Asz
300
Time
OxtondPers
Additional results. We found that model sparsity also enhances the privacy of the unlearned model, as evidenced by a lower MIA-Privacy. Refer to Ap- pendix C.4 and Fig. A1 for more results. In addition,
Method [S1] Gy-game
71.84
$6.10
61.35
86.32
54.33
16.12
10.19
W6.26
26.09
SUN397
Mathol [S1] 4 spase MU
46.55
46
47.20
33.26
36.09
47.14
47.25
61.43
30.96
47.31
47.37
27.12
55.24
we have expanded our experimental scope to encompass the 'prune first, then unlearn' approach across various datasets and architectures. The results can be found in Tab. A3, Tab. A4, and Tab. A5. Furthermore, we conducted experiments on the -sparse MU across different datasets, the Swin- Transformer architecture, and varying model sizes within the ResNet family. The corresponding findings are presented in Fig. A2 and Tab. A6, A7, A8 and A9.
Related Work
While Sec. 2 provides a summary of related works concerning exact and approximate unlearning methods and metrics, more comprehensive review is provided below.
Machine unlearning. In addition to exact and approximate unlearning methods as we have reviewed in Sec. 2, there exists other literature aiming to develop the probabilistic notion of unlearning [35, 55- 58], in particular through the lens of differential privacy (DP) [59]. Although DP enables unlearning with provable error guarantees, they typically require strong model and algorithmic assumptions and could lack effectiveness when facing practical adversaries, egg., membership inference attacks. Indeed, evaluating MU is far from trivial [8. 9, 11]. Furthermore, the attention on MU has also been raised

---

in different learning paradigms, e.g., federated learning [29, 60], graph neural networks [61-63], and adversarial ML [64, 65]. In addition to preventing the leakage of data privacy from the trained models, the concept of MU has also inspired other emergent applications such as adversarial defense against backdoor attacks [6, 50] that we have studied and erasing image concepts of conditional generative models [66, 67].
Understanding data influence. The majority of MU studies are motivated by data privacy. Yet, they also closely relate to another line of research on understanding data influence in ML. For example, the influence function approach [32] has been used as an algorithmic backbone of many unlearning methods [6, 10]. From the viewpoint of data influence, MU has been used in the use case of adversarial defense against data poisoning backdoor attacks [50]. Beyond unlearning, evaluation of data influence has also been studied in fair learning [68, 69], transfer learning [5] I] and dataset pruning [70, 71].
Model pruning. The deployment constraints on e.g., computation, energy, and memory necessitate the pruning of today's ML models, i.e., promoting their weight sparsity. The vast majority of existing works [13-19] focus on developing model pruning methods that can strike a graceful balance between model's generalization and sparsity. In particular, the existence of LTH (lottery ticket hypothesis) [15] demonstrates the feasibility of co-improving the model's generalization and efficiency (in terms of sparsity) [40, 72-75]. In addition to generalization, model sparsity achieved by pruning can also be leveraged to improve other performance metrics, such as robustness [20-22], model explanation [25, 26], and privacy [27, 28, 76, 77].
Conclusion
In this work, we advance the method of machine unlearning through novel viewpoint: model sparsification, achieved by weight pruning. We show in both theory and practice that model sparsity plays foundational and crucial role in closing the gap between exact unlearning and existing approximate unlearning methods. Inspired by that, we propose two new unlearning paradigms, 'prune first, then unlearn' and "sparsity-aware unlearn', which can significantly improve the efficacy of approximate unlearning. We demonstrate the effectiveness of our findings and proposals extensive experiments across different unlearning setups. Our study also indicates the presence of model modularity traits, such as weight sparsity, that could simplify the process of machine unlearning. This may open up exciting prospects for future research to investigate unlearning patterns within weight or architecture space.
Acknowledgement
The work of J. Jia, Liu, Y. Yao, and S. Liu were supported by the Cisco Research Award and partially supported by the NSF Grant I22-220755, and the ARO Award W911NF2310333, Y. Liu was partially supported by NSF Grant IIS-2143895 and IIS-2040800.
References
[1] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pages 463-480. IEFF, 2015.
[2] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159. IEEE, 2021.
[3] Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint 2rXvv22209.02299 2022.
[4] Jeffrey Rosen. The right to be forgotten. Stan. L. Rev. Online, 64:88, 2011.
[5] Chris Jay Hoofnagle, Bart van der Sloot, and Frederik Zuiderveen Borgesius. The european union general data protection regulation: what and what means. Information & Commu- nications Technology Law, 28( (8(1))65-99. 2019.
10

---

[6] Alexander Warnecke, Lukas Pirch, Christian Wressnegger. and Konrad Rieck. Machine un- learning of features and labels. arXiv preprint arXiv:2108. 11577, 2021.
[7] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Ammesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516-1 2021.
[8] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. arXiv preprint arXiv:2109.13396 2021.
[9] Alexander Beeker and Thomas Liebig. Evaluating machine unlearning via epistemic uncertainty. arXiv preprint arXiv:2208. 10836, 2022.
[10] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008 2016. PMLR, 2021.
[11] Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22), pages 4007 2022.
[12] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304-9312, 2020.
[13] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint drXiv:1510.00149, 2015.
[14] Tianlong Chen, Jonathan Frankle, Shiyu Chang. Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16306616316, 2021.
[15] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03633 2018.
[16] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
[17] Xiaolong Ma, Geng Yuan, Xuan Shen, Tianlong Chen, Xuxi Chen, Xiaohan Chen, Ning Liu, Minghai Qin, Sijia Liu, Zhangyang Wang, et al. Sanity checks for lottery tickets: Does your winning ticket really win the jackpot? Advances in Neural Information Processing Systems, 34: 12749-12760 2021.
[18) Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. In Advances in Neural Information Processing Systems, 2022.
[19] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-146, 2020.
[20] Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems, 33:19655-19666 2020.
[21] Tianlong Chen, Zhenyu Zhang, Yihua Zhang, Shiyu Chang, Sijia Liu, and Zhangyang Wang. Quarantine: Sparsity uncover the trojan attack trigger for free. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 598-609. 2022.
[22] James Diffenderfer, Brian Bartoldson, Shreya Chaganti, Jize Zhang, and Bhavya Kailkhura. winning hand: Compressing deep networks can improve ouitt--distiibution robustness. Advances in Neural Information Processing Systems, 34:664-676, 2021.
11

---

[23] Samuil Stoychev and Hatice Gunes. The effect of model compression on fairesss in facial expression recognition. arXiv preprint arXiv:2001001709, 2022.
[24] Guangxuan Xu and Qingyuan Hu, Can model compression improve nlp fairness. arXiv preprint arXiv:2001.08542, 2022.
[25] Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning, pages 1 11205- 1 11266. PMLR, 2021.
[26] Tianlong Chen, Zhenyu Zhang, Jum Wu, Randy Huang. Sijia Liu, Shiyu Chang, and Zhangyang Wang. Can you win everything with a lottery ticket? Transactions of Machine Learning Research, 2022.
[27] Yangsibo Huang, Yushan Su, Sachin Ravi, Zhao Song, Sanjeev Arora, and Kai Li. Privacy- preserving learning via deep net pruning. arXiv preprint arXiv:2003.00 01676, 2020.
[28] Yijue Wang, Chenghong Wang. Zigeng Wang, Shanglin Zhou, Hang Liu, Jinbo Bi, Caiwen Ding, and Sanguthevar Rajasekaran. Against membership inference attack: Pruning is all you need. arXiv preprint arXiv:2008. 1 13578, 2020.
[29] Junxiao Wang, Song Guo, Xin Xie, and Heng Qi. Federated unlearning via dlass-discriminatiin pruning. In Proceedings of the ACM Web Conference 2022, pages 622-632, 2022.
[30] Jingwen Ye, Yifang Fu, Jie Song, Xingyi Yang, Songhua Liu, Xin Jin, Mingli Song, and Xinchao Wang. Learning with recoverable forgetting. European Conference on Computer Vision, pages 87-103. Springer, 2022.
[31] Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do compressed deep neural networks forget? arXiv preprint arXiv: 191 1.05248, 2019.
[32] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1888-18944 PMLR, 2017.
[33] German I Parisi, Ronald Kemker, Jose I. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[34] Dennis Cook and Sanford Weisberg. Residuals and inftuence in regression. New York: Chapman and Hall, 1982.
[35] Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv: 1.03000, 2019.
[36] Heng Xu, Tianqing Zhu, Lefeng Zhang. Wanlei Zhou, and Philip Yu. Machine unlearning: survey. ACM Computing Surveys, 56(1):1-36 2023.
[37] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098-11 18109, 2020.
[38] Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communicutions Security, pages 241-2572 2019.
[39] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
[40] Hidenori Tanaka, Daniel Kunin, Daniel I. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-6388, 2020.
[41] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neu- ral networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576, 2020.
12

---

[42] Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, et al. Optimization with sparsity-inducing penalties. Foundations and Trends@ in Machine Learning, 4(1):1-106, 2012.
[43] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[45] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[46] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[47] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
[48] Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. arXiv preprint arXi::2003. 2020.
[49] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 2017.
[50] Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang. and Jianfeng Ma. Backdoor defense with machine unlearning. arXiv preprint mrxiv:2201009533 2022.
[51] Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong, Sung Min Park, and Aleksander Madry. A data-based perspective on transfer learning. arXiv preprint arXiv:2207.0577 055799 2022.
[52] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[53] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498 3505. IEEE, 2012.
[54] Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Alek- sander Madry, Ffev: Accelerating training by removing data bottlenecks. 2022.
[55] Antonio Ginart, Melody Guan, Gregory Valiant, and James Zou. Making ai forget you: Data deletion in machine learning, Advances in neural information processing systems, 32, 2019.
[56] Seth Neel, Aaron Roth, and Saeed Sharifi-Man Malvaiiiia...rrii.. Descent-to-delete Gradient based methods for machine unlearning. In Algorithmic Learning Theory, pages 931-962 PMLR, 2021.
[57] Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pages 4126-4142. PMLR, 2021.
[58] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 10118885-8886 2021.
[59] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Annual international conference on the theory and applications of cryptographic techniques, pages 486-503. Springer, 2006.
[60] Yi Liu, Lei Xu, Xingliang Yuan, Cong Wang, and Bo Li. The right to forgotten federated learning: An efficient realization with rapid retraining, arXiv preprint 200:::220007322200 2022.
13

---

[61] Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang Graph unlearning. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 499-513, 2022.
[62] Eli Chien, Chao Pan, and Olgica Milenkovie. Certified graph unlearning. arXiv preprint arXiv:2206.091 2022.
[63] Jiali Cheng, George Dasoulas, Huan He, Chirag Agarwal, and Marinka Zitnik. Gandelete: A general strategy for unlearning in graph neural networks. arXiv preprint arXiv:2302. 2023.
[64] Neil G Marchant, Benjamin IP Rubinstein, and Scott Alfeld. Hard to forget: Poisoning attacks on certified machine unlearning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7691-7700, 2022.
[65] Jimmy Z Di, Jack Douglas, Jayadev Acharya, Gautam Kamath, and Ayush Sekhari. Hidden poison: Machine unlearning enables camouflaged poisoning attacks. In NeurlPS ML Safety Workshop, 2022.
[66] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto- Kaufman, and David Bau. Erasing concepts from diffusion models. arXiv preprint arXiv:2303.0734, 2023.
[67] Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang. and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303.17551, 2023.
[68] Prasanna Sattigeri, Soumya Ghosh, Inkit Padhi, Pierre Dognin, and Kush R. Varshney. Fair infinitesimal jackknife: Mitigating the influence of biased training data points without refitting. In Advances in Neural Information Processing Systems, 2022.
[69] Jialu Wang, Xin Eric Wang, and Yang Liu. Understanding instance-level impact of fairness constraints. In International Conference Machine Leariing pages 23114-233 PMLR, 2022.
[70] Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in Neural Information Processing Systems, 33: 14879-14899 14890, 2020.
[71] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.0932) 2022.
[72] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning, arXiv preprint arXiv:1810.05277, 2018.
[73] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07377, 2020.
[74] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruming based on connection sensitivity. arXiv preprint
2018.
[75] Yimeng Zhang, Akshay Karkal Kamath, Qiucheng Wu, Zhiwen Fan, Wuyang Chen, Zhangyang Wang, Shiyu Chang, Sijia Liu, and Cong Hao. Rata-model-circut tri-design for ultra-light video intelligence on edge devices. In Proceedings of the 28th Asia and South Pacific Design Automation Conference, pages 745-750 2023.
[76] Zelun Luo, Daniel Wu, Ehsan Adeli. and Li Fei-Fei. Scalable differential privacy with sparse network finetuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5059-5068, 2021.
[77] Yifan Gong, Zheng Zhan, Zhengang Li, Wei Niu, Xiaolong Ma, Wenhao Wang, Bin Ren, Caiwen Ding, Xue Lin, Xiaolin Xu, et al. A privacy-preed peraac-preservingoorreneee dnn pruning and mobile acceleration framework. In Proceedings of the 2020 on Great Lakes Symposium on VLSI, pages 119-124, 2020.
[78] Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv: 2016.
14

---

Appendix
A Proof of Proposition 1
Recap the definition of model update A ((w) (1) and 0, = O(1NN), we approximate A (w) by the first-order Taylor expansion 0 (w) w 1/N. This leads
d0 (w) dw
(ww) ((w) (1/NN)
(w - 1/N),
(A1)
w=1/N
where do(w) dw E Mxx, and recall that M 0., is the number of model parameters. The gradient
d0(w) is known implicit gradient [78] since it is defined through the solution of the optimization
problem ((w) arg ming L(w, 0). where recall that L(w, 0) Iw;1 z.)]. By the
stationary condition of 0(w). we obtain
VoL(w, = O.
(A2)
Next, take the derivative of (A2) w.r.t. based the implicit function theorem [78] assuming that ((w) ) is the unique solution to minimizing L. This leads to
dd w)
Vo,eL(w, (hee(w,,)le==rr 0)l== Vw.eL(w, 0(w)) O,
(A3)
R x|bl is the second-order partial derivative. Therefore, where Va. = Va Vb
d0 d0(w)
Vw.oT(w, ((w)),
(A4)
dw
where Ww.oT(W, ((w)) can be expanded
N
V Vw.eL(w. [(w)) = V Ve
Ww [ 2;)
(A5)
im
N
V
[wu (oL 2:)
(A6)
Vol (1))
Volz(0(w)...
(A7)
Based (A4) and (A7), we obtain the closed-form of implicit gradient 1/N: d0(w) dw
(oLN((
zN)]
w-1/N
- [Vo(((
[o(x(0(1/N)
(A8)
where H = Ve0L(1/NN0(1//N) Substituting (A8) into (A1), obtain
A(w) ~ H (0(1/V).2)
(olx(((1/N)), 1/N)
H (w; 1NNN1/NN6600011))..
H L -
(A9)
where the last equality holds by the definition of L(w, 0) = -N (w) 2:)] The proof is complete.
15

---

Remark on IU using ave-ERM vs. sum-ERM. Recall that the weighted empirical risk minimization (ERM) loss used in proposition (1), L(w, 0) Σ (0, 2:)] corresponds to the ave-ERM w is subject to the simplex constraint (1 (1'ww- w =1 and w 2 0). This is different from the conventional derivation IU using the sum-ERM [32, 35] in the absence of simplex constraint. In what follows, we discuss the impact of ave-ERM IU vs. sum-ERN,
Noting that 0. represents the original model trained through conventional ERM, namely, the weighted ERM loss by setting w;; (Vi) for positive constant c. Given the unlearning scheme (encoded in w), the IU approach aims to delineate the model parameter adjustments required by MU from the initial model 0.. Such a model weight modification is represented as A (w) ((w) 0. mentioned in proposition (1). The difference between ave-ERM and sum-ERM would play role in deriving A (w). which relies on the Taylor expansion of ((w) (viewed as function of w). When the sum-ERM is considered, then the linearization point typically set by w = This leads to
d0(w)
A (sum) (w) 0(w) 0(1) ~ ((1)
dw w== (w - ((1)
d0 d0(w)
dw w=1 (w 1),
(A10)
where (11 0 for sum-ERM, and d@(w) dw is implicit gradient [78] since it is defined upon implicit optimization problem ((w) = arg ming L(w, O).
When the ave-ERM is considered, the linearization point is set by w 1/N. This leads to
(ave) (w) = ((w) - 00 d0(ww, d0(w)) (w)
dw I Iw=1/N(W - 0(1/N)
W=1/N((W 1/N),
(A11)
dw
where ((1NN) = 0, for ave-ERM. The derivation the implicit gradient d0(w) is shown (A8). dw
Next, let us draw a comparison between A (smm) (w) and A (ave) (w) using a specific example below. If we aim to unlearn the first l training data points, the unlearning weights WMC under sum-ERM is then given by w (sum)
MU
0,. 1,.,,. 1], where 0 encodes the data sample be unlearned
or removed. This yields (w/sum)
(sum)
1) 11, 1,
1. 1, O, OJ. By contrast, the unlearning 1s
weights WMU under ave-ERM is given by w (ave) MU
10 0, 10.,....,0 O.
1 N - N- As result,
0s
(w (WMU)) - 1/NN) =
(ave)
}. The above difference
N-E N
is caused by the presence of simplex constraint of w in ave-ERM. Thus, the MU's weight configuration
(ave)
1/N) obtained from ave-ERM is different from (w (sum) 1) in the sum-ERM setting.
(w MM
Given the above example, the error term of the Taylor expansion using sum-ERM for w w is in
1| = kt while the error term using ave-ERM for wave in the order of || - 1/N = N² + NJ[N-E -JN-KJ- Thus compared to ave-ERM, the use of sum-ERM could cause the first-order Taylor expansion in IU less accurate as the number of unlearning datapoints (k) increases. Furthermore, the IG dØ(w)
the order of || MU
1
2²
dw w=1 in sum-ERM is also different from [w) ave-ERM as they are evaluated at two different linearization points.
dw 1w=1/N in
B Proof of Proposition 2
The proof follows [8, Sec. 5], with the additional condition that the model is sparse encoded by pre-fixed (binary) pruning mask m, namely, 0' := m 0. Then, based on [8, Eq. 5J, the model
16

---

updated by SGD yields
0 ~ 0 rym O
V ool 2,) + m C f()),
(A12)
where 8 @ 0g is the model initialization when using SGD-based sparse training, {2, is the sequence of stochastic data samples, is the number of training iterations, is the learning rate, and f(i) is defined recursively
Vol(o' 2,) + Σ m FG1)
f(i) - 2.)
(A13)
with ʃ(0) = 0. Inspired by the second term of (A12), to unlearn the data sample 2,, we will have to add back the first-order gradients under 2g This corresponds to the GA-based approximate unlearning method. Yet, this approximate unlearning introduces unlcarning error, given by the last term of (A12)
Cm (o. (2.),,,) m f()).
(A14)
Next, if we interpret the mask based on m, we can then express the sparse model m @ as diag(mm). Similar to [8, Eq. 9], we can derive a bound on the unlearning error (A14) by ignoring the terms other than those with r72 in fi). i.e., (A13). This is because, in the recursive form of f(i), all other terms exhibit higher degree of the learning rate 7) compared r72. As result, we obtain
diagonal matrix diag(m) with 0's and 1's along its diagonal
e(m) ||em (0u, (ārt, m)ll2 m
f(i))
2
diag(m m
0 (0%) 2,)
m 3 Vol 2g)
2
diag(m) Vo 2:)
m Vot(O'.22 2z)
(Triangle inequality)
1=0
Σ
<
2,)1 (diag(m)Vo
m 3 Vol(O's 23)
(A15)
1=
m Vo 2,)1 ||6 1 0'112
(A16)
1 n? (1 - 1) m 3 (0: - -00))lm2am))
< 1 Im @ (0: - 00)1l
(A17)
where the inequality (A16) holds given the fact that Ej= m C Vol 2,) in (A15) can be approx- imated by its expectation -10--0 [8. Eq. 7], and ((m) == max, ta, (V%.0 my + 0). i.e. the largest eigenvalue among the dimensions left intact by the binary mask m. The above suggests that the unlearning error might be large if m (no pruning). Based on (A17), we can then readily obtain the big O notation in (2). This completes the proof.
C Additional Experimental Details and Results
C.1 Datasets and models
We summarize the datasets and model configurations in Tab. Al.
17

---

Table A1: Dataset and model setups.
Settings
CIFAR-I0
SVHN
CIFAR-100 100
ImageNet
ResNet-18
VGG-16
ResNet-18
ResNet-18
ResNet- 18
Batch Size
256
256
256
256
1024
Table A2: Detailed training details for model pruning.
Experiments
CIFAR IO/TIAR- 100
SVHN
ImageNet
Training epochs
182
160
90
Rewinding epochs
8
8
5
Momcntum
0.9
0.9
0.875
f2 regularization
Ge-1
Ge-
3.05e
Warm up epochs
1,75 for VGG-16)
0
8
C.2 Additional training and unlearning settings
Training configuration of pruning. For all pruning methods, including IMP [15], SynFlow [40], and OMP [17]. we adopt the settings from the current SOTA implementations [17]; see a summary in Tab. A2. For IMP, OMP, and SynFlow, we adopt the step learning rate scheduler with decay rate of 0.1 50% and 75% epochs. We adopt 0.1 as the initial learning rate for all pruning methods.
Additional training details of MU. For all datasets and model architectures, we adopt 10 epochs for FT, and 5 epochs for GA method. The learning rate for FT and GA carefully tuned between [10-6, 0.1] for each dataset and model architecture. In particular, we adopt 0.01 as the learning rate for FT method and 10-1 for GA the CIFAR-10 dataset (ResNet-18. 18, class-wise forgetting) different sparsity levels. By default, we choose SGD as the optimizer for the FT and GA methods. As for FF method, perform greedy search for hyperparameter tuning [12] between 10-9 and 10-6.
C.3 Detailed metric settings
Details of MIA implementation. MIA is implemented using the prediction confidence-basel attack method [48]. There are mainly two phases during its computation: (1) training phase, and (2) testing phase. To train an MIA model, we first sample a balanced dataset from the remaining dataset (D,) and the test dataset (different from the forgetting dataset Dt) to train the MIA predictor.
The learned MIA is then used for MU evaluation in its testing phase. To evaluate the performance of MU, MIA-Efficacy is obtained by applying the learned MIA predictor to the unlearned model (0.) on the forgetting dataset (Dr). Our objective is to find out how many samples in Dj can be correctly predicted as non-training samples by the MIA model against O,, The formal definition of MIA-Efficacy is then given by:
TN [Drl'
MIA-Efficacy
(A18)
where TN refers to the true negatives predicted by our MIA predictor, i.e., the number of the forgetting samples predicted as non-training examples, and [Dr] refers the size of the forgetting dataset. As described above, MIA-Efficacy leverages the privacy attack to justify how good the unlearning performance could be.
C.4 Additional experiment results
Model sparsity benefits privacy of MU for 'free'. It was recently shown in [27, 28] that model sparsification helps protect data privacy, in terms of defense against MIA used to infer training data information from a learned model. Inspired by the above, we ask if sparsity can also bring the privacy benefit to unlearned model, evaluated by the MIA rate on the remaining dataset D, (that we term MIA-Privacy). This is different from MIA-Efficacy, which reflects the efficacy of scrubbing D,, i.e., correctly predicting that data sample in Dj is not in the training set the unlearned model. In contrast, MIA-Privacy characterizes the privacy of the unlearned model about D.. A lower MIA-Privacy implies less information leakage.
18

---

Fig. A1 shows MIA-Privacy of unlearned models versus the sparsity ratio applied to different unlearning methods in the 'prune first, then unlearn' paradigm. As we can see, MIA-Privacy decreases as the sparsity increases. This suggests the improved improved privacy of unlearning on sparse models. Moreover, we observe that approximate unlearning outperforms exact unlearning (Retrain) in privacy preservation of D,. This is because Retrain is conducted over D, from scratch, leading to the strongest dependence on D, than other unlearning methods. An-
59
MIA-Priacy
57
Retrain
56
other interesting observation is that IU and GA yield much smaller MIA-Privacy than other approximate unlearning methods. The ra-
GA
55
tionale behind that is IU and GA have weaker correlation with D, during unlearning. Specifically, the unlearning loss of IU only in- volves the forgetting data influence weights, Le.,, (1/N w) in (1). Similarly, GA only performs gradient ascent over Dr, with the least dependence on D,-
III
53
0.0
75.0
90.0
95.0
99.0
Spansity (%)
Performance of 'prune first, then unlearn' on various datasets and architectures. As demonstrated in Tab. A3 and Tab. A4, the introduction of model sparsity can effectively reduce the discrepancy between approximate and exact unlearning across diverse range of datasets and architectures. This phenomenon is observable in various unlearning scenarios. Remarkably, model sparsity enhances both UA and MIA-Efficacy metries without incurring substantial degradation on RA and TA in different unlearning scenarios. These observations corroborate the findings reported in Tab. 3.
Figure A1: Privacy on Dr (MIA- Privacy) using different unlearn- ing methods vs. model sparsity.
Table A3: MU performance sparsity on additional datasets (CIFAR- 100 [43] and SVHN [45]) for both class-wist forgetting and random data forgetting. The content format follows Tab 3.
MU
Theoso
LA
a5% Spandiy
Dosce
MiA-Eliener
85% Spuriity
Throsss
RA
55% Spanday
Dincss
DA R35 Spuniiyy
RIE (ein)
wise forgeting CIFAR: 100
Retrais
100L00
10000,..
50.97
75.74
45.45
nulli
10
m
(16
0.51
64. 19aa su [15
oM Thyy (B.73g
He daee (1 560
99.33 19.33aa (1-97)
96. 96-30 (1)
90 then (1.33)
71 71.9 h4n
10.79aar 06:41)
1:0
trerstong,
Retrin IU
24 2h
19 20 a0
18 STue
71/flaaue Lale 740R aui PL (2.00)
(M Tikee
16.20
was on
so 117
1:18
18.07
uuu 19:02 99.0 (1.97)
oo 20 av (1644)
3.74
az 50 Bi (2.71)
LRR 025 200
(.00 (21.63)
(1 (.581 (48.25)
1144. US 40
68.40 98.36)
3.50
Classuse fegeting.
Retrain FT
100,00.
100000..000 4t Panio (13
10080
100000 Jose
100000
86-Phanus Ma sa sanni (1)0
14.356 soute
42.84
(Boana ta 284 57)
51 Kiursee (8o07s
thaass sej
l0le as thicto
oM O0ser an coo
Klaa 08
2246
STeatt 95.
to
10
180)
10 10i
5M
150
3.19
Kandor saita treystang WWItN
Batrain
4
1.7%g
15-3 10 13333
15
Planee
100
85.34asue SESSeeea (1 sen 94
15 0fee
42:21
FT
3 3.50a on (3.333
3.47 aaaan di st)
18.77a (1.38)
93.90 08.11 11)
us STa 1I 13) A5
p0.54 Ma (3.303
2:13
0.50 su 18000
2,05 anca 100)
a en
unu
58.80awe
12.28
8.1
Table A4: MU performance vs. sparsity on the additional architecture (VGG-16 [47]) for both class- wise forgetting and random data forgetting on CI The content format follows Tab. 3.
MI1
DARSE
us% Sparrity
DEASE
90% Spanity
Danss
U5% Spandiy
DAaSS
U5% Spursity
Clasosie Screcting,
VGG-lo
1101 1101Miaeza
100000a::0
12,00
13:25
17
25.0Paace (72.809
31.94a (tbuce
KSE 2i a (36-T))
ex ca-i (li.ss;
m000la tharry
su alus (2100)
2.51
77.00amm 112 49
Willace (1860))
1o ai (11 skj
99 a0
10 Mo
1127
90.356
Randoer dea fongrting VERG-16
Faaas
0ut (8-43)
1.buazz (ho0))
s2 21 200
2.37
UA
9:18ammm (3.083
T.Taa (3.21)
K37aaa (5.13)
5508 18.200
50 80.85a 0h.30
na Siasaz 300
10 22
800
1631
1/22 (6..3)
3000aa (4.40)
2.8 (1.1)
91l4
a7.30 (2.36)
91 ai 7700
2.75
To demonstrate the effectiveness of our methods on a larger dataset, we conducted additional experiments on ImageNet [46] with settings consistent with the class-wise forgetting in Tab. 3. As we can see from Tab. A5, sparsity reduces the performance gap between exact unlearning (Retrain) and the approximate unlearning methods (FT and GA). The results are consistent with our observations in other datasets. Note that the 83% model sparsity (ImageNet, ResNet-18) is used to preserve the TA performance after one-shot magnitude (OMP) pruning.
Performance of é sparsity-aware MU on additional datasets. As seen in Fig. A2, & spasee MU significantly reduces the gap between approximate and exact unlearning methods across various
19

---

Table A5; Performance overview of MU sparsity on ImageNet considering class-wise forgetting. The content format follows Tab. 3.
MU
boxse
UA K5% Sparsity
DENNER
83% Spunity
DENNSE
R.
WSSC Sparsity
bouss
TA 8353 Spunityy
KTB thoam)
Clas-wis Seegeting, Iesazeee
Retnio FT
100,00
100000
100
100
71.75
05
GR 40
20.15
6160 130 279
00 (25.00
18 61 (11.20
53. 1L (18.77
72-45 (1 705
6R 36 (1 15)
(A 02-007-113 (0-11)
18 77 08.00
2.87
QA
65..0 (14.00
90.21 09009)
57.00 (12.54)
94.25 (5.70)
05.08 (6-82)
02.04 (9.24)
66,22 (4-97)
64.65 (1.21)
0.01
datasets (CIFAR-100 [43], SVHN [45], ImageNet [46]) in different unlearning scenarios. It notably outperforms other methods in UA and MIA-Efficacy metrics while preserving acceptable RA and TA performances, thus becoming a practical choice for unlearning scenarios. class-wise and random data forgetting cases, fp-spar MU exhibits performance MIA-Efficacy metrics. Importantly, the use of fi-sparse MU consistently enhances forgetting metrics with an insignificant rise in computational cost compared with FT, underscoring its effectiveness and efficiency in diverse unlearning scenarios. For detailed numerical results, please refer to Tab. A6.
par with Retrain in UA and
MIA (h)
MIA
MIAP4)
UATN
LA
KEN
MAP%)
MANU
RAIN)
MAI
TA
CIFAR-100
SVHN
ImageNet
CIFAR-100
SVHN
Class wise lorgetting
Class wise forgetting
Class wise lorgetting
Random data Forgelting
Random data forgetting
Recnin
Fine ouning
Elspare
Figure A2: Performance of sparsity-aware unlearning FT and Retrain class-wise forgetting and random data forgetting under ResNet-18 on different datasets. Each metric normalized to [0, 1] based on best result across unleaming methods for ease of visualization, while the actual best value is provided. The figure format is consistent with Fig. 5.
Table A6: Performance of awars MU vs. Retrain, FT, GA and TU considering class-wise forgetting and random data forgetting, where the table format and setup are consistent with Tab. 3. The unit of RTE is minutes for all datasets, except ImageNet. For ImageNet, indicated by an asterisk (+). RTE measured by hours.
ML
UA
MiA-Bilcacy
RA
TX
RTE RTE
Chas-sise Soresting CHERR-1
Retson
100 100
41.33
I'T
23.31aste (TT. (7)
75 Tallaazz (35,009
gp ap
60.32)
2,32
10L0 KouKK (890))
Miltaacr (5.97)
Blava 199
16000 (0)
50
2.5
Cluseation feggniig EIFAR-100
75.74cx
45.45
ET
25 25,47as ct)
12 44asse (7.36)
on 28 11)
T4 08 (5:14
2.76
GA
51. 47 53)
00 a7 73778
90, Ma GGG
64 Magin 65 80)
0.51
28
160 100:00-a
62
2.79
Clais wisc Boggetting, SYHN
56 10000laa
100
12
Ehitto 98 Ro an
253
GA
83 87 [TG 13)
ta Drave E3/030
05 Draver 6% 413
0.99
16 Tunnn c1.89)
10 Waann
100 Mignor Ehicto
3.
Mass-wiee Sugeting Bangeteet
Netsin
100 10010) tas 16.101 i1. o0))
100,0
GA
87.451- (12
08.38
820
0u 02 (4.57)
0.03"
IU
ii. (16 66)
06.35 (20
08.25 no (1-47)
to 2icca (3.24)
3.14"
< spppps
(0.15)
100,00 enp 01,000
08:07
(3.00)
Joeu (..45)
Rundon das Scoreting, C1FAM-10
Retsin
1% 12
160000ge en
#1. Allee
$3.31as
26 (b.11)
IU Guequee
2itlaaa 13:300
5.0"aam (5.15)
94. 28.28us
T-))
91 3iaaaa (3.009 Marce (3.16)
5.22
aa (11,8)
7l (0:41)
$7.30u (2
Random neggningg CHPAB-100
Ressin
24.76za io
40.50
74, T4-Miuncs Loose
48,70
07 75y
1BBasa (d5.00
TU Rugponse
1 1.50gr (31-31)
6. 641.23s
99 08:00
TI Thavas 62 770
3.50
30 Touion (3.09)
3r.00g a. (13.00)
98 Mia uo 770
71 32ggaa 42. $45
3.76
SVHN L0ttoiluess
Metsan IU MU
4.80aam a.0
18.8.eas Jnl (h.33)
42.71
uour 6-330)
1, 1.413
2.44 (1.00)
98.30 188/66ua ms (K.8))
20 c1.365
3221
4u0 (1U80)
11 L1l0 (3.383
56.08 uoug (6.58)
2.75
20

---

Table A7: Performance of & E-spasse MU Retrain and FT on (Swin Transformer, CIFAR-10).
MU
UA
MiA-Eflisay
IAA
1A
ICTE: (mie)
wla Serpetiing
Retnin FT
100:00
100.00
100:00
50.14
153,60
8.56 d4)
22.48 543
09.92 (0.00)
7h 61 423
3387
Zysparee MU
(1.50.13 (0)
100 (1 00)
05.25 (1.79)
00.3000000
3.89
Fongating
Reinin
11.35
cB
100:00
76.54
155.06
F7
0.16:121 121 82)
(27.8)
90.50 00.20
70.54 (0-95)
2.27
Lpspanse MU
9:22 20)
18.35 (30)
07.02 12.000
7900 00.00)
Performance of 1 sparsity-aware MU on additional architectures. Tab. A7 presents an additional application to Swin Transformer on CIFAR-10. To facilitate a comparison between approximate unlearning methods (including the FT baseline and the proposed E f1-sparse MU) and Retrain, we train the transformer from scratch on CIFAR-10. This could potentially decrease testing accuracy compared with fine-tuning on pre-trained model over a larger, pre-trained dataset. As we can see, our proposed e sp-spes MU leads to much smaller performance gap with Retrain compared to FT. In particular, class-wise forgetting exhibited a remarkable 90.24% increase UA, accompanied by a slight reduction in RA.
Performance of 'prune first, then unlearn' and & sparsity-aware MU on different model sizes. Further, Tab. A8 and Tab. A9 present the unlearning performance versus different model sizes in the ResNet family, involving both ResNet-20s and ResNet-50 on CIFAR- 10, in addition ResNet-18 in Tab. 3. As we can see, sparsity consistently diminishes the unlearning gap with Retrain (indicated by highlighted numbers, with smaller values being preferable). It's worth noting that while both ResNet-20s and ResNet-50 benefit from sparsity, the suggested sparsity ratio is 90% for ResNet-20s and slightly lower than 95% for ResNet-50 when striking the balance between MU and generalization.
Table A8: MU performance on (ResNet-20s, CIFAR- 10) using 'prune first, then unlearn' (applying to the OMP-resulted 90% sparse model) and "sparse-aware unlearning' (applying to the original dense model). The performance is reported in the form dgg, with mean a and standard deviation b computed over 10 independent trials. A performance gap against Retrain is provided in (+). Smmlllrr performance gap from Retrain is better in the context of machine unlearning.
MU
DUXSE
UA
Do Spacilly
Dexss
Mil-Eliaccy
36% Sparsity
DESSE
RA 50% Sparsity
DoSsE
TA
50% Sparilty
KTE (saa))
Classeii
Ramii
200000um aa 50
10000buuee aidTass s/a
200000 Ta pouui (1L4))
1000 (1)) zia
18,2 on
12:00 aaua als
02.22 usar
28.56 uoup 3..3aamm 969 adu
25.07
Miae (1-213
GA
is
E.spane MU
96.5 Lam (2.4))
ML1R (8.6))
50 18 (2.04)
1.00
Retmis FF
5.02 Jalius 14,55)
12 12
14. 9,83 (0.41)
10 cl 12.82 (3.80)
10 00.57 eap 11
12230
ML.M0 ae Bi- 8N Mise (1.98)
Mi 4i
25.20
6.00. 15.60 60
91.90 Loca 1200
7. sgaaa MI
6 Ma 156
13 1a 75)
1.50
Table A9: MU performance on (ResNet-50, CIFAR-1O) following the format of Table A8.)
MUU
Drssn
UA
50% Spansit
Dessc
MiX-Bllaacy
50% Sparsit
DENiE
RA
553 Spursity
Dowsse
028 Spensity
XTE (184))
Classisiee Beggtring
Kerrin GA
100 ds
110 Wiliiiee ao
100/0
100 10ii0l
4i 1itee on
St 16.22 (8.75) alu
58-48 a0de E0)
98.00uas 60.10 ala
96,30oos 100
58 98.44. sa
98 (0 (8.00)
87 S7 (6.8L)
$7.50 (6
6.30
Lopeass MU
56.46 Mauue 64)
100 10 an (5.03)
1o Wil eu (0.99)
92 88 (1.83)
6.005
Randonn feggtiig
Kerrin
10100
11 en
43 2i
166
anaa
0A
242 (2.29)
6 6.7a 03,22)
6.20 26.20)
& TBunc (4.00) alu
00 20, (3.50)
00.4 00.4 (3,20)
50 10.l2aaz (3.90
50.4T (4.29)
6.82
Eupaase MU
0. (3.22)
12.29 Jouze (1 300
97 12 au (3.48)
1L (2.90)
6.10
D Broader Impacts and Limitations
Broader impacts. Our study on model sparsity-inspice MU provides a versatile solution to forget arbitrary data points and could give general solution for dealing with different concerns, such as the model's privacy, efficiency, and robustness. Moreover, the applicability of our method extends beyond these aspects, with potential impacts in the following areas. ① Regulatory compliance: Our method enables industries, such as healthcare and finance, to adhere to regulations that require the forgetting of data after a specified period. This capability ensures compliance while preserving the utility and performance of machine learning models. @ Fairness: Our method could also play a crucial role in addressing fairness concerns by facilitating the unlearning of biased datasets or subsets. By removing biased information from the training data, our method contributes to mitigating bias in machine
21

---

learning models, ultimately fostering the development of fairer models. @ ML with adaptation and sustainabillit: Our method could promote the dynamic adaptation of machine learning models by enabling the unlearning of outdated information, and thus, could enhance the accuracy and relevance of the models to the evolving trends and dynamics of the target domain. This capability fosters sustainability by ensuring that ML models remain up-to-date and adaptable, thus enabling their continued usefulness and effectiveness over time.
Limitations. One potential limitation of our study is the absence of provable guarantees for fi-sparse MU. Since model sparsification integrated with model training as a soft regulariza- tion, the lack of formal proof may raise concerns about the reliability and robustness of the approach. Furthermore, while our proposed unlearning framework is generic, its applications have mainly focused on solving computer vision tasks. As result, its effectiveness in the domain of natural language processing (NLP) remains unverified. This consideration becomes particularly relevant when considering large language models. Therefore, further investigation is necessary for future studies explore the applicability and performance of the framework in NLP tasks.
22


Reviews:
[{"review_id": "JyAsNNFA2c", "reviewer": "Reviewer_MsfJ", "summary": "The paper proposes that model sparsity leads to models that are easier to \"unlearn from\". The authors discuss in depth the technical measures that are and that should be used to evaluate various methods of unlearning, and suggest and demonstrate that sparsity is an effective tool in boosting these measures across a wide variety of unlearning applications.", "questions": "1. Why were approximate methods in eps-delta not compared? If the authors strongly feel that this is out of scope I think it needs to be adequately justified.\n2. Classwise and random-data methods were primarily evaluated; why not individual samples?\n3. Gradient norms are often used to evaluate removal success, any reason why there were excluded?\n4. Some main paper experimental results were limited to 95% sparsity, and there are a large number of references to the appendix with additional results. \n\nMinor:\n1. Depending on how the authors treat the approximate/DP/eps-delta, those might be included in the \"approximate MU methods\" in Section 2; I was concerned that this highly relevant work was not mentioned as I was reading through.\n2. The metrics described in Section 2 are very similar to the \"read-out functions\" in [12], might be worth mentioning/referencing.\n3. It could be helpful to clearly indicate \"higher is better\" \"lower is better\" for the various metrics, perhaps using simple up or down arrows, in the text and and in the tables. I do appreciate the authors detailed discussion of results however, careful reading covers all bases. Just thought it may help people skimming. ", "limitations": "1. It's unclear if sparsity-promoting methods help unlearning moreso than other methods that improve model performance generally.\n2. A reasonable set of related work in the form of $(\\epsilon,\\delta)$ forgetting is largely left un-evaluated.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper strongly supports its main claim, with extensive discussion and experimental evaluation that shows beyond much doubt that model sparsity leads to easier and stronger unlearning results in practice.\n\nThe writing is extremely clear and all internal referencing and definitions help readers easily follow and understand the motivations and results. I particularly appreciate the use of emphasis and acronyms, and the way in which the paper makes it easy to flip back and forth to find where a term was defined or where a term is used.\n\nWeaknesses and questions below notwithstanding, the paper is quite solid albeit with a narrow scope. I appreciate the time and detail that went into the experimental evaluations.\n", "weaknesses": "I have two main concerns:\n\n1) The authors don't seem to discuss or acknowledge that sparsity seems to lead to simply better (read: better generalizing) models, and as such we would expect better generalization to lead to less dependence on specific subsets of the training data, and thus easier unlearning. In essence, how does sparsity as a proxy for generalization aid unlearning more than other generalization methods? I would of course expect that strong regularization would affect performance, but perhaps other methods for better generalization that maintain performance may also \"simplify machine unlearning\".\n\n2) My other main concern with this work is the sidelining of the body of work on $\\epsilon-\\delta$ forgetting. The authors reference this work in their Related Work section (probabilistic, DP), but there are many places within the main text up to that point where I was left wondering how those approaches may stack up. Simple methods such as Guo et al.'s [54] seem like an easy enough place to do a quick comparison. I acknowledge that those methods tend to depend heavily on hyperparameter choices (as stated by the authors), but a lot of that work seems highly relevant. The discussion in Section 2 around Proposition 1 is very suggestive and brings to mind work in Sekhari [57], and this citing paper seems to be trying to solve a similar problem as the authors here with approximating the Hessian inverse:\nDeep Unlearning via Randomized Conditionally Independent Hessians. Ronak Mehta, Sourav Pal, Vikas Singh, Sathya N. Ravi. CVPR 2022.\n(not major: I wonder if updating a subset of parameters is \"similar\" in some form to a sparsity approach?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer MsfJ for acknowledging the contributions, soundness, and presentation quality of our paper. And greatly appreciate Reviewer MsfJ for proposing these insightful questions. Below, we provide our responses to the comments, denoted by **[W]** for weaknesses and **[Q]** for questions.\n\n**Response to W1:**\nAs shown in L162-165 and Fig. 2, sparsity indeed benefits model generalization, particularly when using iterative magnitude pruning (**IMP**). However, we refrain from concluding that better generalization simplifies and improves machine unlearning (**MU**), as the method of achieving generalization improvement strongly influences MU performance. For instance, in the comparison of pruning methods shown in Fig. 4, IMP exhibits the best generalization performance compared to other pruning methods (OMP and SynFlow) but leads to the worst unlearning accuracy (with the largest gap from Retrain). The reason is IMP's strong dependence on the forgetting dataset, as stated in L224-225 and L252-255.\n\nThus, generalization improvement alone may not be a precise indicator for easier unlearning; it depends on the approach used to achieve generalization. To further substantiate this point, we performed additional experiments using Sharpness-aware minimization (**SAM**) [R1] during model training to enhance generalization before unlearning; see **Tab. R4** in the attached PDF. We did not observe a significantly reduced performance gap between FT and Retrain (against the former's variance) when compared to empirical risk minimization (**ERM**) training.\n\nWe propose that *if the generalization improvement method does not rely on additional dependence on the forgetting dataset, it could serve as an indicator for easier unlearning*. In such cases, improved generalization may suggest that the model suffers less from spurious correlations [R2] in the training data, potentially aiding in unlearning, as suggested by the reviewer. However, a more comprehensive investigation is warranted. This question poses valuable insights for future research.\n\n**Response to W2 & Q1:** First, we will include a discussion on $\\epsilon-\\delta$ forgetting in Sec. 2. We will highlight its connection with influence unlearning (**IU**). Recent unlearning works [Sec 2.2, R3], [Sec 5.1, R4] have considered $\\epsilon-\\delta$ forgetting as part of IU.\n\nSecond, our focus in this paper is on efficient approximate unlearning on pre-trained models. However, the MU approach in [R5] requires modifying the model training pipeline and integrating it into the certified data removal process (Algorithm 1, 2 of [R5]). In addition, their MU paradigm is limited to linear classifiers or linear probing, which only updates the linear classification head for DL models. This setup differs from ours, where we investigate unlearning on the full DL model. This limitation was also noted in [Tab. 4, R6; Related work, R7]. Upon reviewing the implementation code of [R5], we found that even in the case of linear probing, they considered binary classifiers, rather than the prediction head ResNet used. Considering these factors, [R5] may not be an ideal candidate for efficient unlearning comparison, although we are happy to provide further elaboration on the distinctions in the paper.\n\nThird, we sincerely appreciate your suggestion to consider reference [R8]. It indeed provides relevant insights into MU. In that work, they utilized a portion of the parameters to approximate the inversion of the Hessian matrix, enhancing the Hessian-based (IU) method. In contrast, our study focuses on revealing a crucial factor, weight sparsity, which impacts various MU methods. Our research encompasses both practical and theoretical aspects (Sec. 3 & Sec. 5), novel MU methods (Sec. 4), and emerging MU applications (Sec. 5.2).\n\n**Response to Q2:** We chose not to consider individual samples for unlearning due to several compelling reasons.\n\nFirst, we did not explore unlearning individual samples, as it can lead to substantial variance in unlearning performance depending on the selected forgetting sample. For instance, the UA for an individual sample would be either 100 or 0, resulting in significant variability that hampers meaningful comparisons across different settings or methods.\n\nAdditionally, we performed a literature review to validate the prevalence of class-wise and random data forgetting as primary unlearning settings. Supporting evidence for our approach can be found in [Tab. 1, R9], [Fig. 2, R7] for class-wise forgetting, and [Tab. 1, R9] along with [Sec 5.1, R10] for random data forgetting.\n\nLastly, we focus on random data forgetting and class-wise forgetting due to their direct relevance to the applications discussed in our paper. The former is aligned with the use case of model cleansing, while the latter is particularly relevant for enhancing transfer learning performance.\n\n**Response to Q3**: We admit that *gradient residual norms* (**GRN**) could be a useful metric for evaluating MU methods. Although this was introduced by [R5] to offer insights into approximation errors, we exclude it due to the following reason.\n\nExisting unlearning metrics center around three primary aspects: 1) efficiency (RTE); 2) fidelity (RA, TA); 3) efficacy (UA, MIA-Efficacy). In relation to GRN, it unveils the convergence of model retraining over the retained dataset, making it akin to a fidelity metric similar to RA (remaining accuracy). As evidenced in [Fig. 5, R8], GRN exhibits a closely aligned trend with RA under the same architecture, albeit with greater variance. Hence, we opt for RA to provide a more intuitive performance measure.\n\n**Response to Q4:** We will move more results from the appendix to the main paper in our revised version.\n\nFor the response to the minor questions, please refer to the **Supplement Response to Reviewer MsfJ** part of the General Response."}, {"review_id": "0l9gILoXgT", "reviewer": "Reviewer_hHiz", "summary": "This paper studies the machine unlearning problem from the perspective of model sparsification. Specifically, the paper proposes two types of model sparsification methods: data-independent (e.g., OMP) and data-dependent (e.g., sparsity-aware unlearning). An extensive set of simulations are shown in the paper to validate the unlearning performance for the sparse models.", "questions": "Overall this is a good empirical paper and I do not have questions about the simulations. Please see the weakness section for the concerns on methodology.", "limitations": "I think the biggest limitation is that it would be hard to perform theoretical analysis for the update rule Eq (3) like those influential function-based methods which require the loss to be strongly convex. Also, the current simulations are all on computer vision tasks with CNN-based models, which limits the application domain. CV models are known to be redundant and remember training samples within coefficients, so it would be good to try other tasks in different domains.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper performs a comprehensive empirical study on the influence of model sparsity on the unlearned model performance, which is an intuitive but less explored direction in machine unlearning literature. The authors give some theoretical insights on how model sparsity will affect the \"unlearning error\" in Proposition 2, although the loss there would be restricted to be convex and the unlearning mechanism is gradient ascent. For the simulations, different types of unlearning methods are evaluated with the same pipeline to study the unlearned model performance under dense and sparse regimes. Furthermore, the authors show two possible applications of machine unlearning, namely data cleaning and transfer learning, where the model sparsity may also help with the final model performance.", "weaknesses": "Although this paper provides extensive simulations to show that sparse models can help increase the unlearning accuracy and the MIA efficacy while maintaining good remaining accuracy and testing accuracy, it still does not fully answer the very basic question \"How to decide the level of model sparsity?\" It is obvious that when we set the sparsity to 100%, the unlearned model should perform perfectly on UA and MIA-Efficacy, so there is no surprise that they should still perform well at 95% sparsity. As for the good performance on RA and TA, such observations have already been well studied in previous literature on lottery ticket hypothesis and model compression. So if one wants to incorporate sparse models for unlearning purposes, the first and most important question would be the sparsification level. Unlike previous problems where a sparse model is good when it performs well on the training statistics, in unlearning we do not know the forget set in advance, so we are not able to decide how well a sparse model will behave beforehand regarding the UA and MIA-Efficacy. The authors try to answer this question via the sparsity-aware fine-tuning, but again it falls back to decide the trade-off coefficient $\\gamma$ and the current regularization scheduler is pure heuristic with little insights. Nevertheless, I understand that this is a challenging question even in general LTH problems and there would be no easy solution for that, especially in the context of unlearning.\n\nAs for the presentation of the paper, sometimes the notations or the terms are introduced without explanations. For example, in Proposition 1, the term $\\mathbf{1}/N$ is used without explaining that $\\theta_o=\\theta(\\textbf{1}/{N})$; in Proposition 2, what does the learning error $e(\\mathbf{m})$ really means? Also, Proposition 1 does not seem to have any relationship with the remaining content and it is just a reformulation of the results in previous literature, so maybe it can be moved to the appendix to save space for the figures and tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer hHiz for providing a detailed summary of our strengths. Below, we present our detailed responses to the comments, indicating **[W]** for weaknesses and **[Q]** for questions.\n\n**W1:** How to decide the level of model sparsity? And how to decide the trade-off coefficient $\\gamma$?\n\n**Response to W1:** Thank you for posing these insightful questions. As noted by Reviewer hHiz, *\"Nevertheless, I understand that determining the best sparsity level is a challenging question even in general LTH problems.\"* Indeed, identifying the optimal sparsity level for unlearning can be intricate, especially considering variations across datasets and architectures. For instance, similar to LTH, our empirical findings indicate that the optimal sparsity level to improve the efficacy of MU on ImageNet is approximately 80%, different from the 95% observed for CIFAR-10. While the optimal sparsity level for MU may differ, we possess some general intuition to aid in sparsity selection. The main criterion is to pinpoint a sparsity level that improves the unlearning efficacy while maintaining the generalization performance comparable to that of the original model.\n\nFurthermore, the sparsity-aware unlearning method (Eq. 3) could help circumvent the imposition of a strict threshold on model sparsity. As clarified in Lines 277-279, the optimal choice for the tradeoff coefficient $\\gamma$ tends to align with a decaying scheduler, as indicated by the minimized unlearning performance gap with Retrain in Tab. 2. This schedule underscores the advantage of emphasizing sparsity enhancement during the initial stages of unlearning, gradually transitioning to heightened attention on refining fine-tuning accuracy over the retained dataset.\n\n**W2:** In Prop. 1, the term 1/N is used without explaining. In Prop. 2, the term e(m) lacks explanation.\n\n**Response to W2:** Thank you for the careful reading. $w=1/N$ signifies the uniform weights employed for Empirical Risk Minimization (**ERM**) training. Thus, $\\theta(1/N)$ pertains to the original model trained via ERM, as elaborated in Line 87.\n\nRegarding $e(m)$ in Prop. 2, it pertains to the unlearning error when comparing the Gradient Ascent (**GA**)-based unlearning with the retrained model under the model sparsity mask $m$, see Eq. (A12) in Appendix B. When $m=1$ (no pruning involved), this concept was initially introduced in [Sec. 5.1, R11] to calculate the reverse GA steps that need to be reintegrated into the trained model for unlearning.\n\n**W3:** Prop. 1 is just a reformulation of the results in previous literature, moving to the appendix.\n\n**Response to W3:** We will move Prop. 1 to the Appendix. However, there's a specific reason behind the detailed exposition of IU (Prop. 1) in the initial submission. Our derived IU approach exhibits a minor yet crucial distinction from existing methods in the literature, such as [Eq. 1, R5] and [Eq. 7, R12]. As outlined in Lines 132-134, our work has accounted for the normalization effect of data influence weights ($\\mathbf 1^T \\mathbf w = \\mathbf 1$) during the IU approach derivation. In practical terms, we have observed that IU with weight normalization outperforms existing IU methods, given their sensitivity to hyperparameter tuning.\n\n**Limitations:** The current simulations are all on computer vision tasks with CNN-based models.\n\n**Response to Limitations:** Thank you for your insightful suggestions. We've included an additional experiment in our study, focusing on the application of Swin Transformer to the CIFAR-10 dataset. This new experiment is presented in **Tab. R5** of the attached PDF. To facilitate a comparison between the assessed approximate unlearning methods (including the FT baseline and the proposed $\\ell_1$-spare MU) and Retrain, we train the transformer from scratch on CIFAR-10. This could potentially result in a decrease in testing accuracy when compared with fine-tuning on a pre-trained model over a larger, pre-trained dataset.\n\nIn Tab. R5, the results are noteworthy: Substantial enhancements were observed using our proposed $\\ell_1$-spare MU, leading to a much smaller performance gap with Retrain compared to FT. In particular, class-wise forgetting exhibited a remarkable 90.24% increase in UA, accompanied by a slight reduction in RA.\n\nMotivated by this comment, we will also explore the application of our approach to language models in the future. We will incorporate this aspect into our Conclusion and Limitations sections."}, {"review_id": "QsW2VshsPd", "reviewer": "Reviewer_r13y", "summary": "The authors propose to consider network sparsification as a way to improve machine unlearning (MU), the task of unlearning evidence of a given set of training samples from a neural network. In particular, they consider LTH based pruning schemes as well as a regularized training loss to complement standard MU approaches and show that this enhances the unlearning process in a variety of benchmarks and evaluation metrics. For gradient ascent-based MU, they further show an improvement in an error bound with mask-based sparsification, such as LTH pruning. Overall, this paper introduces NN sparsification to the MU domain and shows the effectiveness of this combination on standard benchmarks.\n\nEDIT: After the fruitful discussion and additional experiments, I raise my score to 7 (Accept).", "questions": "Given the focus of this work on the experiments (i.e., what is the impact of sparsification beyond GA approaches), my main concerns are around the results (tables are by the way very tiny), my questions are in order of appearance.\n\n1.\tThe bottomline of Tab.2 drawn in the paper is that the “linearly decreasing gamma scheduler outperforms other schemes”, which I do not see, as neither approach is consistently better in all metrics, for example UA and MIA constant gamma is better. How do you explain this difference and what are the potential effects?\n2.\tTable 3 is broadly summarized in 5.2 as generally showing improvement of metrics through sparsification. Yet, for random data forgetting we see that the sparsification partially reduces UA and MIA, depending on the method and especially MIA is bad. For SVHN, UA and MIA completely break down. For CIFAR100, it becomes evident that we essentially have a strong trade-off between RA,TA and UA,MIA performance between original model and sparsified model. The paper would greatly benefit from insights and critical discussion of these results.\n3.\tOne of the biggest issues I have is that sparsity-regularized unlearning does not appear in any of these experiments as comparison. This is highly suspicious, as this was introduced in this paper as another approach of model sparsification. Given the limited success of l1-based pruning in the LTH field, I would really like to see how this performs here. Moreover, the results in Fig.5 are a comparison against FT, claiming to beat FT in terms of UA and MIA. But FT was literally described as trading of UA and MIA for better RA and TA in the sparse regime before and hence is likely the easiest to beat. Please properly compare your method against all other methods on the considered benchmarks.\n4.\tTrojan model cleanse: Again, why is sparsity-aware unlearning not in the comparison? What would be its performance?\n5.\tWhy do you not compare to finetuning with LTH sparsification?\n6.\tProof of Prop. 2: I do not fully understand how you build the “diagonal of the mask m” after equation A12 so that diag(m)theta actually corresponds to the original mask hadprod theta. I feel this is an essential step from the known proof towards your proposition. Could you please explain in more detail?\n\nMinor:\n\n1.\tFigure 1 is not instructive, from the visualization it is not clear what the left part ‘Data’ has to do with the rest. It is also unclear how pruning and unlearning are combined here, it is an overly simplistic prictorial of the title. I would suggest removing it to gain space for a proper discussion of the results.\n2.\tIn proposition 1, L has not been introduced before.\n3.\tSometimes you reference your methods as sparsity-regularized unlearning, sometimes sparsity-aware unlearning. Please be consistent.\n", "limitations": "As indicated by my questions, the limitations of this paper are not properly discussed in terms of the obtained results. In particular, differences in trends within a benchmark and across different datasets are not discussed, as well as the proposed regularized approach not compared to the prune-and-unlearn approach, which raises some concerns about limitations.\nThat being said, there is no free lunch, so a *critical* comparison and *appropriate discussion* of results and limitations answering my raised concerns would benefit the paper and consequently improve my rating.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "To the best of my knowledge, this is the first attempt to combine the recent advances of networks sparsification with the field of machine unlearning and hence presents itself as an original work.\nThe paper is clearly written and organized, which greatly eases the reading experience. The experiments are on a variety of tasks, benchmark datasets and models, which mostly support the success of combining model sparsity approaches for machine unlearning. To evaluate the successful unlearning apart from standard metrics, the authors propose a membership inference attack-based metric that allows a proper evaluation of how well a model forgot a given sample, which is an interesting and intuitive way to measure MU success.\n", "weaknesses": "The overall idea of combining sparsification with MU is not necessarily innovative, especially given that it is currently applied in almost any subfield given the hype around the LTH. While this itself is not bad---often the simplest ideas can make a lasting impact---I feel the authors make their life a bit too easy by considering all MU approaches at once and speaking about general statements that often do not hold across datasets and the breadth of presented results. Especially since the focus is on the extensive results rather than theoretical insights, I would expect a much more faceted discussion, with the different effects of sparsification on different MU approaches that is evident from the tables (see Questions). Similarly, I would expect sparsity-aware unlearning, as the suggested algorithm of this paper, to be compared to in all experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the very insightful comment. Below, we provide our point-to-point responses to the comments, denoted by [W] for weaknesses and [Q] for questions.\n\n**W1:** The overall idea of combining sparsification with MU is not necessarily innovative.\n\n**Response to W1:** We are sorry to hear about the lack of novelty regarding our work. Please allow us to make the following clarifications.\n\n1. As Reviewer r13y pointed out in the strengths section *\"To the best of my knowledge, this is the first attempt to … hence presents itself as an original work.\"*, our work offers an innovative concept to probe the effect of sparsity on unlearning performance, which is substantiated by our thorough investigation across various approximate unlearning methods and metrics.\n2. We respectfully disagree that *\"our life is made a bit too easy by considering all MU approaches at once and speaking about general statements that often do not hold across datasets\"*. It's important to note that we measure unlearning performance based on the smallest performance gap with the gold-standard retrained model (Retrain), as emphasized in Line 145 and indicated by blue numbers in Tab. 2 and 3. Our findings clearly demonstrate that sparsity indeed plays a pivotal role in reducing the unlearning gap across datasets (further elaborated in our response to the next question). Furthermore, our contribution goes beyond incremental advancements in the model pruning (or LTH) field. Our investigation into \"Why sparsity for MU\" encompasses crucial aspects:\na. We provided theoretical insights into the benefit of model sparsity (Prop. 2).\nb. We showed that what is best for LTH (IMP) does not necessarily lead to the best unlearning performance and the pruning criteria for MU need to be carefully examined (Lines 220-257 and Fig. 4).\nc. The proposed sparsity-aware unlearning is also a great improvement in integrating sparsity with MU. The above, together with our extensive empirical studies, clearly shows the novelty of our work.\n\n**Response to Q1:** Thanks for raising this question. However, we believe this is caused by a misunderstanding of the \"good\" unlearning performance. As previously mentioned in the literature [R4, R9] and emphasized within our paper (e.g., Line 145, Tab. 3, and Fig. 3), the effectiveness of MU is gauged by its performance is closer to that of the gold-standard retrained model (Retrain). We invite the reviewer to revisit Tab. 2 for a detailed comparison. The decaying schedule, as evident in the table, results in the smallest performance gap with Retrain across the MU metrics highlighted in blue: (0.06, 0.41, 2.61, 3.16).\n\n**Response to Q2:** Given our prior explanation regarding Q1, we invite the reviewer to reassess the outcomes in Tab. 3. To facilitate a more accessible comparison, we extended it to **Tab. R3**. This updated table introduces a consolidated metric termed \"Disparity Average,\" which essentially computes the average of the performance gaps between each unlearning method and Retrain across all metrics. By consulting this metric, it becomes evident that sparsity consistently yields advantages for different MU methods.\n\n**W2:** I expect sparsity-aware unlearning to be compared in all experiments.\n\n**Response to W2, Q3, Q4:** Thank you for raising these questions. We apologize for any confusion regarding the experiments on sparsity-aware unlearning.\n1. We opted not to directly juxtapose \"sparsity-aware unlearning\" with \"prune first, then unlearn\" due to the fact that the former is employed on a dense model (unlike OMP, L1 regularization doesn't involve a hard thresholding operation on model weights), while the latter operates on a sparse model. Given the distinct initial models for MU, we were cautious about the fairness of such a comparison. Nonetheless, in response to the reviewer's query, we have extended Tab. 3 to Tab. R3 to encompass the performance outcomes of \"sparsity-aware unlearning.\"\n2. The reason for only comparing sparsity-aware unlearning with FT in Fig. 5 lies in the fact that the objective function used in sparsity-aware unlearning is specified by the fine-tuning (FT) loss; see Line 254. However, we did extend the scope of comparisons to include sparsity-aware unlearning vs. Retrain, FT, and IU (influence unlearning) in Tab. A6 (Appendix). We refrained from incorporating GA due to its substantial generalization drop. For a complete comparison, including GA, please refer to **Tab. R6**.\n3. In Fig. 6, the omission of sparsity-aware unlearning in model cleansing stems from our intention to demonstrate the unlearning performance and the backdoor attack success rate for different models' sparsity levels. Since sparsity-aware unlearning is applied to a dense model (lacking the hard thresholding for model weight sparsity), it contributes just a single data point at sparsity = 0% in Fig. 6. **Fig. R1** includes the absent model cleansing performance using sparsity-aware unlearning. Clearly, it can also effectively remove the backdoor effect while largely preserving the model's generalization.\n\n**Response to Q5:** We would like to clarify that Fig. 4 has a comparison within the context of the LTH regime. Specifically, the label **Dense** signifies the FT-based model unlearning method applied directly to the dense model, while **IMP** represents the pruning approach recommended by LTH. It's worth noting that, in the figure, even though IMP exhibits enhanced generalization performance, it does not necessarily translate to improved unlearning efficacy (measured by UA and MIA-efficacy) in comparison to the 'Dense' method. In contrast, the OMP and SynFlow pruning techniques, which exhibit reduced reliance on training data, demonstrate significant improvements in model unlearning. Further details regarding this analysis can be found in Lines 220-257.\n\nFor the response to the Q6 and minor questions, please refer to the **Supplement Response to Reviewer r13y** part of the General Response."}, {"review_id": "pIntq1EXoH", "reviewer": "Reviewer_8xJa", "summary": "This paper considers the benefit of leveraging sparsity to improve standard unlearning techniques. They empirically verify that across a wide range of datasets and architectures. the sparsity benefits unlearning. ", "questions": "1. My main question is, do the readers have an intuition for when sparsity helps or hurts the unlearning process? Across several of the experiments, while it is true that sparsity generally improves unlearning, there are several cases where this is not the case. Are there patterns the authors noticed? For a practitioner, I think a deeper understanding of this is helpful.\n2. It seems that most of the architectures are restricted to convolution-based architectures. Have the authors tried extending this to other architecture, such as transformers?", "limitations": "I would have liked it if the authors had recognized the slight inconsistency of their empirical results. However, on the whole, the limitations are addressed. ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The recap of this paper on the unlearning literature and metrics is very thorough and helpful for readers who are more unfamiliar with the setting of unlearning. \n2. The intuition of this paper is quite elegant and helpful. The address of different pruning methodologies is insightful.\n3. The empirical results are seemingly quite promising. \n\nOverall, this paper uses a simple idea to yield strong benefits. While a theoretical analysis would have been nice, the paper is strong as it is. I vote for acceptance strongly then. \n\nI also believe this work is missing a citation. Pruning has also been shown in in its relation to generalization error (see \"Generalization Bounds for Magnitude-Based Pruning via Sparse Matrix Sketching\"). This, however, does not affect my score. ", "weaknesses": "1. There is a slight inconsistency with the empirical results. There are times when sparsity significantly hurts the unlearning. There does not seem to be an observable pattern to this. I believe such an analysis is necessary.\n2. Is it possible for the authors to conduct a small experiment to see how the effect of sparsity on unlearning scales with model size? For example, with varying Resnet sizes, does the effect of sparsity on unlearning change? I think this is a slight concern for me.\n3. There are no theoretical analyses here. However, I won't be too harsh about this as the paper does mention this in the limitations. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful comments that precisely recognize the strengths of our work. We are particularly excited about citing the referenced paper that establishes a connection between model pruning and generalization. Below, we offer our detailed responses to your comments, categorized by **[W]** for weaknesses and **[Q]** for questions.\n\n**Response to W1, Q1**: Thank you for raising these questions.\n\nAs previously indicated in the literature [R4, R9] and highlighted within our paper (e.g., Line 145, Tab. 3, and Fig. 3), the effectiveness of MU is assessed by its performance gap to the gold-standard retrained model (Retrain). Following this criterion, our experimental results (e.g., Tab. 2 and 3) consistently demonstrate that incorporating sparsity enhances unlearning efficacy (UA, MIT-Efficacy) and narrows the performance gap (i.e., the numbers marked by the blue color in Tab. 2 and 3) between approximate unlearning and Retrain. In **Tab. R3** presented in the attached PDF, we illustrate the averaged unlearning performance disparity against Retrain across various methods. The advantage of sparsity in unlearning is evident. On the other hand, we acknowledge that an excessively aggressive sparsity choice could potentially compromise generalization performance and/or remaining accuracy (RA). This tradeoff is akin to the one encountered in model pruning alone. Nevertheless, with an appropriate sparsity level or the integration of soft sparsity-aware regularization, we can achieve significant gains in unlearning efficacy without substantially sacrificing generalization.\n\n**Response to W2:** Thank you for your suggestion. In response, we conducted additional experiments involving both Resnet20s and Resnet50 on CIFAR-10, in addition to ResNet-18 in the paper (Tab. 3). The outcomes of these experiments are detailed in **Tab. R1** and **Tab. R2** of the attached PDF. In these new experiments, we assess the performance of both the \"prune first, then unlearn\" approach and the \"sparsity-aware unlearning\" technique (note that the latter is applied to the dense model rather than a predefined sparse model). During the rebuttal period, we opted to exclude the IU-based unlearning baseline due to its considerable computational demands and the challenge of tuning hyperparameters for optimal MU performance. Notably, our results reveal that across different model sizes, sparsity consistently diminishes the unlearning gap with Retrain (indicated by highlighted blue numbers, where smaller values are preferable). It's worth noting that while both ResNet20s and ResNet50 benefit from sparsity, the suggested sparsity ratio is 90% for ResNet20s and slightly lower than 95% for ResNet50 when striking the balance between MU and generalization.\n\n**Response to W3:** Thank you for your valuable feedback. We acknowledge that while Prop. 2 theoretically demonstrates the reduction in unlearning error of gradient ascent-based unlearning with the presence of model sparsity, this result does not universally apply to all the unlearning methods we examined, despite their promising empirical performance. To address this, we will incorporate a discussion of this limitation in the Limitations section of our paper (Appendix D).\n\n**Response to Q2:** Thank you for your insightful suggestion. We've included an additional experiment in our study, focusing on the application of Swin Transformer to CIFAR-10. This new experiment is presented in **Tab. R5** of the attached PDF. To facilitate a comparison between the assessed approximate unlearning methods (including the FT baseline and the proposed $\\ell_1$-spare MU which also uses the FT loss as the unlearning objective function) and Retrain, we train the transformer from scratch on CIFAR-10. This could potentially result in a decrease in testing accuracy when compared with fine-tuning on a pre-trained model over a larger, pre-trained dataset.\n\nIn Tab. R5, the results are noteworthy: in both class-wise forgetting and random data forgetting contexts, substantial enhancements were observed using our proposed $\\ell_1$-spare MU, leading to a much smaller performance gap with Retrain compared to FT. In particular, class-wise forgetting exhibited a remarkable 90.24% increase in UA, accompanied by a slight reduction in RA.\n\nMotivated by this comment, we will also explore the application of our approach to language models in the future. We will incorporate this aspect into our Conclusion and Limitations sections."}]

Author rebuttal:
Dear Reviewers, ACs, and PCs:

We are glad to receive valuable and constructive comments from all the reviewers. We have made a substantial effort to clarify reviewers' doubts and enrich our experiments in the rebuttal phase. In our responses, **Tab. R**xx or **Fig. R**xx refers to the new **R**ebuttal results in the attached PDF. And **Tab. A**xx or **Fig. A**xx refers to the existing results in **A**ppendix. Below is a summary of our responses:

**Reviewer [MsfJ](https://openreview.net/forum?id=0jZH883i34&noteId=JyAsNNFA2c):**
1. We endeavored to elucidate the relationship between model generalization and unlearning efficacy, supplemented by additional experiments (**Tab. R4**) to substantiate our viewpoint.
2. We clarified the importance of ε - δ forgetting and the rationale behind not using it as a comparison method in our work.
3. We provided an explanation for our decision not to use individual samples as an unlearning setting.
4. We provided clarification regarding our decision not to use the gradient residual norm as an evaluation metric.

**Supplement Response to Reviewer [MsfJ](https://openreview.net/forum?id=0jZH883i34&noteId=JyAsNNFA2c):**

- **Response to Minor Q1:** Thank you for bringing this to our attention. We will enhance Sec. 2 by offering a more comprehensive discussion about the methodologies of approximate/DP/$\epsilon-\delta$. Furthermore, we will explicitly elucidate why these methods were not incorporated in our study, providing clear explanations that align with the responses we have already given for W2 and Q1.
- **Response to Minor Q2:** Thank you for pointing this out. We will mention this in revision.
- **Response to Minor Q3:** Thank you for bringing this to our attention. As highlighted in Lines 144-146, an unlearned model exhibiting performance closer to Retrain should be deemed superior. This explains our practice of presenting the performance gap with Retrain in blue within our results (e.g., Tab. 2 and Tab. 3). Consequently, drawing a simple conclusion that higher values are always superior, or vice versa, might not accurately capture the scenario.

**Reviewer [hHiz](https://openreview.net/forum?id=0jZH883i34&noteId=0l9gILoXgT):**
1. We have provided further clarification on how we determine the pruning ratio and the parameters of sparsity-aware unlearning.
2. We included an experiment on MU using Swin Transformer (**Tab. R5**).

**Reviewer [r13y](https://openreview.net/forum?id=0jZH883i34&noteId=QsW2VshsPd):**
1. We provided extra clarifications regarding our contributions.
2. We further clarified the evaluation metrics and improved the presentation of the main table (**Tab. R3**).
3. We extended the main tables to include a more thorough comparison of the unlearning performance between the sparsity-aware unlearning and the "prune first, then unlearn" approach (**Tab. R3 and R6**).
4. We included sparsity-aware unlearning on the Trojan model cleanse application (**Fig. R1**).

**Supplement Response to Reviewer [r13y](https://openreview.net/forum?id=0jZH883i34&noteId=QsW2VshsPd):**

- **Response to Q6:** Thank you for pointing out this. We used the following facts: $diag(m) = [ m_1, 0, …, 0; 0, m_2, 0, …, 0; …; 0, …, 0, m_d ]$ is a diagonal matrix with the diagonal line given by the d-dimensional vector $m$, and thus the matrix-vector product yields $diag(\mathbf m)\boldsymbol \theta = \mathbf m \odot \boldsymbol \theta $, where $\odot$ is the element-wise product. We will make this clearer in our revision.
- **Response to Minor Questions:** We appreciate your suggestion regarding the visual representation in Fig. 1. We will certainly consider your recommendation to remove the teaser figure and provide a more detailed results analysis. Regarding Prop. 1, "L" corresponds to the empirical risk minimization loss function, as indicated in Line 123. Additionally, we will make sure that our references to "sparsity-aware unlearning" remain consistent throughout the paper.

**Reviewer [8xJa](https://openreview.net/forum?id=0jZH883i34&noteId=pIntq1EXoH):**
1. We provided explanations on the observed inconsistency in our results and revised the main table (**Tab. R3**).
2. We included additional experiments on the ResNet-20 and ResNet-50 architectures (**Tab. R1, R2**).
3. We conducted an additional experiment on the SwinTransformer architecture (**Tab. R5**).

**References used in authors' response:**

> [R1] Foret et al. Sharpness-aware minimization for efficiently improving generalization. ICLR 2021.
>
> [R2] Sagawa et al. An investigation of why overparameterization exacerbates spurious correlations. ICML 2020.
>
> [R3] Wang et al. Federated unlearning via class-discriminative pruning. WWW 2022.
>
> [R4] Xu et al. Machine Unlearning: A Survey. ACM Computing Surveys 2023.
>
> [R5] Guo et al. Certified data removal from machine learning models. ICML 2020.
>
> [R6] Nguyen et al. A survey of machine unlearning. arXiv 2022.
>
> [R7] Graves et al. Amnesiac machine learning. AAAI 2021.
>
> [R8] Mehta et al. Deep unlearning via randomized conditionally independent hessians. CVPR 2022.
>
> [R9] Golatkar et al. Eternal sunshine of the spotless net: Selective forgetting in deep networks. CVPR 2020.
>
> [R10] Golatkar et al. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. ECCV 2020
>
> [R11] Thudi et al. Unrolling sgd: Understanding factors influencing machine unlearning. EuroS&P 2022.
>
> [R12] Warnecke et al. Machine unlearning of features and labels. NDSS 2021.

Meta-review: I would like to congratulate both the authors and the reviewers for a very productive round of reviews, rebuttals, comments and replies. The paper went initially from an average < 6 to an average of 7.

I personally like the global message carried by the paper.

The authors now have to take the necessary steps to improve the submission, including the experiments, technical content, up to the precision level asked for by reviewers (e.g. r13y).
Decision: Accept (spotlight)