SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A4: MU performance vs. sparsity on the additional architecture (VGG-16 [47]) for both class-wise forgetting and random data forgetting on CIFAR-10. The content format follows Tab. 3.
MU	UA	MIA-Efficacy	RA	TA	RTE
DENSE	95% Sparsity	DENSE	95% Sparsity	DENSE	95% Sparsity	DENSE	95% Sparsity	(min)
Class-wise forgetting, VGG-16
Retrain	100.00±0.00	100.00±0.00	100.00±0.00	100.00±0.00	100.00±0.01	99.97±0.00	94.83±0.10	92.93±0.06	30.38
FT	28.00±8.16 (72.00)	34.94±5.37 (65.06)	63.23±17.68 (36.77)	68.02±12.03 (31.98)	99.87±0.05 (0.13)	99.60±0.08 (0.37)	92.80±1.28 (2.03)	92.96±0.85 (0.03)	1.81
GA	77.51±3.47 (22.49)	83.93±2.14 (16.07)	80.13±4.27 (19.87)	88.04±3.18 (11.96)	96.09±0.13 (3.91)	97.33±0.08 (2.64)	88.80±1.33 (6.03)	89.95±0.78 (2.98)	0.27
IU	88.58±0.86 (11.42)	98.78±0.44 (1.22)	92.27±1.14 (7.73)	99.91±0.05 (0.09)	96.89±0.27 (3.11)	93.18±0.28 (6.79)	89.81±1.01 (5.02)	87.45±0.81 (5.48)	2.51
Random data forgetting, VGG-16
Retrain	7.13±0.60	7.47±0.30	13.02±0.77	13.51±0.50	100.00±0.01	99.93±0.01	92.80±0.17	91.98±0.22	30.29
FT	0.86±0.29 (6.27)	1.46±0.22 (6.01)	2.62±0.47 (10.40)	3.82±0.41 (9.69)	99.76±0.12 (0.24)	99.47±0.11 (0.53)	92.21±0.13 (0.59)	92.03±0.37 (0.05)	1.77
GA	9.11±0.83 (1.98)	6.91±0.96 (0.56)	7.77±1.01 (5.25)	8.37±1.35 (5.14)	93.08±0.93 (6.92)	93.63±1.16 (6.30)	86.44±1.32 (6.36)	89.22±1.59 (4.53)	0.31
IU	1.02±0.43 (6.11)	3.07±0.50 (4.40)	2.51±0.61 (9.51)	6.86±0.67 (6.65)	99.14±0.03 (0.86)	97.35±0.31 (2.58)	91.01±0.29 (1.79)	89.49±0.37 (2.49)	2.78
To demonstrate the effectiveness of our methods on a larger dataset, we conducted additional experiments on ImageNet [46] with settings consistent with the class-wise forgetting in Tab. 3. As we can see from Tab. A5, sparsity reduces the performance gap between exact unlearning (Retrain) and the approximate unlearning methods (FT and GA). The results are consistent with our observations in other datasets. Note that the  $83\%$  model sparsity (ImageNet, ResNet-18) is used to preserve the TA performance after one-shot magnitude (OMP) pruning.
Performance of  $\ell_1$  sparsity-aware MU on additional datasets. As seen in Fig. A2,  $\ell_1$ -sparse MU significantly reduces the gap between approximate and exact unlearning methods across various
Table A5: Performance overview of MU vs. sparsity on ImageNet considering class-wise forgetting. The content format follows Tab. 3.
MU	UA	MIA-Efficacy	RA	TA	RTE
DENSE	83% Sparsity	DENSE	83% Sparsity	DENSE	83% Sparsity	DENSE	83% Sparsity	(hours)
Class-wise forgetting, ImageNet
Retrain	100.00	100.00	100.00	100.00	71.75	69.18	69.49	68.86	26.18
FT	63.60 (36.40)	74.66 (25.34)	68.61 (31.39)	81.43 (18.57)	72.45 (0.70)	69.36 (0.18)	69.80 (0.31)	68.77 (0.09)	2.87
GA	85.10 (14.90)	90.21 (9.79)	87.46 (12.54)	94.25 (5.75)	65.93 (5.82)	62.94 (6.24)	64.62 (4.87)	64.65 (4.21)	0.01
datasets (CIFAR-100 [43], SVHN [45], ImageNet [46]) in different unlearning scenarios. It notably outperforms other methods in UA and MIA-Efficacy metrics while preserving acceptable RA and TA performances, thus becoming a practical choice for unlearning scenarios. In class-wise and random data forgetting cases,  $\ell_1$ -sparse MU exhibits performance on par with Retrain in UA and MIA-Efficacy metrics. Importantly, the use of  $\ell_1$ -sparse MU consistently enhances forgetting metrics with an insignificant rise in computational cost compared with FT, underscoring its effectiveness and efficiency in diverse unlearning scenarios. For detailed numerical results, please refer to Tab. A6.
Class-wise forgetting
SVHN
Class-wise forgetting
ImageNet
Class-wise forgetting
Figure A2: Performance of sparsity-aware unlearning vs. FT and Retrain on class-wise forgetting and random data forgetting under ResNet-18 on different datasets. Each metric is normalized to  $[0, 1]$  based on the best result across unlearning methods for ease of visualization, while the actual best value is provided. The figure format is consistent with Fig. 5.
CIFAR-100
Random data forgetting
SVHN
Random data forgetting
Table A6: Performance of sparsity-aware MU vs. Retrain, FT, GA and IU considering class-wise forgetting and random data forgetting, where the table format and setup are consistent with Tab. 3. The unit of RTE is minutes for all datasets, except ImageNet. For ImageNet, indicated by an asterisk  $(\ast)$ , RTE is measured by hours.
MU	UA	MIA-Efficacy	RA	TA	RTE (min)
Class-wise forgetting, CIFAR-10
Retrain	100.00±0.00	100.00±0.00	100.00±0.00	94.83±0.11	43.23
FT	22.53±8.16 (77.47)	75.00±14.68 (25.00)	99.87±0.04 (0.13)	94.31±0.19 (0.52)	2.52
GA	93.08±2.29 (6.92)	94.03±3.27 (5.97)	92.60±0.25 (7.40)	86.64±0.28 (8.19)	0.33
IU	87.82±3.15 (12.18)	95.96±0.21 (4.04)	97.98±0.21 (2.02)	91.42±0.21 (3.41)	3.25
\( \ell_1 \)-sparse MU	100.00±0.00 (0.00)	100.00±0.00 (0.00)	98.99±0.12 (1.01)	93.40±0.43 (1.43)	2.53
Class-wise forgetting, CIFAR-100
Retrain	100.00±0.00	100.00±0.00	99.97±0.01	73.74±0.19	48.45
FT	26.45±6.29 (73.55)	92.44±15.93 (7.56)	99.86±0.04 (0.11)	74.08±0.23 (0.34)	3.76
GA	81.47±0.32 (18.53)	93.47±4.56 (6.53)	90.33±1.71 (9.64)	64.94±0.74 (8.80)	0.21
IU	84.12±0.34 (15.88)	98.44±0.45 (1.56)	96.23±0.02 (3.74)	71.24±0.22 (2.50)	4.30
\( \ell_1 \)-sparse MU	95.67±0.53 (4.33)	100.00±0.00 (0.00)	98.01±0.02 (1.96)	71.35±0.22 (2.39)	3.79
Class-wise forgetting, SVHN
Retrain	100.00±0.00	100.00±0.00	100.00±0.00	95.71±0.12	42.84
FT	11.48±8.12 (88.52)	86.12±9.62 (13.88)	100.00±0.00 (0.00)	95.99±0.07 (0.28)	2.86
GA	83.87±0.19 (16.13)	99.97±0.02 (0.03)	99.60±0.15 (0.40)	95.27±0.02 (0.44)	0.28
IU	95.11±0.02 (4.89)	99.89±0.04 (0.11)	100.00±0.00 (0.00)	95.70±0.09 (0.01)	3.19
\( \ell_1 \)-sparse MU	100.00±0.00 (0.00)	100.00±0.00 (0.00)	99.99±0.01 (0.01)	95.88±0.14 (0.17)	2.88
Class-wise forgetting, ImageNet
Retrain	100.00±0.00	100.00±0.00	71.75±0.45	69.49±0.27	26.18*
FT	63.60±7.11 (36.40)	68.61±9.04 (31.39)	72.45±0.16 (0.70)	69.80±0.23 (0.31)	2.87*
GA	85.10±5.92 (14.90)	87.46±7.20 (12.54)	65.93±0.49 (5.82)	64.62±0.82 (4.87)	0.01*
IU	43.35±5.26 (56.65)	60.83±6.17 (39.17)	66.28±0.77 (5.47)	66.25±0.53 (3.24)	3.14*
\( \ell_1 \)-sparse MU	99.85±0.07 (0.15)	100.00±0.00 (0.00)	68.07±0.13 (3.68)	68.01±0.21 (1.48)	2.87*
Random data forgetting, CIFAR-10
Retrain	5.41±0.11	13.12±0.14	100.00±0.00	94.42±0.09	42.15
FT	6.83±0.51 (1.42)	14.97±0.62 (1.85)	96.61±0.25 (3.39)	90.13±0.26 (4.29)	2.33
GA	7.54±0.29 (2.13)	10.04±0.31 (3.08)	93.31±0.04 (6.69)	89.28±0.07 (5.14)	0.31
IU	2.03±0.43 (3.38)	5.07±0.74 (8.05)	98.26±0.29 (1.74)	91.33±0.22 (3.09)	3.22
\( \ell_1 \)-sparse MU	5.35±0.22 (0.06)	12.71±0.31 (0.41)	97.39±0.19 (2.61)	91.26±0.20 (3.16)	2.34
Random data forgetting, CIFAR-100
Retrain	24.76±0.12	49.80±0.26	99.98±0.02	74.46±0.08	48.70
FT	0.78±0.34 (23.98)	1.13±0.40 (48.67)	99.93±0.02 (0.05)	75.14±0.09 (0.68)	3.74
GA	0.04±0.02 (24.72)	3.80±0.87 (46.00)	99.97±0.01 (0.01)	74.07±0.11 (0.39)	0.24
IU	1.53±0.36 (23.23)	6.58±0.42 (43.22)	99.01±0.28 (0.97)	71.76±0.31 (2.70)	3.80
\( \ell_1 \)-sparse MU	20.77±0.27 (3.99)	36.80±0.44 (13.00)	98.26±0.15 (1.72)	71.52±0.21 (2.94)	3.76
Random data forgetting, SVHN
Retrain	4.89±0.11	15.38±0.14	100.00±0.00	95.54±0.09	42.71
FT	3.56±0.27 (1.33)	10.05±0.24 (5.33)	99.89±0.04 (0.11)	93.55±0.12 (1.99)	2.73
GA	0.99±0.42 (3.90)	3.07±0.53 (12.31)	99.43±0.22 (0.57)	94.03±0.21 (1.51)	0.26
IU	3.48±0.13 (1.41)	9.44±0.27 (5.94)	96.30±0.08 (3.70)	91.59±0.11 (3.95)	3.21
\( \ell_1 \)-sparse MU	4.06±0.14 (0.83)	11.80±0.22 (3.58)	99.96±0.01 (0.04)	94.98±0.03 (0.56)	2.73
Table A7: Performance of  $\ell_1$ -sparse MU vs. Retrain and FT on (Swin Transformer, CIFAR-10).
MU	UA	MIA-Efficacy	RA	TA	RTE (min)
Class-wise forgetting
Retrain	100.00	100.00	100.00	80.14	153.60
FT	8.56 (91.44)	22.46 (77.54)	99.92 (0.08)	79.72 (0.42)	3.87
\( \ell_1 \)-sparse MU	98.80 (1.20)	100.00 (0.00)	98.25 (1.75)	80.20 (0.06)	3.89
Random data forgetting
Retrain	21.48	28.44	100.00	78.59	155.06
FT	0.16 (21.32)	1.26 (27.18)	99.80 (0.20)	79.54 (0.95)	7.77
\( \ell_1 \)-sparse MU	9.22 (12.26)	18.33 (10.11)	97.92 (2.08)	79.09 (0.50)	7.84
Performance of  $\ell_{1}$  sparsity-aware MU on additional architectures. Tab. A7 presents an additional application to Swin Transformer on CIFAR-10. To facilitate a comparison between approximate unlearning methods (including the FT baseline and the proposed  $\ell_{1}$ -sparse MU) and Retrain, we train the transformer from scratch on CIFAR-10. This could potentially decrease testing accuracy compared with fine-tuning on a pre-trained model over a larger, pre-trained dataset. As we can see, our proposed  $\ell_{1}$ -sparse MU leads to a much smaller performance gap with Retrain compared to FT. In particular, class-wise forgetting exhibited a remarkable  $90.24\%$  increase in UA, accompanied by a slight reduction in RA.
Performance of 'prune first, then unlearn' and  $\ell_1$  sparsity-aware MU on different model sizes. Further, Tab. A8 and Tab. A9 present the unlearning performance versus different model sizes in the ResNet family, involving both ResNet-20s and ResNet-50 on CIFAR-10, in addition to ResNet-18 in Tab. 3. As we can see, sparsity consistently diminishes the unlearning gap with Retrain (indicated by highlighted numbers, with smaller values being preferable). It's worth noting that while both ResNet-20s and ResNet-50 benefit from sparsity, the suggested sparsity ratio is  $90\%$  for ResNet-20s and slightly lower than  $95\%$  for ResNet-50 when striking the balance between MU and generalization.
Table A8: MU performance on (ResNet-20s, CIFAR-10) using 'prune first, then unlearn' (applying to the OMP-resulted  $90\%$  sparse model) and 'sparse-aware unlearning' (applying to the original dense model). The performance is reported in the form  $a_{\pm b}$ , with mean  $a$  and standard deviation  $b$  computed over 10 independent trials. A performance gap against Retrain is provided in  $(\bullet)$ .Smaller performance gap from Retrain is better in the context of machine unlearning.
MU	UA	MIA-Efficacy	RA	TA	RTE
DENSE	90% Sparsity	DENSE	90% Sparsity	DENSE	90% Sparsity	DENSE	90% Sparsity	(min)
Class-wise forgetting
Retrain	100.00±0.00	100.00±0.00	100.00±0.00	100.00±0.00	99.76±0.03	92.95±0.20	92.22±0.20	88.58±0.29	25.27
FT	83.10±4.83 (16.90)	91.67±3.81 (8.33)	97.17±0.75 (2.83)	99.37±0.29 (0.63)	98.14±0.28 (1.62)	93.33±0.80 (0.38)	90.99±0.40 (1.23)	88.90±0.63 (0.32)	1.57
GA	88.48±3.47 (11.52)	90.57±2.14 (9.43)	92.55±4.27 (7.45)	97.37±2.18 (2.63)	91.42±0.53 (8.34)	86.75±0.88 (6.20)	85.46±1.33 (6.76)	83.33±0.78 (5.25)	0.10
ε1-sparse MU	98.57±0.86 (1.43)	n/a	100.00±0.00 (0.00)	n/a	96.18±0.91 (3.58)	n/a	90.18±0.14 (2.04)	n/a	1.60
Random data forgetting
Retrain	8.02±0.36	12.33±0.38	14.94±0.46	16.46±0.83	100.00±0.00	92.33±0.18	91.10±0.27	86.46±0.02	25.29
FT	3.46±0.32 (4.56)	8.93±0.52 (3.40)	9.33±0.45 (5.61)	12.62±0.51 (3.84)	98.57±0.20 (1.43)	93.59±0.33 (1.26)	90.71±0.14 (0.39)	88.15±0.12 (1.69)	1.58
GA	1.84±0.53 (6.18)	6.88±0.41 (5.45)	6.53±0.42 (8.41)	9.57±0.56 (6.89)	97.41±0.21 (2.59)	94.78±0.11 (2.45)	91.03±0.74 (0.07)	89.15±0.31 (2.69)	0.10
ε1-sparse MU	6.44±0.23 (1.58)	n/a	13.15±0.31 (1.79)	n/a	96.31±0.14 (3.69)	n/a	89.14±0.26 (1.96)	n/a	1.58
Table A9: MU performance on (ResNet-50, CIFAR-10) following the format of Table A8.)
MU	UA	MIA-Efficacy	RA	TA	RTE(min)
DENSE	95% Sparsity	DENSE	95% Sparsity	DENSE	95% Sparsity	DENSE	95% Sparsity
Class-wise forgetting
RetrainFTGA\( \ell_1 \)-sparse MU	\( 100.00_{\pm 0.00} \)49.76±2.04(50.24)93.41±0.24(6.59)96.46±0.51(3.54)	\( 100.00_{\pm 0.00} \)57.84±3.10(42.16)93.90±0.21(6.10)n/a	\( 100.00_{\pm 0.00} \)84.67±6.90(15.33)95.90±0.18(4.10)100.00±0.00(0.00)	\( 100.00_{\pm 0.00} \)88.20±3.70(11.80)96.22±2.24(3.78)n/a	\( 100.00_{\pm 0.03} \)99.62±0.12(0.38)93.44±0.53(6.56)99.11±0.42(0.89)	\( 100.00_{\pm 0.00} \)99.65±0.06(0.35)93.05±0.26(6.95)n/a	\( 94.18_{\pm 0.38} \)94.11±0.30(0.07)87.37±0.15(6.81)92.83±0.10(1.35)	\( 94.12_{\pm 0.07} \)93.54±0.15(0.58)87.22±0.08(6.90)n/a	96.296.020.306.05
Random data forgetting
RetrainFTGA\( \ell_1 \)-sparse MU	\( 5.81_{\pm 0.29} \)5.17±0.75(0.64)3.42±0.25(2.39)6.13±0.17(0.32)	\( 6.09_{\pm 0.45} \)5.84±0.45(0.25)5.77±0.37(0.32)n/a	\( 11.99_{\pm 0.94} \)10.93±0.94(1.06)5.20±0.42(6.79)12.29±0.20(0.30)	\( 12.76_{\pm 0.86} \)12.17±0.82(0.59)8.73±0.56(4.03)n/a	\( 100.00_{\pm 0.00} \)97.64±0.22(2.36)96.20±0.19(3.80)97.12±0.16(2.88)	\( 99.00_{\pm 0.00} \)97.24±0.35(1.76)95.41±0.14(3.59)n/a	\( 93.62_{\pm 0.21} \)91.13±0.14(2.49)90.12±0.21(3.50)91.12±0.15(2.50)	\( 93.76_{\pm 0.03} \)90.81±0.12(2.95)89.47±0.26(4.29)n/a	96.306.020.326.10
Broader impacts. Our study on model sparsity-inspired MU provides a versatile solution to forget arbitrary data points and could give a general solution for dealing with different concerns, such as the model's privacy, efficiency, and robustness. Moreover, the applicability of our method extends beyond these aspects, with potential impacts in the following areas. ① Regulatory compliance: Our method enables industries, such as healthcare and finance, to adhere to regulations that require the forgetting of data after a specified period. This capability ensures compliance while preserving the utility and performance of machine learning models. ② Fairness: Our method could also play a crucial role in addressing fairness concerns by facilitating the unlearning of biased datasets or subsets. By removing biased information from the training data, our method contributes to mitigating bias in machine
learning models, ultimately fostering the development of fairer models.  $③$  ML with adaptation and sustainability: Our method could promote the dynamic adaptation of machine learning models by enabling the unlearning of outdated information, and thus, could enhance the accuracy and relevance of the models to the evolving trends and dynamics of the target domain. This capability fosters sustainability by ensuring that ML models remain up-to-date and adaptable, thus enabling their continued usefulness and effectiveness over time.
Limitations. One potential limitation of our study is the absence of provable guarantees for  $\ell_1$ -sparse MU. Since model sparsification is integrated with model training as a soft regularization, the lack of formal proof may raise concerns about the reliability and robustness of the approach. Furthermore, while our proposed unlearning framework is generic, its applications have mainly focused on solving computer vision tasks. As a result, its effectiveness in the domain of natural language processing (NLP) remains unverified. This consideration becomes particularly relevant when considering large language models. Therefore, further investigation is necessary for future studies to explore the applicability and performance of the framework in NLP tasks.