SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model	CIFAR-10	Lacuna-10
Test error (↓) mean	std	Retain error (↓) mean	std	Forget error (↑) mean	std	Test error (↓) mean	std	Retain error (↓) mean	std	Forget error (↑) mean	std
Retrain	16.47	0.21	0.0	0.0	25.67	2.31	1.6	0.44	0.0	0.0	0.67	0.58
Original	16.43	0.08	0.0	0.0	0.0	0.0	1.53	0.31	0.0	0.0	0.0	0.0
Finetune	16.5	0.18	0.0	0.0	0.0	0.0	1.43	0.21	0.0	0.0	0.0	0.0
Fisher	21.39	1.22	4.0	1.44	13.0	11.27	1.87	0.21	0.01	0.02	0.0	0.0
NegGrad+	21.36	0.34	3.23	0.37	45.33	2.89	2.77	0.25	0.40	0.07	8.67	0.58
CF-k	16.29	0.07	0.00	0.00	0.00	0.00	1.53	0.31	0.00	0.00	0.00	0.00
EU-k	17.62	0.61	0.11	0.11	0.33	0.58	1.83	0.47	0.00	0.00	0.00	0.00
Bad-T	22.43	0.37	10.13	0.15	1.67	1.25	4.90	2.10	1.34	1.20	0.67	0.94
SCRUB	16.55	0.11	0.0	0.0	29.33	3.21	2.07	0.31	0.0	0.0	1.67	0.58
Table 12: Large-scale, selective unlearning results with All-CNN for the Removing Biases (RB) application. SCRUB and NegGrad+ are the top-performers in terms of forgetting, though NegGrad+ has worse test performance than SCRUB in both cases. Note also that NegGrad+ isn't as consistent at forgetting across settings as SCRUB, as can be seen in Figure 2
model	Test error (↓)	Retain error (↓)	Forget error (↑)	IC test error (↓)	IC retain error (↓)	Fgt test error (↓)	Fgt retain error (↓)
mean	std	mean	std	mean	std	mean	std	mean	std	mean	std	mean	std
Retrain	26.67	2.87	0.0	0.0	90.33	1.53	24.0	1.8	0.0	0.0	18.33	4.16	0.0	0.0
Original	41.0	2.09	0.0	0.0	0.0	0.0	56.0	3.04	0.0	0.0	92.0	7.94	0.0	0.0
Finetune	38.13	1.42	0.0	0.0	0.0	0.0	52.0	3.12	0.0	0.0	79.33	10.07	0.0	0.0
NegGrad+	36.27	0.42	0.0	0.0	12.67	21.94	47.5	5.27	0.0	0.0	69.0	13.53	0.0	0.0
CF-k	39.6	1.64	0.0	0.0	0.0	0.0	54.83	2.02	0.0	0.0	85.33	7.02	0.0	0.0
EU-k	37.47	1.62	7.33	1.26	43.67	2.08	47.0	4.77	8.33	4.73	63.33	9.71	3.67	2.52
Fisher	44.8	2.36	21.33	3.45	32.0	11.53	51.5	7.47	26.33	9.5	79.0	3.61	20.0	7.94
NTK	32.6	2.51	0.0	0.0	60.33	0.58	37.5	4.0	0.0	0.0	52.0	10.58	0.0	0.0
SCRUB	25.93	3.13	1.08	0.52	96.0	1.73	19.0	3.91	0.0	0.0	19.67	7.51	0.0	0.0
15 Additional Results for Resolving Confusion (RC)
We show the full results are in Tables 13, 14, 15, 16, 17, 18, 19, 20 for all settings. We observe that across the board, SCRUB is a top-performer on this metric too (see the captions of the individual tables for more details about performance profile).
16 Additional results for User Privacy (UP)
We present Basic MIA results for all settings in Tables 21, 22, 23, 24, 25, 26, 27, 28. We find that SCRUB, especially equipped with its rewinding procedure, is able to consistently have a strong defense against MIAs.
Table 13: Results on CIFAR-5 with ResNet for the Resolving Confusion (RC) application. (Confused class 0,1; 50-50 samples). SCRUB is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first 3 groups of columns).
model	Test error (↓)	Retain error (↓)	Forget error (↑)	IC test error (↓)	IC retain error (↓)	Fgt test error (↓)	Fgt retain error (↓)
mean	std	mean	std	mean	std	mean	std	mean	std	mean	std	mean	std
Retrain	24.4	2.75	0.0	0.0	90.67	4.04	19.0	1.32	0.0	0.0	11.33	4.62	0.0	0.0
Original	37.07	4.67	1.5	2.6	5.67	9.81	49.0	4.77	6.0	10.39	80.67	12.58	6.0	10.39
Finetune	34.33	3.35	0.0	0.0	3.0	5.2	43.67	7.29	0.0	0.0	67.33	16.04	0.0	0.0
NegGrad+	33.53	4.47	0.0	0.0	13.33	21.36	42.33	11.34	0.0	0.0	62.0	22.65	0.0	0.0
CF-k	36.13	4.21	0.0	0.0	0.33	0.58	47.83	5.8	0.0	0.0	76.33	14.43	0.0	0.0
EU-k	51.6	1.0	27.67	3.5	52.67	6.03	59.5	5.22	38.33	6.66	68.67	15.57	19.67	10.41
Fisher	51.93	2.95	35.17	3.92	31.0	11.53	56.83	8.69	31.67	14.01	78.33	15.53	17.67	11.5
NTK	32.2	2.84	0.75	1.3	43.33	14.15	36.67	4.07	3.0	5.2	54.33	9.02	3.0	5.2
SCRUB	25.0	3.14	0.0	0.0	93.33	2.52	26.0	4.44	0.0	0.0	18.0	11.14	0.0	0.0
Table 14: Results on CIFAR-5 with All-CNN for the Resolving Confusion (RC) application. (Confused class 0,1; 50-50 samples). SCRUB is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first 3 groups of columns).
model	Test error (↓)	Retain error (↓)	Forget error (↑)	IC test error (↓)	IC retain error (↓)	Fgt test error (↓)	Fgt retain error (↓)
mean	std	mean	std	mean	std	mean	std	mean	std	mean	std	mean	std
Retrain	6.0	0.2	0.0	0.0	99.67	0.58	7.17	2.57	0.0	0.0	0.0	0.0	0.0	0.0
Original	27.07	3.33	1.67	0.88	4.33	1.53	57.5	6.26	6.67	3.51	108.0	14.18	6.67	3.51
Finetune	18.8	4.26	0.0	0.0	14.67	6.03	37.67	11.15	0.0	0.0	63.67	22.01	0.0	0.0
NegGrad+	17.8	2.95	1.67	0.72	55.33	2.08	33.17	5.25	5.33	1.53	56.67	12.9	4.33	1.53
CF-k	22.27	4.31	0.08	0.14	10.67	5.03	46.33	10.97	0.33	0.58	81.67	23.01	0.33	0.58
EU-k	15.27	3.19	0.83	0.38	62.0	12.49	29.33	9.0	2.33	1.53	43.67	16.29	0.33	0.58
Fisher	35.87	3.33	17.75	3.78	27.33	3.79	60.0	5.27	31.0	7.94	109.0	14.53	30.0	7.0
NTK	14.53	5.22	0.0	0.0	51.67	23.18	27.17	11.3	0.0	0.0	43.33	25.32	0.0	0.0
SCRUB	8.47	1.17	0.33	0.14	96.0	1.0	11.33	3.82	1.33	0.58	9.33	1.53	1.33	0.58
Table 15: Results on Lacuna-5 with ResNet for the Resolving Confusion (RC) application. (Confused class 0,1; 50-50 samples). SCRUB is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first 3 groups of columns). NTK in some cases is able to resolve confusion, but not consistently, and it also suffers from higher Test Error.
model	Test error (↓)	Retain error (↓)	Forget error (↑)	IC test error (↓)	IC retain error (↓)	Fgt test error (↓)	Fgt retain error (↓)
mean	std	mean	std	mean	std	mean	std	mean	std	mean	std	mean	std
Retrain	4.2	0.87	0.0	0.0	100.0	0.0	5.33	2.25	0.0	0.0	0.0	0.0	0.0	0.0
Original	25.47	2.32	5.75	5.63	20.33	25.74	56.17	4.93	23.0	22.54	105.67	8.08	23.0	22.54
Finetune	12.8	2.8	0.0	0.0	23.0	7.94	25.83	7.75	0.0	0.0	39.67	12.74	0.0	0.0
NegGrad+	12.8	9.06	2.5	3.12	90.0	6.56	20.33	17.04	5.0	6.24	12.67	11.68	2.67	3.79
CF-k	21.27	1.63	0.58	0.8	9.33	0.58	47.0	4.58	2.33	3.21	82.67	10.12	2.33	3.21
EU-k	17.0	8.91	3.92	3.99	92.33	4.93	35.0	18.26	13.0	11.36	3.67	4.73	0.0	0.0
Fisher	49.6	4.73	39.25	7.45	40.0	9.54	57.67	10.79	42.33	11.59	88.67	11.68	29.67	16.86
NTK	12.87	6.63	2.83	4.91	72.33	12.06	25.5	17.88	11.33	19.63	35.67	24.03	10.0	17.32
SCRUB	3.87	0.7	0.0	0.0	100.0	0.0	4.33	1.26	0.0	0.0	0.0	0.0	0.0	0.0
Table 16: Results on Lacuna-5 with All-CNN for the Resolving Confusion (RC) application. (Confused class 0,1; 50-50 samples). SCRUB is the best-performer by far in terms of eliminating the confusion via unlearning (see the IC error and Fgt error columns), while not hurting performance for other classes (see e.g. the usual Error metrics in the first 3 groups of columns).
model	Test error (↓)	Retain error (↓)	Forget error (↑)	IC test error (↓)	IC retain error (↓)	Fgt test error (↓)	Fgt retain error (↓)
mean	std	mean	std	mean	std	mean	std	mean	std	mean	std	mean	std
retrain	18.7	0.07	0.0	0.0	98.57	0.28	14.78	0.18	0.0	0.0	31.33	2.08	0.0	0.0
original	21.86	0.37	0.0	0.0	0.0	0.0	31.23	0.45	0.0	0.0	356.0	11.53	0.0	0.0
finetune	20.85	0.37	0.0	0.0	0.0	0.0	26.75	0.48	0.0	0.0	255.0	10.58	0.0	0.0
NegGrad+	23.41	0.32	3.87	0.31	80.07	6.77	41.08	0.6	20.29	1.52	46.0	8.72	0.67	1.15
CF-k	20.93	0.38	0.0	0.0	0.0	0.0	27.27	0.76	0.0	0.0	267.33	16.17	0.0	0.0
EU-k	20.03	0.19	0.25	0.08	95.55	0.54	17.85	0.67	0.18	0.03	53.0	7.94	3.33	2.31
SCRUB	18.01	0.18	0.02	0.01	95.45	0.26	15.07	0.99	0.04	0.03	30.33	3.79	0.33	0.58
Table 17: Results on CIFAR-10 with ResNet for the Resolving Confusion (RC) application. (Confused class 0,1; 2000-2000 samples).
model	Test error (↓)	Retain error (↓)	Forget error (↑)	IC test error (↓)	IC retain error (↓)	Fgt test error (↓)	Fgt retain error (↓)
mean	std	mean	std	mean	std	mean	std	mean	std	mean	std	mean	std
retrain	16.43	0.03	0.0	0.0	98.42	0.15	14.37	0.24	0.0	0.0	23.67	2.31	0.0	0.0
original	19.95	0.23	0.0	0.0	0.0	0.0	30.18	0.66	0.0	0.0	348.67	13.58	0.0	0.0
finetune	18.72	0.11	0.0	0.0	1.05	0.61	24.33	0.2	0.0	0.0	223.67	6.66	0.0	0.0
NegGrad+	21.74	0.44	4.48	0.34	87.65	2.98	40.05	0.44	21.8	0.66	44.0	5.2	2.33	3.21
CF-k	19.31	0.23	0.0	0.0	0.0	0.0	27.45	0.61	0.0	0.0	294.0	4.36	0.0	0.0
EU-k	17.66	0.23	1.36	0.19	87.9	1.28	16.82	0.79	2.89	0.46	63.67	8.62	91.67	12.58
SCRUB	15.92	0.17	0.2	0.06	87.47	1.46	14.98	0.13	0.39	0.15	54.0	3.61	9.67	2.52
Table 18: Results on CIFAR-10 with All-CNN for the Resolving Confusion (RC) application. (Confused class 0,1; 2000-2000 samples).
model	Test error (↓)	Retain error (↓)	Forget error (↑)	IC test error (↓)	IC retain error (↓)	Fgt test error (↓)	Fgt retain error (↓)
mean	std	mean	std	mean	std	mean	std	mean	std	mean	std	mean	std
retrain	2.43	0.32	0.0	0.0	99.83	0.29	3.33	0.58	0.0	0.0	0.0	0.0	0.0	0.0
original	7.37	0.31	1.21	0.12	15.83	4.19	27.67	1.26	8.26	0.8	46.0	4.36	36.33	3.51
finetune	4.17	0.5	0.0	0.0	56.83	9.44	11.17	1.76	0.0	0.0	15.67	3.51	0.0	0.0
NegGrad+	5.63	0.38	0.31	0.22	71.33	9.88	19.33	1.04	2.12	1.51	8.33	2.31	0.0	0.0
CF-k	5.4	0.4	0.07	0.06	33.83	3.33	17.33	1.76	0.45	0.39	27.67	3.51	2.0	1.73
EU-k	3.0	0.26	0.0	0.0	90.17	4.65	6.0	1.8	0.0	0.0	2.0	2.65	0.0	0.0
SCRUB	3.07	0.59	0.0	0.0	98.5	0.5	6.83	1.26	0.0	0.0	0.67	0.58	0.0	0.0
Table 19: Results on Lacuna-10 with ResNet for the Resolving Confusion (RC) application. (Confused class 0,1; 200-200 samples).
model	Test error (↓) mean	std	Retain error (↓) mean	std	Forget error (↑) mean	std	IC test error (↓) mean	std	IC retain error (↓) mean	std	Fgt test error (↓) mean	std	Fgt retain error (↓) mean	std
retrain	2.13	0.25	0.0	0.0	99.83	0.29	2.5	1.32	0.0	0.0	0.0	0.0	0.0	0.0
original	7.83	0.55	1.21	0.52	16.0	4.92	31.33	1.76	8.26	3.53	56.33	3.79	36.33	15.53
finetune	3.0	0.7	0.0	0.0	74.5	6.08	6.5	2.0	0.0	0.0	9.0	3.0	0.0	0.0
NegGrad+	4.3	0.52	0.4	0.06	89.67	4.25	15.33	2.47	2.73	0.39	4.67	3.21	0.0	0.0
CF-k	5.27	0.47	0.11	0.07	33.33	1.61	18.5	2.29	0.76	0.47	31.33	3.51	3.33	2.08
EU-k	2.53	0.67	0.09	0.02	97.83	2.08	5.17	1.04	0.38	0.35	0.33	0.58	0.67	0.58
SCRUB	2.1	0.4	0.0	0.0	97.5	1.73	4.17	0.58	0.0	0.0	0.33	0.58	0.0	0.0
Table 20: Results on Lacuna-10 with All-CNN for the Resolving Confusion (RC) application. (Confused class 0,1; 200-200 samples).
method	Test error	Forget error	Retain error	MIA
mean	std	mean	std	mean	std	mean	std
Retrain	16.71	0.05	26.67	3.09	0.00	0.00	51.33	6.13
Original	16.71	0.07	0.00	0.00	0.00	0.00	68.67	3.09
Finetune	16.86	0.13	0.00	0.00	0.00	0.00	69.33	2.05
NegGrad+	21.65	0.40	47.00	3.74	4.54	0.70	73.00	1.41
CF-k	16.82	0.03	0.00	0.00	0.00	0.00	69.67	1.89
EU-k	18.44	0.21	0.33	0.47	0.32	0.02	66.00	2.94
Bad-T	22.43	0.37	1.67	1.25	10.13	0.15	77.67	4.11
SCRUB	17.01	0.20	33.00	5.89	0.00	0.00	51.00	1.41
SCRUB+R	16.88	0.19	26.33	4.50	0.00	0.00	49.33	2.49
Table 21: Basic MIA for All-CNN architecture on CIFAR-10 for selective unlearning, for the User Privacy (UP) application.
method	Test error	Forget error	Retain error	MIA
mean	std	mean	std	mean	std	mean	std
Retrain	13.98	0.07	100.00	0.00	0.00	0.00	48.73	0.24
Original	15.70	0.09	0.00	0.00	0.00	0.00	71.40	0.70
Finetune	14.53	0.13	1.31	0.54	0.00	0.00	74.97	1.27
NegGrad+	17.04	0.11	59.91	1.53	0.43	0.09	70.03	1.92
CF-k	15.72	0.06	0.00	0.00	0.00	0.00	72.93	1.06
EU-k	15.76	0.28	100.00	0.00	0.24	0.02	51.60	1.22
Bad-T	16.98	0.40	81.93	3.50	5.84	0.43	58.07	1.76
SCRUB	14.93	0.17	100.00	0.00	0.09	0.02	54.30	2.24
SCRUB+R	14.93	0.17	100.00	0.00	0.09	0.02	54.30	2.24
Table 22: Basic MIA for All-CNN architecture on CIFAR-10 for class unlearning, for the User Privacy (UP) application.
method	Test error	Forget error	Retain error	MIA
mean	std	mean	std	mean	std	mean	std
Retrain	17.38	0.15	29.33	2.49	0.00	0.00	54.00	1.63
Original	17.41	0.15	0.00	0.00	0.00	0.00	65.33	0.47
Finetune	17.48	0.16	0.00	0.00	0.00	0.00	64.00	0.82
NegGrad+	21.69	0.07	45.33	2.62	3.94	0.43	66.67	1.70
CF-k	17.53	0.19	0.00	0.00	0.00	0.00	65.00	0.00
EU-k	19.77	0.04	13.67	0.47	0.06	0.01	53.00	3.27
Bad-T	23.47	1.57	34.67	1.70	14.53	1.65	59.67	4.19
SCRUB	17.01	0.03	71.67	0.94	0.01	0.01	78.00	2.45
SCRUB+R	17.54	0.28	19.33	14.64	0.01	0.01	58.67	1.89
Table 23: Basic MIA for ResNet architecture on CIFAR-10 for selective unlearning, for the User Privacy (UP) application.
method	Test error	Forget error	Retain error	MIA
mean	std	mean	std	mean	std	mean	std
Retrain	14.69	0.10	100.00	0.00	0.00	0.00	49.33	1.67
Original	16.33	0.14	0.00	0.00	0.00	0.00	71.10	0.67
Finetune	15.10	0.16	0.33	0.17	0.00	0.00	75.57	0.69
NegGrad+	17.41	0.09	61.00	1.14	0.44	0.05	69.57	1.19
CF-k	15.29	0.02	0.04	0.04	0.00	0.00	75.73	0.34
EU-k	17.05	0.07	97.48	0.28	0.05	0.01	54.20	2.27
Bad-T	19.56	1.44	11.34	1.82	94.67	6.12	54.33	0.31
SCRUB	15.33	0.06	100.00	0.00	0.08	0.01	52.20	1.71
SCRUB+R	15.33	0.06	100.00	0.00	0.08	0.01	52.20	1.71
Table 24: Basic MIA for ResNet architecture on CIFAR-10 for class unlearning, for the User Privacy (UP) application.
method	Test error	Forget error	Retain error	MIA
mean	std	mean	std	mean	std	mean	std
Retrain	1.50	0.08	0.33	0.47	0.00	0.00	52.00	2.16
Original	1.57	0.24	0.00	0.00	0.00	0.00	59.00	2.16
Finetune	1.40	0.16	0.00	0.00	0.00	0.00	57.33	3.30
NegGrad+	3.60	0.14	14.33	1.25	0.87	0.07	51.00	1.63
CF-k	1.57	0.12	0.00	0.00	0.00	0.00	58.33	2.49
EU-k	3.90	1.47	0.00	0.00	0.76	0.63	52.00	3.56
Bad-T	4.90	2.10	1.34	1.20	0.67	0.94	67.67	6.94
SCRUB	1.67	0.19	0.00	0.00	0.00	0.00	57.67	0.94
SCRUB+R	1.67	0.19	0.00	0.00	0.00	0.00	57.67	0.94
Table 25: Basic MIA for All-CNN architecture on Lacuna-10 for selective unlearning, for the User Privacy (UP) application.
method	Test error	Forget error	Retain error	MIA
mean	std	mean	std	mean	std	mean	std
Retrain	1.67	0.09	100.00	0.00	0.00	0.00	55.67	2.62
Original	1.70	0.21	0.00	0.00	0.00	0.00	58.00	1.63
Finetune	1.67	0.27	0.00	0.00	0.00	0.00	56.33	1.25
NegGrad+	2.00	0.00	14.27	0.74	0.00	0.00	54.33	2.05
CF-k	2.07	0.14	0.00	0.00	0.00	0.00	52.33	2.05
EU-k	4.15	1.22	62.08	44.26	0.81	0.53	52.67	3.68
Bad-T	2.56	0.09	38.65	36.80	0.37	0.18	63.33	2.49
SCRUB	1.96	0.34	100.00	0.00	0.00	0.00	50.33	2.62
SCRUB+R	1.96	0.34	100.00	0.00	0.00	0.00	50.33	2.62
Table 26: Basic MIA for All-CNN architecture on Lacuna-10 for class unlearning, for the User Privacy (UP) application.
method	Test error	Forget error	Retain error	MIA
mean	std	mean	std	mean	std	mean	std
Retrain	2.50	0.24	1.67	0.94	0.00	0.00	49.67	3.09
Original	2.53	0.25	0.00	0.00	0.00	0.00	56.67	1.70
Finetune	2.67	0.05	0.00	0.00	0.00	0.00	53.67	0.94
NegGrad+	4.30	0.43	12.67	3.30	0.95	0.08	54.00	2.16
CF-k	2.47	0.25	0.00	0.00	0.00	0.00	56.00	0.82
EU-k	2.60	0.00	0.00	0.00	0.03	0.00	56.00	2.83
Bad-T	7.30	2.20	3.26	1.83	0.33	0.47	67.33	3.40
SCRUB	2.97	0.25	6.00	3.27	0.00	0.00	50.67	4.03
SCRUB+R	2.97	0.25	6.00	3.27	0.00	0.00	50.67	4.03
Table 27: Basic MIA for ResNet architecture on Lacuna-10 for selective unlearning, for the User Privacy (UP) application.
method	Test error	Forget error	Retain error	MIA
mean	std	mean	std	mean	std	mean	std
Retrain	2.52	0.19	100.00	0.00	0.00	0.00	55.00	2.94
Original	2.81	0.28	0.00	0.00	0.00	0.00	56.00	2.45
Finetune	3.04	0.19	0.00	0.00	0.00	0.00	54.67	1.25
NegGrad+	2.74	0.26	9.48	0.64	0.00	0.00	53.67	4.03
CF-k	2.81	0.28	0.00	0.00	0.00	0.00	56.00	2.45
EU-k	2.48	0.14	7.71	2.52	0.00	0.00	54.33	3.09
Bad-T	3.37	0.50	67.60	24.26	1.06	0.47	58.00	2.94
SCRUB	3.26	0.38	99.90	0.15	0.07	0.05	54.33	2.49
SCRUB+R	3.26	0.38	99.90	0.15	0.07	0.05	54.33	2.49
Table 28: Basic MIA for ResNet architecture on Lacuna-10 for class unlearning, for the User Privacy (UP) application.