SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MovieLens 1M (ML-1M): This is a stable version of the MovieLens dataset, containing 1 million ratings; and iii) Amazon Digital Music  $(\mathbf{ADM})^3$ : The Amazon dataset contains several sub-datasets according to the categories of Amazon products. ADM is the sub-dataset containing digital music reviews. To ensure reliable evaluations, we filter out the users and items that have less than 5 interactions. Specifically, we use  $80\%$  of ratings for training,  $10\%$  as a validation set for tuning hyper-parameters, and the remainder for testing. Table 1 summarizes the statistics of three datasets.
Table 1: Summary of datasets.
Dataset	User #	Item #	Rating #	Sparsity
ML-100K	943	1,682	100,000	93.695%
ML-1M	6040	3,950	1,000,209	95.814%
ADM	478,235	266,414	836,006	99.999%
5.1.2 Compared Models and Methods
Recommendation Models. Our proposed U1traRE is model-agnostic, enabling its application to any recommendation model. In this paper, we select two representative recommendation models: i) a classic model, i.e., Deep Matrix Factorization (DMF) [47], and ii) the SOTA model, i.e. LightGCN [18], for testing. Following the original papers, we adopt normalized binary cross entropy loss and Bayesian personalized ranking loss for DMF and LightGCN respectively, and employ Adam optimizer to train the above models. We run all experiments for 10 trials and report the average results. Following [7], we use WMF [19] as a pre-training model to generate user and item embedding for the purpose of clustering.
Unlearning Methods. We compare U1traRE with the benchmark and the SOTA methods, including: i) Retrain: Retraining the model from scratch on the updated dataset; ii) SISA [4]: the SOTA generic exact unlearning method which is based on ensemble retraining framework; and iii) RecEraser [7]: the SOTA recommendation unlearning method which modifies SISA to boost performance in recommendation tasks. Following [4, 7], we set the number of shards to 10 for all unlearning methods that involve division, i.e., SISA, RecEraser, and U1traRE.
$^{2}$ https://grouplens.org/datasets/movielens/
<sup>3</sup>http://jmcauley.ucsd.edu/data/amazon/
Table 2: Running time (s) of the learning process.
ML-100K	DMF	LightGCN
Retrain	SISA	RecEraser	UltraRE	Retrain	SISA	RecEraser	UltraRE
Stage I	0.000	0.013	0.662	0.048	0.000	0.014	0.645	0.050
Stage II	1,442.263	193.317	189.742	190.289	4,422.531	582.659	594.714	591.638
Stage III	0.000	0.001	42.673	31.540	0.000	0.003	383.132	315.822
Total	1,442.263	193.331	233.077	221.877	4,422.531	582.676	978.491	907.510
ML-1M	DMF	LightGCN
Retrain	SISA	RecEraser	UltraRE	Retrain	SISA	RecEraser	UltraRE
Stage I	0.000	0.046	23.904	0.429	0.000	0.058	23.950	0.467
Stage II	3,732.366	376.237	378.475	374.999	11,372.446	1,218.335	1,211.275	1,218.875
Stage III	0.000	0.003	93.519	69.327	0.000	0.004	682.182	624.349
Total	3,732.366	376.286	495.898	444.755	11,372.446	1,218.397	1,917.407	1,843.691
ADM	DMF	LightGCN
Retrain	SISA	RecEraser	UltraRE	Retrain	SISA	RecEraser	UltraRE
Stage I	0.000	0.052	20.864	0.373	0.000	0.054	20.851	0.352
Stage II	2,012.655	207.457	201.769	206.417	6,724.734	698.737	679.051	682.742
Stage III	0.000	0.003	57.352	48.975	0.000	0.004	404.110	374.253
Total	2,012.655	207.512	279.985	255.765	6,724.743	698.795	1,104.012	1,057.347
5.2.1 Unlearning Efficiency (G2)
We use running time to evaluate the efficiency of unlearning. To fully exploit the efficiency of the ensemble retraining framework, we run all shards in parallel. Specifically, we measure the running time of each stage during the learning process and report the results in Table 2. As the shard division (stage I) and combination weights (stage III) are determined during learning and remain unchanged during unlearning, their time cost during unlearning is negligible during unlearning compared to retraining (stage II). Since all shards run in parallel, the running time of stage II remains consistent between learning and unlearning. Therefore, by measuring the running time of stage II during learning, we can evaluate the running time of unlearning. We run all experiments on the same Ubuntu 20.04 LTS System server with 48-core CPU, 256GB RAM and NVIDIA GeForce RTX 3090 GPU. From Table 2, we observe that i) compared with Retrain, the ensemble retraining frameworks, i.e., SISA, RecEraser, and U1traRE, significantly enhance unlearning efficiency. Although spending more time in stages I and III, these frameworks decrease the total running time by an average of  $85.39\%$ ; ii) Using random balanced division and average combiner, SISA enjoys a notably faster speed than other ensemble retraining frameworks. Nevertheless, this simple design also limits SISA's performance regarding model utility (see Section 5.2.2); and iii) Among the comparison of recommendation unlearning frameworks, i.e., RecEraser and U1traRE, our proposed U1traRE decreases the running time of both stages I and III. In stage I, our proposed optimal balanced clustering algorithm can improve the efficiency by  $98.11\%$  on average. This improvement is even greater when dealing with larger datasets, i.e., ML-1M and ADM. In stage III, we simplify the choice of the model combiner, resulting in an average efficiency increase of  $11.95\%$ .
5.2.2 Model Utility (G3)
We use two common metrics, i.e., Normalized Discounted Cumulative Gain (NDCG) and Hit Ratio (HR), to evaluate the performance of recommender models [17, 47]. For both metrics, we truncate the ranked list at 10, and report NDCG@10 and HR@10 during both the learning and unlearning processes. To simulate user-wise unlearning, we randomly select  $q\%$  of users to unlearn, where  $q$  is investigated in \{5, 10\} across all datasets. From Table 3, we observe that i) Retrain achieves the best performance, suggesting using ensemble retraining frameworks (including SISA, RecEraser, and U1traRE) may come at the cost of slightly increased error rates. This trade-off occurs because these frameworks prioritize unlearning efficiency over preserving model utility; ii) The performance of cluster-based division methods, i.e., RecEraser and U1traRE, surpasses that of the random division method, i.e., SISA. This implies that by clustering similar samples together, redundancy loss can be
Table 3: Recommendation performance during learning and unlearning (G3: model utility). The best results except Retrain are highlighted in **bold**. The superscript * indicates  $p < 0.01$  for the t-test of UltraRE against RecEraser.
ML-100K	DMF	LightGCN
Retrain	SISA	RecEraser	UltraRE	Retrain	SISA	RecEraser	UltraRE
Learn	NDCG	0.3956	0.3716	0.3795	0.3847*	0.3997	0.3684	0.3812	0.3859*
HR	0.4374	0.4227	0.4273	0.4326*	0.4395	0.4203	0.4257	0.4312*
Unlearn@5	NDCG	0.3934	0.3709	0.3762	0.3815*	0.3976	0.3677	0.3799	0.3841*
HR	0.4359	0.4222	0.4232	0.4297*	0.4387	0.4201	0.4233	0.4295*
Unlearn@10	NDCG	0.3905	0.3702	0.3734	0.3786*	0.3953	0.3669	0.3793	0.3836*
HR	0.4353	0.4205	0.4183	0.4245*	0.4362	0.4186	0.4221	0.4267*
ML-1M	DMF	LightGCN
Retrain	SISA	RecEraser	UltraRE	Retrain	SISA	RecEraser	UltraRE
Learn	NDCG	0.4382	0.3956	0.3973	0.4042*	0.4437	0.3941	0.4171	0.4241*
HR	0.5402	0.5141	0.5134	0.5183*	0.5493	0.4672	0.5160	0.5217*
Unlearn@5	NDCG	0.4398	0.3937	0.3965	0.4031*	0.4403	0.3908	0.4138	0.4189*
HR	0.5411	0.5125	0.5129	0.5179*	0.5464	0.4543	0.5140	0.5206*
Unlearn@10	NDCG	0.4391	0.3914	0.3963	0.4013*	0.4396	0.3912	0.4129	0.4181*
HR	0.5411	0.5108	0.5127	0.5184*	0.5421	0.4495	0.5090	0.5146*
ADM	DMF	LightGCN
Retrain	SISA	RecEraser	UltraRE	Retrain	SISA	RecEraser	UltraRE
Learn	NDCG	0.4549	0.4063	0.4234	0.4294*	0.4603	0.4025	0.4281	0.4345*
HR	0.7914	0.7012	0.7230	0.7311*	0.7931	0.6912	0.7406	0.7472*
Unlearn@5	NDCG	0.4551	0.4021	0.4214	0.4273*	0.4594	0.3963	0.4251	0.4311*
HR	0.7915	0.6959	0.7202	0.7283*	0.7945	0.6851	0.7311	0.7403*
Unlearn@10	NDCG	0.4554	0.3973	0.4198	0.4262*	0.4595	0.4043	0.4239	0.4306*
HR	0.7916	0.6772	0.7222	0.7298*	0.7936	0.6781	0.7192	0.7255*
significantly reduced. iii) Our proposed U1traRE demonstrates superior performance compared to the SOTA recommendation unlearning framework, i.e., RecEraser. On average, our model achieved an improvement in recommendation metrics of  $1.31\%$ ,  $1.26\%$ , and  $1.26\%$ , on ML-100k, ML-1M and ADM respectively. The improved results can be attributed to U1traRE's ability to generate better clustering outcomes while meeting the balance requirement; and iv) Strong performance during the learning process serves as a reliable indicator of strong performance during unlearning.
5.2.3 Ablation Study
To fully understand the effectiveness of our proposed UltraraRE, we conduct ablation studies regarding the main modifications during stages I (non-overlapping division) and III (model combination), as well as varying numbers of shards.
Effect of Division. In stage I, we propose a novel division algorithm, named Optimal Balanced Clustering (OBC). We conduct an ablation study to compare OBC with Balanced Random Division (BRD, used in SISA [4]), Balanced  $k$ -means (BKM, used in RecEraser [7]) and  $k$ -means (KM). Although KM causes the issue of imbalance in clustering, it can still be a valuable baseline for evaluating division performance. We
use inertia in Eq (1) to evaluate division performance. A lower inertia value indicates better division performance. We present the results obtained with  $k = 10$  in Figure 4. From it, we observe that i) Among balanced grouping algorithms, our proposed OBC significantly outperforms BKM (by  $46.32\%$  on average) and BRD (by  $22.01\%$  on average); ii) OBC also achieves comparable inertia to KM, indicating that OBC achieves balanced grouping without compromising sample similarity, and iii) BRD even outperforms BKM. This indicates that the usage of a priority list in BKM leads to a significant degradation in division performance, making it worse than random division.
Figure 4: Effect of division (Stage I)
Effect of Combination. In stage III, we simplify the choice of the model combiner. To validate our choice, we compare our logistic regression (LR) based combiner with Average combiner (AVG which is used in SISA [4]) and Attention combiner (ATT which is used in RecEraser [7]). As shown in Figure 5, model-based combiners, i.e., LR and ATT, outperform AVG by a significant margin. On average, AVG's performance is decreased by  $3.53\%$  on DMF and
$6.51\%$  on LightGCN. Among model-based combiners, LR can achieve comparable performance with ATT while substantially reducing the model complexity.
(a) DMF
(b) LightGCN
Figure 5: Effect of combination (Stage III).
Effect of Shard Number. To demonstrate the robustness of ULtraRE, we empirically study the effect of shard number and compare it with the SOTA recommendation unlearning framework RecEraser. We investigate the number of shards  $S$  in  $\{10,20,50\}$ . As shown in Figure 6, ULtraRE yields consistent improvements over RecEraser regarding both unlearning efficiency (time) and model utility (NDCG@10). Moreover, as the shard number increases, ULtraRE
exhibits greater efficiency enhancement over RecEraser. This is because a larger shard number reduces the time taken in stage II, thereby highlighting the efficiency of U1traRE in stages I and III.
(a) DMF
Figure 6: Effect of shard number (ML-1M).
(b) LightGCN
In this paper, we refine the SOTA exact unlearning framework, i.e., ensemble retraining framework, in recommendation tasks. Our proposed U1traRE breaks away from the prior work's intuition on preserving collaboration [7, 8, 26]. Instead, it is guided by the theoretical analysis of error decomposition in ensemble systems. Among the three stages in the ensemble retraining framework, our modifications mainly lie in stages I (non-overlapping division) and III (model combination). In stage I, we proposed an optimal balanced clustering algorithm that can achieve an adaptive equilibrium between sample similarity and shard balance. In stage III, we simplify the complexity of the model combiner without increasing the combination loss. Extensive experiments on three real-world recommendation datasets demonstrate that U1traRE can not only greatly enhance unlearning efficiency, but also outperform the SOTA unlearning models in terms of model utility.
7 Broader Impacts and Limitations
Recommendation unlearning can have various implications on society, including addressing issues related to privacy, fairness, bias, and manipulation. UltraRE can also be generalized to other unlearning tasks and is especially adept in association-sensitive tasks, e.g., graph learning. A common limitation of the ensemble retraining framework is the absence of experiments on large-scale datasets with a large shard number [4, 7, 8, 26]. As shown in Section 5.2.1, the major time of unlearning is spent on stage II, which cannot fully demonstrate the efficiency of the ensemble retraining framework. We manage to investigate the effect of large shard number in Section 5.2.3.
This work was supported in part by Leading Expert of Ten Thousands Talent Program of Zhejiang Province (No.2021R52001) and the National Natural Science Foundation of China (No.62172362).
[1] S Basu, P Pope, and S Feizi. Influence functions in deep learning are fragile. In In International Conference on Learning Representations (ICLR), 2021.
[42] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.
[43] Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security), pages 4007-4022, 2022.
[44] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10(3152676):10-5555, 2017.
[45] Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. In Network and Distributed System Security (NDSS) Symposium 2023, 2023.
[46] David H Wolpert. Stacked generalization. Neural networks, 5(2):241-259, 1992.
[47] Hong-Jian Xue, Xinyu Dai, Jianbing Zhang, Shujian Huang, and Jiajun Chen. Deep matrix factorization models for recommender systems. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), volume 17, pages 3203-3209. Melbourne, Australia, 2017.
[48] Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. Arcane: An efficient architecture for exact machine unlearning. In Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI), 2022.
A Theoretical Validation of Balanced Division
Supposing there is no prior knowledge regarding the distribution of unlearning requests, it is optimal to presume that users submit requests with equal probability. Under this condition, a balanced division in the ensemble retraining framework can achieve maximum unlearning efficiency.
Proof. Let  $h_i$  denote the training overhead of shard  $S_i$ , which is in proportion to the shard size, i.e., the number of samples in the shard. Since the users submit unlearning requests with equal probability, the probability of a request located in shard  $S_i$  is  $P_i = h_i / Z$  where  $Z = \sum_j h_j$ . Thus, the expectation of retraining overhead when dealing with an unlearning request is
$$
\mathbb {E} (H) = \sum_ {i = 1} ^ {k} P _ {i} \cdot h _ {i} = \frac {1}{Z} \sum_ {i = 1} ^ {k} h _ {i} ^ {2}. \tag {6}
$$
According to Cauchy-Schwarz Inequality [2], for  $n$  random variables  $x_{i}$ , we can get a lower bound of  $\sum_{i} x_{i}^{2}$  as:
$$
\sum_ {i = 1} ^ {n} x _ {i} ^ {2} \geq \frac {\left(\sum_ {i = 1} ^ {n} x _ {i}\right) ^ {2}}{n}. \tag {7}
$$
Taking it into (6), we have
$$
\mathbb {E} (H) = \frac {1}{Z} \sum_ {i = 1} ^ {k} h _ {i} ^ {2} \geq \frac {Z}{k}. \tag {8}
$$
We can easily find that setting  $h_i = Z / k$  achieves this lower bound, which indicates that a balanced division achieves the maximum unlearning efficiency.
B Sinkhorn Algorithm
In this section, we provide the details of optimizing the objective of the optimal balanced clustering algorithm. Firstly, we smooth the objective with an entropic regularization term:
$$
\begin{array}{l} \min  _ {\boldsymbol {w} \in \Gamma} \left[ \sum_ {j = 1} ^ {k} \sum_ {i = 1} ^ {N} w _ {i j} \cdot \left\| \boldsymbol {x} _ {i} - \boldsymbol {\mu} _ {j} \right\| _ {2} ^ {2} + \epsilon \cdot \sum_ {j = 1} ^ {k} \sum_ {i = 1} ^ {N} w _ {i j} \cdot (\log (w _ {i j}) - 1) \right]. \tag {9} \\ s. t. \| \boldsymbol {w} \| _ {1} = 1, \boldsymbol {w} \succeq 0, \sum_ {i} w _ {i j} = \frac {1}{k}, \sum_ {j} w _ {i j} = \frac {1}{N} \\ \end{array}
$$
We rewrite Eq (9) with Lagrange multipliers as
$$
\begin{array}{l} \max _ {\pmb {f}, \pmb {g}} \min _ {\pmb {w}} \mathcal {J} = \left\{\sum_ {j = 1} ^ {k} \sum_ {i = 1} ^ {N} w _ {i j} c _ {i j} + \epsilon \cdot \sum_ {j = 1} ^ {k} \sum_ {i = 1} ^ {N} w _ {i j} \cdot (\log (w _ {i j}) - 1) - \sum_ {j = 1} ^ {k} f _ {j} \left[ \left(\sum_ {i = 1} ^ {N} w _ {i j}\right) - \frac {1}{k} \right] \right. \\ \left. - \sum_ {i = 1} ^ {N} g _ {i} \left[ \left(\sum_ {j = 1} ^ {k} w _ {i j}\right) - \frac {1}{N} \right] \right\}, c _ {i j} = \| \boldsymbol {x} _ {\boldsymbol {i}} - \boldsymbol {\mu} _ {j} \| _ {2} ^ {2}. \tag {10} \\ \end{array}
$$
Taking the differentiation w.r.t.  $w_{ij}$  on Eq (10), we have
$$
\frac {\partial \mathcal {J}}{\partial w _ {i j}} = 0 \Rightarrow c _ {i j} + \epsilon \log \left(w _ {i j}\right) - f _ {j} - g _ {i} = 0. \tag {11}
$$
To update our variables, we first fix  $g_{i}$  and update  $f_{j}$  with
$$
f _ {j} ^ {(t + 1)} = \epsilon \left\{\log \left(\frac {1}{k}\right) - \log \left[ \sum_ {i = 1} ^ {N} \exp \left(\frac {g _ {i} ^ {(t)} - c _ {i j}}{\epsilon}\right) \right] \right\}. \tag {12}
$$
Then we fix  $f_{j}$  and update  $g_{i}$  with
$$
g _ {i} ^ {(t + 1)} = \epsilon \left\{\log \left(\frac {1}{N}\right) - \log \left[ \sum_ {j = 1} ^ {k} \exp \left(\frac {f _ {j} ^ {(t)} - c _ {i j}}{\epsilon}\right) \right] \right\}. \tag {13}
$$
In summary, we can iteratively update  $f_{j}$  and  $g_{i}$  until we obtain the final solutions.
Table 4: Running time of the learning process on ML-10M.
ML-10M	DMF	LightGCN
RecEraser	UltraRE	RecEraser	UltraRE
Stage I	872.53m	259.18s	879.06m	257.83s
Stage II	213.55s	208.44s	860.12s	852.32s
Stage III	83.74s	67.26s	384.50s	376.57s
Total	877.48m	534.88s	899.80m	1,486.72s
C Additional Experiments on the Large-scale dataset
As illustrated in Section 7, current experiments do not sufficiently demonstrate the efficiency enhancement of U1traRE, because the majority of time is spent during stage II (independent training) while the enhancements occur in stages I (non-overlapping division) and III (model combination). Thus, following [7], we further conduct experiments on ML-10M (the largest dataset used in [7]) with 50 shards (a large shard number), and report the results in Table 4. From it, we observe that U1traRE significantly improves efficiency compared to RecEraser. Specifically, in stages I and III, UltraRE demonstrates average efficiency enhancements of  $20,227.87\%$  and  $13.30\%$  respectively. Note that, in the large-scale dataset, our proposed clustering algorithm (OBC) outperforms the BKM used in RecEraser, reducing clustering time from several hours to just a few minutes (in stage I,  $872.53\mathrm{m} / 879.06\mathrm{m}$  vs  $259.18\mathrm{s} / 257.83\mathrm{s}$ ). The experimental results demonstrate higher efficiency improvements when compared to the results reported in the main text, providing additional evidence of the substantial efficiency enhancements achieved by our method.