Adversarial Mixup Unlearning
Authors: Zhuoyi PENG
Yixuan Tang
Yi Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Machine Unlearning
Catastrophic Unlearning
Mixup Regularization
Adversarial Training
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Machine unlearning is a critical area of research aimed at safeguarding data privacy by enabling the removal of sensitive information from machine learning models. One unique challenge in this field is catastrophic unlearning, where erasing specific data from a well-trained model unintentionally removes essential knowledge, causing the model to deviate significantly from a retrained one. To address this, we introduce a novel approach that regularizes the unlearning process by utilizing synthesized mixup samples, which simulate the data susceptible to catastrophic effects. At the core of our approach is a generator-unlearner framework, MixUnlearn, where a generator adversarially produces challenging mixup examples, and the unlearner effectively forgets target information based on these synthesized data. Specifically, we first introduce a novel contrastive objective to train the generator in an adversarial direction: generating examples that prompt the unlearner to reveal information that should be forgotten, while losing essential knowledge. Then the unlearner, guided by two other contrastive loss terms, processes the synthesized and real data jointly to ensure accurate unlearning without losing critical knowledge, overcoming catastrophic effects. Extensive evaluations across benchmark datasets demonstrate that our method significantly outperforms state-of-the-art approaches, offering a robust solution to machine unlearning. This work not only deepens understanding of unlearning mechanisms but also lays the foundation for effective machine unlearning with mixup augmentation.

Published as a conference paper at ICLR 2025
ADVERSARIAL MIXUP UNLEARNING
Zhuoyi Peng Vixuan Tang Yi Yang
Department of ISOM, The Hong Kong University of Science and Technology {zpengaf, ytangch @ connect .ustuukt yiyangeust. .hk
ABSTRACT
Machine unlearning is a critical area of research aimed at safeguarding data pri- vacy by enabling the removal of sensitive information from machine learning models. One unique challenge in this field is catastrophic unlearning, where erasing specific data from a well-trained model unintentionally removes essential knowledge, causing the model to deviate significantly from a retrained one. To address this, we introduce a novel approach that regularizes the unlearning pro- cess by utilizing synthesized mixup samples, which simulate the data susceptible to catastrophic effects. At the core of our approach a generator-unleamu frame- work, Milurrn,,, where generator adversarially produces challenging mixup examples, and the unlearner effectively forgets target information based on these synthesized data. Specifically, first introduce a novel contrastive objective train the generator an adversarial direction: generating examples that prompt the unlearner to reveal information that should be forgotten, while losing essential knowledge. Then the unlearner, guided by two other contrastive loss terms, pro- cesses the synthesized and real data jointly to ensure accurate unlearning without losing critical knowledge, overcoming catastrophic effects. Extensive evaluations across benchmark datasets demonstrate that our method significantly outperforms state-of-the -iceo---te-arttt approaches, offering robust solution to machine unlearning. This work not only deepens understanding of unlearning mechanisms but also lays the foundation for effective machine unlearning with mixup augmentation.
INTRODUCTION
Machine unlearning (Bourtoule et al. 2021) has gained significant attention due to growing con- cerns about data privacy. In particular, legislators have enacted regulations such as the GDPR, which grants data owners the "right to be forgotten". This has spurred the development of ma- chine unlearning techniques that enable models to forget sensitive data. While retraining models without sensitive data can achieve accurate unlearning, it is often impractical due to the substantial computational costs involved. As result, researchers and practitioners
increasingly exploring approximate unlearning methods (Thudi et al. 2022. Chen et al.) 2023) Chundawat et al. 2023a Kurmanji et al., 2024 Shen al. 2024). These aim create models that function if they were retrained from scratch, but without the prohibitive expenses associated with full retraining.
While approximate unlearning holds promise, it faaes the issue of catastrophic unlearning-4 a phe- nomenon where a target unlearner model, during the process of unlearning certain data, inadver- tently forgets or loses knowledge should retain, resulting in a divergence from the retrained one. Although recent advances (Chen et al.) 2023 Chundawat et al., 2023a Shen et al.) 2024) have intro- duced retention operations on the remaining data facilitate preserving the model's generalizability, the issue of catastrophic unlearning persists. Despite its importance, effective strategies to address this problem remain inadequately understood (Choi et al. (024).. bridge this research gap, our paper answers the key question: How can we overcome catastrophic unlearning ensure precise prvetting-erasinng target information without compromising essential knowledge?
To understand why the retaining process cannot handle catastrophic effects, we provide toy ex- ample in Figure [] which shows the challenge of unlearning a specific class for a classifier. This example demonstrates how issues arise when the forgetting (applied the Forgetting samples) and the retaining (applied to the Remaining samples) potentially interfere with one another. Specifically, the forgetting process may disrupt the retaining process, weakening the model's ability to general-

---

Published as a conference paper at ICLR 2025
Forgetting Sample
Unseen Sample
(Forgetting harms generalizaility)
Catastrophic Unlearning
Remaining Sample
Mixup Sample
Unleaminng with Regularization on Mixup Samples (Oely Preserve the Knowledge from Aemaining)
Forget Forgetting's Info.
Retaln Remaining's Info.
Figure 1: A toy example of unlearning focal class (in orange) to motivate our solution. The shaded orange represents the effects of forgetting, while the shaded blue indicates the retention of remain- ing knowledge. This example demonstrates how forgetting can inadvertently harm the knowledge that should be retained, as shown by the overlapping shaded regions, ultimately degrading the gen- eralizability unseen data (shown as stars). Notably, strategically mixed from Forgetting and Remaining- mimic the data vulnerable to the overlap- ping catastrophic effects. By removing the forgetting effects and retaining remaining knowledge on these mixup samples, we can overcome catastrophic unlearning.
sythesize mixup samples-which
ize- ereeccall in the intermediate space between the Forgetting and Remaining samples. However, by strategically using mixup (Zhang et al. 20188) Liu et al. 2022 Qin et al. 2024), which generates intermediate samples through linear interpolation between the forgetting and remaining data, we address this issue. These synthesized mixup samples can simulate data points with catastrophic effects, enabling us to regularize the target unlearner for more effective unlearning. by minimizing the negative forgetting impact on the mixup samples, make the model behave as if were trained solely on the remaining data.
To fully leverage mixed samples for addressing catastrophie unlearning, we propose a novel generator-unlearner framework going beyond simply adopting the vanilla mixup: an adversarial generator creates challenging mixup samples from mixing Forgetting and Remaining data, and then the unlearner is regularized with these hard samples to enhance unlearning robustness. Specifically, unlike traditional mixup relying on handcrafted rules, our method incorporates mixup genera- tor trained by a proposed contrastive loss in an adversarial manner. This generator is designed to synthesize mixed samples that deliberately challenge the unlearner, causing to forget remaining knowledge while revealing information about forgetting data (a reversed direction of unlearning which removes Forgetting and retains Remaining). The target unlearner then processes both these synthetic mixed samples and real data, employing two distinct contrastive losses-one for each data type- to effectively forget concerning information while retaining essential knowledge. Notably, the overall method can perform unlearning without explicit labels for forgetting and remaining data, making it particularly suitable for scenarios where models are initially trained on datasets and labels are largely absent during unlearning. Extensive experiments across benchmark datasets demonstrate that our method significantly outperforms existing state-of-the-arr unlearning techniques (Bourtoulc et al. 2021 Chen et al. [2033] Chundawat al., [023a Kurmanji al. 20244] Choi et al. 2024; Shen et al. 2024), across both label-agnostic and label-aware setups Our work underscores the potential benefits of leveraging mixup samples for machine unlearning.
First, we introduce novel unlearning approach that leverages mixup samples to regularize the unlearner, offering a new strategy for addressing the issue of catas- trophic unlearning. Second, we extend beyond the traditional mixup by proposing generator- unlearner framework, where wwwrssrriall-ggnerrteee examples improve the unlearning process. This framework is trained using novel contrastive losses without any need for explicit labels for forgetting and retaining data. Third, empirical evaluations across multiple benchmark datasets demonstrate that our approach surpasses existing unlearning techniques. We have released the code for MiXUnlaamm
The contributions are as follows:
Sabbel-agnottcc unlearning refers algorithms that facilitate unlearning without any need for labels, which particularly advantageous in real-world scenarios where much data may be unannotated, is often the case weakly labeled or semi-supervised learning environments. contrast, label-aware unlearning methods depend on laheled data to execute the unlearning process.
https: / /i thub. com/ real jeremybat/M xun earee.rnnigg
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
General Unlearning Work. Unlearning techniques are generally divided into two categories: exact unlearning and approximate unlearning. Exact unlearning focuses on efficiently retraining a model using only the remaining data. For example, the SISA algorithm (Bourtoule et al. (02I)) enhances retraining efficiency by initially training multiple model checkpoints on distinct data shards, only retraining the specific checkpoints linked to the data that must be forgotten. As models and datasets become increasingly complex, the limitations of exact unlearning have led to the rise of approxi- mate unlcarning methods. These methods aim to modify an already trained model by utilizing both forgotten and retained data, striving to replicate the performance of model retrained from scratch (Nguyen et al. 2020 Golatkar et al. 20206 a Liu et al. 2021 Thudi et al., 2022] Chen et al.. [203]] Tarun et al. 2023] Chundawat al. 2023a Kurmanji et al. 2024). Noteworthy recent ad vance-
ments Include Chundawat et al. (2023a), which employs two teacher modeels-nne teacher helps the
unlearner retain essential knowledge, while the other guides forgetting undesired information; Chen
iet al. (2023). which refines decision boundaries to facilitate unlearning: and Thudi et al. 2022)
which reverses parameter updates associated with the Forgetting data.
Data Augmentation for Unlearning. Recent research has explored the use of data augmentation
techniques to support the unlearning process (Chundawat et al. 2023b; Tarun et al. 2023 Huang
et al. 2021 Choi et al., 2024). Notable methods include UNSIR (Tarun et al. 2023). GLI (Choi
et al., 2024), and DSMixup (Zhou et al., 2022). UNSIR generates artificial noise to increase clas- sification loss, thereby facilitating unlearning with noisy data. GLI perturbs retained samples with noise and maintains model generalizability by preserving performance on noisy retained samples. Our method offers two advantages over these approaches: (1) Unlike UNSIR, which is limited to class-level unlcarning, our approach is more versatile, supporting both class-level and data-level un- learning; and (2) While GLI attempts to preserve utility through sample perturbation, it does not adequately address catastrophic unlearning. DSMixup, one notable method, improves the SISA framework by converting retraining shards into a smaller set of mixup shards, enhancing retraining efficiency. Although DSMixup employs mixup, its purpose and adaptation differ largely from ours. DSMixup is aimed "exact unlearning" and improving retraining efficiency, while our approach focuses on mitigating catastrophic effects in well-trained models, as an "approximate unlearning" method. Additionally, DSMixup mixes data shards, whereas our method mixes Forgetting and Re- maining samples. While DSMixup prioritizes efficiency, sometimes at the expense of accuracy, our method can preserve model generalizability. To our knowledge, we are among the first leverage mixup to address catastrophic unlearning, challenge unique to approximate unlearning.
Mixup for Traditional Machine Learning. Mixup techniques have been widely used in both supervised and semi-supervised learning settings (Zhang et al. 2018 Verma et al.] 2022 Berthelot ctt al. 2019, Jin et al., 2024). Notable advanced methods include AutoMix (Liu et al. 2022) and AdAutoMix (Qin et al. 2024), which employ Iearnable generators to create mixup samples, though their optimization objectives differ. Despite their success traditional learning applications, the use of mixup strategies in the context of unlearning remains underexplored. In particular, training mixup generator to effectively enhance the unlearning process is still an open challenge.
3 PRELIMINARY
Let fo be a deep model trained on dataset D, which maps instance x E X' to a distribution y E y over class labels. Unlearning on fo is to eliminate the knowledge associated with subset of the data, denoted as Dj D, while retaining the knowledge derived from the remaining data, D, = D Dy. With an approximate unlearning approach, denoted by U, the original model fn is transformed into an unlearned model fu = U(DD, Dr, D2). The goal of the unlearned model fo is to approximate the performance of fD.. model trained solely on D,.. We refer to fD as the initial model. fu as the unlearner, D as the Forgetting dataset, and D, as the Remaining datasct.
Conceptually, the unlearning algorithm U can be categorized into two types: label-agnostic and label-aware. Label-agnostic unlearning does not require access to labels, making it particularly suited for scenarios where the initial model is trained on largely unannotated datasets, and labels are not guaranteed to be available during the unlearning phase. In contrast, label-aware unlearning relies on label information during the unlearning process facilitate removal of knowledge.
3

---

Published as a conference paper at ICLR 2025
Adversanil Optimization
Hard Mixed Sample
Forgetting
Reveal
Information
Sample
Challenge
Fixed
Generator
Unlearner
Aemoining
Lose
Knowledge
Sample
The goal of our generator to produce hard mixed samples that challenge the unlearner. These mixed samples can prompt the unlearner to reveal information related to Forgetting and si- multaneously disrupt the retention of Remaining, reversing the direction of unlearning process (the goal of unlearning is to forget Forgetting while preserving Remaining). To train the generator, we hold the unlearner fixed and update the generator's parameters with a proposed contrastive objective of Eq. 3 The unlearner then performs effective unlearning based on these mixed samples.
Figure 2:
MIXUNLEARN: AN ADVERSARIAL MIXUP UNLEARNING FRAMEWORK
Why does retention apply to the remaining data, yet catastrophic effects on generalizability persist? This issue likely arises from insufficient model regularization in intermediate regions or interpolation zones, where the effects of forgetting and retention intersect. For example, consider a classifier trained to recognize cats and dogs, where aim for the model to forget the cat class while retaining knowledge of the dog class. Although retention works for training samples of dogs, unseen dog samples that exist in the overlapping benefit. These unseen samples are likely close to the cat class in feature space, making it challenging fully mitigate the forgetting effect. Consequently, the model's ability to generalize these unseen dog samples diminishes due to the residual impact of forgetting, leading to catastrophic unlearning.
interpolation space between cats and dogs gain minimal
To mitigate catastrophic unlearning, we propose integrating mixup samples into the machine un- learning process. Mixup, introduced by Zhang et al.] (2018). is a data augmentation technique that creates new samples by linearly interpolating between two existing samples. By strategically mix- ing samples from Forgetting and Remaining, we simulate instances that experience the conflicting forces of forgetting and retention. These mixup samples allow us to regularize the unlearning pro- cess, reducing the catastrophic effects associated with them.
However, vanilla mixup may not pose significant challenge to the unlearning process, as relies on fixed patterns linear combinations that the unlearner handle with relative case. To more effectively utilize mixed samples, we propose a mixup generator that adversarially produces more challenging mixup data, optimized through novel contrastive loss. These samples are specifically crafted to exploit the weaknesses of the unlearner by pushing it to discard the Remaining knowledge while revealing information about Forgetting- the reverse process of unlearning, whose goal is to forget the Forgetling and retain the Remaining. By introducing these challenging samples, the unlearner is exposed complex scenarios, resulting in stronger regularization compared standard mixup. Next, we will introduce our ad adversarial generator and then illustrate how to incorporate these challenging mixup samples into unlearning.
4.1 LEARNING ADVERSARIAL GENERATOR TO CHALLENGE UNLEARNER
The illustration of our generator is shown in Figure 2 In the generator's forward pass, each mixed sample is mixed from one instance in Forgetting set with one in Remaining set. Unlike standard mixup, which uses simple interpolation (i.c., .mizz Ar, + (1 A)æz). ) our generator employs learnable mixing function This allows for tailoring loss function to increase the complexity of the mixed samples to challenge the unlearner. The mixed sample mim defined as:
miz = I[ A). Fjj
(1)
where g is the learnable mixing function, 24 a sample from Forgetting set, xj from Remaining set, and X, drawn from a Beta distribution, determines the mixing ratio between 2; and zz.
Mixing Forgetting and Remaining. To operationalize a learnable generator, we utilize the param- eterized network module Miz from prior work (Qin et al. 2024) the generator g. This M MixBlock can generate attention scores for each element of x; and xy. allowing for dynamic mix- ing of the two samples. Since with a customized loss function. The main distinctions of our generator compared to previous work QQin al. 2024) in traditional learning are: its novel application within the emerging domain of
is trainable, we also tailor the generator's r' behavior
4

---

Published as a conference paper at ICLR 2025
machine unlearning; 2) strategic mixup of Forgetting and Remaining samples, rather than random mixing; and 3) the introduction of a new contrastive objective that efficiently directs the generator to challenge the unlearner, will discuss in detail later.
Specifically, with learnable Block module to produce mixup samples, we extend as:
miz MiizBBock(hs((()))b0
(2)
where (xx) and (zy) are dense feature representations obtained from initial model /n F] tably, Mix. has only 66K parameters and efficient to train
No-
Adversarial Optimization. The distinctiveness of mixup examples lie in their ability to chal- lenge the unlearning process, where they force the unlearner to expose information about what is being forgotten while leading it to lose knowledge that should be retained (this is reversed di- rection of unlearning). This adversarial goal is achieved by optimizing the generator to output hard examples with proposed contrastive loss. Specifically, novel contrastive loss is applied on mixed sample z to penalize the generator g to generate hard sample (note the "_" sign to achieve adver- sarial purpose) for challenging unlearner fu:
exp ((1 (1- A) (f ,((r)) Σ log Σ Ex,EB, exp (X Sim (x )(p(),,()))))))),)))))) (/gg))))n))))) My B.B.
gen
(3)
with p(-) operation retrieves the one-hot label or "sharpened" (Goodfellww al. 2016 class distri- bution generated by initial model fo
p(x) -
(ro)=((happpa (S otherwise (label-agnostic)
the label y of x is available (label-aware)
The intuition behind the contrastive loss in oppeatts ages the model to reveal the distributional information of the forgetting sample (i.e., p(x;)) from the output of the mixed sample (su (f (1)ft)). This is reflected the denominator, where the negative sign serves an adversarial purpose. Second, it simultaneously disrupts the retention of distribu- tional knowledge about the retaining sample (p(æ,)) from the output of the mixed sample (again, Jo (aggi))) )). This is indicated by the numerator, with a negative sign o Together, this loss essen- tially reverses the unlearning process: while "the objective of unlearning" is to forget the forgetting sample and retain knowledge of the remaining sample, this loss works the opposite direction.
operates on two key principles. First, it encour-
Specifically, in Eq. 3 By denotes batch of data to be forgotten, while B, represents a batch of data to be retained. xy refers to a sample to be forgotten, Iÿ to a sample to be retained, and their mixed sample given by zij mii g(,,, F,j A). The controls the weights between two Sim for and y according to mixing ratio. SémLoss is defined as the cosine similarity loss, expressed as (1 - cosine similarity). Moreover, the hyperparameter Tgen adjusts the sensitivity of SimLoss, particularly in the denominator.
4.2 UNLEARNING WITH ADVERSARIAL MIXED SAMPLES
Tij Recall that we have obtained the mixed sample min which encapsulates information from Forget-
ting and Remaining and is designed to challenge the unlearner. We then feed aiii into the unlearner
Machine unlearning starts with a vell-trained fo, allowing us to use its existing feature extractor.
"M ix Block uses parameterized convolutional networks to extract deeper features and compute element- wise attention masks M, mask the elements in Iz. With both and zy masked, the mixed sample computed as miz I; ② M, + x, O (1 M.). where Block module can he found Section 3.2 of of et al. (2024).
tensor with the same dimensions as z., with all
elements set to 1. Full details on
'The formula of Sharpen is shown in Appendix 1.12 The Sharpen results in sharper contrasts during optimization, ensuring that the generated examples are challenging increase adversarial difficulty.
AAthoogh other adversarial objectives were explored, we found this contrastive loss perform the best. This success is attributed to the appropriate application of Sim Loss, which effectively bounds the loss terms and reduces the risk of exposing the model to large negative gradients that could negatively impact its parameters.
5

---

Published as a conference paper at ICLR 2025
fu. ensuring that the unleamer's prediction no longer retains knowledge of z4, while only preserving that of 2,. The regularization on unlearner with mixed sample is achieved by optimizing fu with:
exp ((1 2) SimLoss (2igie) ),P(z;)))
Lmix
Σ log EyEB,
(5)
E EmEB exp (A SimLoss (fo (x(g1))) p(x;))))))))))))))))) Timis)
This loss function only change Eq. with different sign and works reverse: it directs fu to remove information about while retaining the knowledge of 2j. A temperature hyperparameter Tmizs again, is used to adjust the sensitivity of the similarity loss.
To further enhance the unlearning process, we add another contrastive loss that operates on the original real samples v; and z.:
exp((iimloss((((()).....a. Σ log Ez,B exp (SimLoss (z4), ((1:))px/()))))))) ZyEB.
Treal
(6)
This loss helps the model unlearn original real examples a; and zj, reinforcing the retention of fD(z;) while forgetting distributional information (x;)) (z; is a forgetting sample and I, is remaining sample). Finally, combine the two losses weighted manner optimize the unlearner fu:
Luntcam = Lmix + whreaas
(7)
where w balances the importance of the two losses. Our overall framework operates by iteratively optimizing generator and unlearner fv- Notably, 9 only needs to learn 66K parameters in the lightweight MirBlock, significantly lower number compared to the unlearner (e.g., 11.3M pa- rameters in ResNet-18). This design contributes to the overall efficiency. To further enhance effi- ciency, we optimize g at a specific interval during adversarial training, rather than at every unlearning iteration. For example, in our experiments, we optimize g once every four iterations.
5 EXPERIMENTS
We conduct series of experiments to validate the unlearning effectiveness of Mix
5.1 DATASETS AND MODELS
Following prior works (Bourtoule et al. 2021 Chen et al. 2023 Shen et al. 2024), we conduct
experiments four datasets: CIFARTO Krizhevsky et al. 2009). SVHN (etzer et al. 201T) and MNIST (LeCun et al. 1998), FASHION- MNIST (Xiao et a 2017). For the CIFARIO and SVHN,
adopt 18-layer ResNet architecture (He et al. 2016). For the MNIST and FASHION, use a simple convolutional neural network (CNN) (LeCun et al. 1995) with two convolutional layers.
5.2 BASELINES
We employ two categories of baselines for comparison: label-aware, which require full label infor- mation, and label aabel-agnoticc, which do not rely on any explicit label to function unlearning.
Label-Aware Baselines. This category includes Retrain, the gold standard. The baselines also encompass Boundary (Chen et al., 2023). SISA (Bourtoule et al. (021). Unroll (Thudi et al.) (002). T-S (Chundawat al. 2023a)). SCRUB (Kurmanji et al. 2024). GLI (Choi et al. 2004).. DSMixup (hhou et al., (022), and Label-Agnostic Forgetting with Repairing (LAF+R) (Shen et al.) 2024).
Among these, LAF+R stands out by performing unlearning at the representation level using multiple VAEs (Kingma & Welling 2014), followed by single retraining epoch. Another notable method GLI, recent data auta--uummenttion based approach that applies augmentation to retained samples to preserve generalizability. Although DSMixup employs mixup, it serves different goal: it mixes data shards into a smaller set of mixed shards to accelerate retraining, whereas our approach mixes Forgetting and Remaining maintain generalizability. Additionally, DSMixup is designed for exact

---

Published as a conference paper at ICLR 2025
unlearning, whereas our method is tailored for approximate unlearning. Full label information is provided to both these baselines and MixUrleam during our comparisons.
Label-Agnostic Baselines. This category includes Label Agnostic Forgetting (LAF) (Shen et al, 2024). a pioneering approach. Due to the scarcity of label-agnostic basclines, we propose two base- lines. The first, RandLabel, assigns random labels to the forgetting data using MeanSquaredEroor (MSE) loss. An additional MSE loss is applied to preserve the original output distribution for the remaining data. The second, L-Mix, extends the LAF by incorporating mixup. In L-Mix, mixed samples are created by mixing forgetting data with retaining data using the standard rule. The labels for these mixed samples are mixed by random labels (for the forgetting) and the model's self- generated labels (for the retaining). An MSE loss is applied to these mixed samples and labels upon LAF. The I.-Mix intended to assess the potential of a straightforward adaptation of mixup. No label information is provided to these baselines and Mix Unlearn during our comparisons.
Full details are shown in Appendix A.3 Experiments are repeated five times with random seeds.
5.3 EVALUATION SETUP AND METRICS
We evaluate MixUnleam using unlearning setups following Shen et al. (2024). First, in the Class- Level Unlearning setup, all data from class 0 is removed. We obtain the initial model for unlearming by training with full dataset with labels. Then we get the Retrain by retraining on the labeled Remaining subset, following [Shnn et al. (204)). Second, in the Data-Level Unlearning (Basic) setup, we randomly remove 40% of the training data labeled with classes 5 through The model initialization and retraining process follow the same procedure as in the Class-Level Unlearning.
For robustness check, we introduce two additional configurations, extending the Data-Level Un- learning (Basic) setup. The first involves assessing unlearning methods in noisy label scenario, re- ferred to as Data-Level Unlearning (Noisy). The second evaluates unlearning in a semi-supervised setting, termed Data-Level Unlearning (Semi-Supervised), The key findings are presented in Sec- tion 5.8 with detailed descriptions provided in Appendix (A.6
Metrics. In the Class-Level Unlearning setup, the evaluation metrics are: Test,, (test accuracy on the remaining classes), Test, (test accuracy on the forgotten class), and ASR (attack success rate of membership inference attacks, proposed by (Shokri et al. (2017)). For the Deva-Level Unlearn- ing setups (Basic, Noisy, and Semi-Supervised), the relevant metrics include: Train, (accuracy on the remaining training data post-unlearning), Train, (accuracy the training data targeted for moval), and Test (test accuracy on the test dataset). These metrics reflect the performance, with results closer Retrain indicating more effective unlearning outcomes.
5.4 MAIN RESULTS
We present the main results upon two unlearning scenarios: class-level and data-level (Basic), along with two label configurations: agnostic and aware. For MixUnlearn, depending on label awareness, apply different forms of Eq. 4 resulting in two variants for label-agnostic and label-aware setups.
First, Tables I [ssow that our method consistently outperforms existing baselines across a range of configurations, including both label-agnostic and label-aware settings, as well as data-level and class-level unlearning (while achieving comparable results in MNIST data-level unlearning). Key metrics, such test accuracy and attack success rate (ASR), show that our method delivers compet- itive results close to Retrain. For example, in class-level unlearning tests on CIFAR- 10 and SVHN, our label-awar -wwee approach achieves average test accuracies of 87. 1.00% and 93.95%, respectively, out- performing state-of-the-an methods such as LAF+R, SISA (an exact unlearning approach). GL.I (a recent data augmentation-basen method for approximate unlearning), and DSMixup (an exact un- learning method utilizing mixup to accelerate retraining). These findings highlight the effectiveness of our unlcarning strategy. We visualize a mixed sample in Appendix AA99 and more results on ImageNet (Deng et al. 2009) with ViT (Dosovitskiy et al. 2021 are shown in SectionA.11
Second, although our proposed baseline, L-Mix, outperforms LAF by incorporating standard mixup, approach takes it step further by fully utilizing mixup with hard examples and incorporating series of effective contrastive losses. While simpler mixup implementations do lead to performance improvements, method achieves significantly greater gains. For instance, MixUrleamn improves
7

---

Published as a conference paper at ICLR 2025
upon L-Mix on CIFAR-10 in Class-Level Unlearning (86.32 vs. 82.34 in Test,). This highlights that our method is not only highly effective but also innovative in how it leverages mixup.
Third, it is crucial to recognize that the label-agnostic setup poses significantly greater challenges compared to its label-aware counterpart. The Test, metric for label-agnostic MixUnlearn clearly highlights this disparity when compared to the label-aware version, underscoring the challenges unlearning algorithms face without explicit labels. Nevertheless, our method effectively handles both label-agnostic and label-aware setups.
Table 1:
Class-Level Unlearning Performance (Mean Stdd%).. Closer "Retrain" is better. "Aware" is label-aware; "Agnostic" is label-agnostic, Bold indicates the best result for each metric.
CIFAR-10
SVHN
Method
Test.
Testy
ASR
Method
Test,
Testy
ASR
Rotrain
86.8020.89
67.98+2.21
Retrain
94.20=0.78
0z5
59.63&1.77
NegGrrdd
58.48a3.41
0.0
51.53+1.21
NegGrad
77.12a1.5
5.98&1.33
53.22 10
Boundary
82.98+1.98
1.52+0.33
62.33+2.98
Boundary
91.28*1 .02
13.12+3.91
61.34+3.05
Aware
SISA
73.01+0.56
0++
51,53+0.12
SISA
92.04+0.88
0+0
62.45+1.33
Unrolling
83.89:2..02
040
67.32+ .92
Unrolling
92.01=1.19
88. 88.1243.12
58.30&3.02
T-S
86.31+1.12
1.20+1.32
46,9843.26
T-S
92.89+0.82
7.86+2.06
48.97+0.88
SCRUB
34.12 1:23
0z0
49,89t11.2
SCRUB
20.33=0.65 6.65
040
64.21#1.12
DSMixup
64.31a2.01
50.83+1.93
DSMixup
82.31=1.9
010
50.62:1.77
GLI
84.8220.65
47.2845.70 70
73.03:0.80
GLI
93.4422.50
63.22+7.77
86. 10:0.32
87.20+0.69
0.20+0.04
56.89+1.23
LAF+R
91.35++0.73
D+0
61.8941.37
Ours
87.10H0,78
0=0
68.30+2.77
Ours 93.95=0.69
0z00
89.97+2.68
Agnostic
LAF
82.01+0.89
3.10+0.88
51.34+1.27
LAF RandLabel
84.89
0,7840.22
56.45+0.65
RandLabel
81 00.677
19.90a0.16
64.76a1.56
91.05e1 90
91.11=1.55
62. 62.5920.83
22.949
0.66+1.57
13.61+1.88
88.74
0.30+0.47
52.00=1.11
Ours
86.32+0.56
0=0
68.48+00.7
Ours
93.40=1.35
0z0
18:172272
MNIST
FASHION-MNIST
Method
Test,-
Testy
ASR
Method
Test,
Testy
ASR
Retrain
98.7920.20
010
26. 36.5611.75
Retrain
92.71=0.47
0z0
38.04 11
NegGrad
98.8940.22
81.8955.22 2.33
38. 12
NegGrad
88.91=0.92
1.2340.34
37.98&2.45
Boundary
98.43:0.37
96.24#1.32
38.8942.41
Boundary
86.41=1.78
1.32+0.38
38.45=2.15
Aware
SISA
99.1120.05
0a0
50.2440.35
SISA
92.3320.12
0s0
49 89.80.0.13
Unrolling
97.3240.76
83.4155.41
37.4584.12
Unrolling
87.41=1.22 .23
0.40=0.1 12
41 414122111
T-S
61,53+10.45
0.2140.05
36.83+3.11 12
T-S
91.51=0.64
21 21.43+5.55 55
26. 13+0.78
SCRUB
99.12=0004
89.3 39.3122.13
32.4143.21
SCRUB
91.32=0..6 64
0.51 13
35. 14:0 89
DSMixup
99.0920.18
DSMixup
91.5620.47
35.33=1.01
GL.I
98.99=0.10
99.51
61.37#0.22
GL.I
90,87=0.61
86.42s3.74
60.93=0.04
LAF-R
99.1240.05
0,2340005
25.01:1.224
LAF+R
91.8520.34
0.33+0.07
34.8 34.8922.99
Ours
98.8510.04
010
27.13
Ours 92.82=0.66
0::
38.15+2.54
Agnostic
LAF
97.9720.26
0.2740.06
49.3113.23
LAF RandLabel
89.8820.88
3.12:0.82
318911.32
RandLabel
95.68=1.21
50.91=10.88
27.89+2.12
87.96. 14
19.12.3.36
33.29
L-Mix
98,05+0.14
24.0140.19
90.15=1.33
5.01+22.94
32.95+1.50
Ours
98.58=0.07
0:0
27.37+1.60
Ours 91,80=1.33
0::0
38.07=0.65
5.5 ABLATION
We systematically remove individual components from our method (lahel-agnostic version), with the results in Table Table (Appendix [A.4]. Specifically, in the w/o MB ablation, we replace the hard mixup samples with vanilla mixup samples, controlling mix ratio with different values of ar (1) a = 0.35, which produces mixup ratios skewed close to I (the extreme case of represents no mixing); (2) a 1.5, which generates mixup ratios centered near 0.5 (indicating equal mixing); and (3) a = 0.75, which produces broader range of ratios. Notably, this ablation differs from the proposed L-Mix baseline in main results, uses our contrastive losses (Eq. 5 and Eq. 6
The results for w/o MB highlight the advantages of using learnable generator, which improves both test accuracy and ASR. Among the configurations, a = 0.75 consistently yields better results, likely due to the creation of more diverse mixup samples, in contrast to a 0.35 and a = 1.5, which tend to constrain the mixup ratios closer to or 0.5, respectively. Additionally, the combination of Lmix and Lreal performs better together than individually. Finally, the sharpening mechanism provides modest improvement. Similar trends appear in the data-level unlearning results (Appendix Section data-level unlearning setup, the inclusion of MixBlock
A.4). For example, in the MASHION-MNIS results in effective forgetting of samples, achieving an average Train, (the accuracy on forgetting data) of 91.99 11.99, which closely matches the 92.15 achieved by the Retrain. In contrast, the configura-
8

---

Published as a conference paper at ICLR 2025
Table 2: Data-Level (Basic) Unlearning Performance (Mean Std
CHFAR-10
SVHN
Method
Train,
Trainy y
ASR
Method
Train,
Traing
Test
ASR
Retrain
84.15a0.37
77.85&1.56
86.9920.78
57.42+1.33
Retrain
83.70=0.35
75.38+1.03
93.4420.88
58.58+1.59
NegGrad
78.5620.83
68. 18.78+3.12
82.56+1.23
56.03+0.67
NogGrad
81.22=0.58
68.89+149
11
17,86+1.22
Boundary
55.3421.58
16.9923.13
52.8943.67
1..34
Boundary
65.12=1.56
30.0142.01
72.02=1.12
87.58+3,91
Aware
SISA
66.81+0.22
53.45=0.68
14.49=0.10
37.88:0.009
SISA
83.01=0.25
67.9110.57
82.98=0.96
51,03+0.58
Unrolling
58.34+1.78
31.04+1.52
60.34+1.89
17.01+1.22
Cnrolling
70.12=2.13
47.23+2.09
83.88=0.78
55.45+1.23
T-S
71.54+1.42
71.98+312
77.01=3.21
55
T-S
78.52:0.55
73.01+0.89
91.01+0778
56.12+1.76
SCRUB
29.05+1.45
0.32+0.10
23.8921.23
56.01+2.02
SCRUB
23.45=0.10
0.23+0.05
20,89=0.23
64.3242.33
DSMixup
58.19=1.55
53,97H11.39
63.44a1.22
51.12+1.49
DSMisup
73.33=0.98
65.0340.87
52.26:1.10
GLI
79.19:1.28
13
84.34=1.08
73.37+0.83
GLI
82.10=0.83
76.0841.51
93.10=0.36
78.48:0.36
LAF+R
79.20x11.20
79.20+0.91
14.8
57.8440.888
LAF+R
83.40:0.566
76. 12x0.68
93.6820.77
57.9120.44
Ours
82.01=0.68
76.97=0.52
85.99=1.15
57.45=00.7
Ours 83.37=0.39
75.89:40.68
93.3==0,55
58.
Agnostic
LAF
78.12+1.03
73.4243.56
82.33a2.03
57.70a0.77
LAF
81.60=0.67
76.34
92.24
57.8020.98
RandLabel
77.9042.67
72.55+2.88
83,38=2.79
56.3340.91
RandLabel
77.90=1
1.55
83.38=0.89
16.33+1.33
L-Mix
79.01+1.78
79 19.6542.21
84.56+1 146
35.8990.87
L-Mix
81 81.65=0.51
75.91+0..77
92.44=0.69
57.01+1.01
Ours
79.18=0.98
78.48
84.82+1.39
57.29=1.02
Ours
81.77=0.28
75.31=1.25
92A622047
57.8840.74
MNIST
FASHION-MNIST
Method
Train.
Trainy
Test
ASR
Method
Train,-
Train
Test
ASR
Retrain
99.50+0.08
98. 83+000
99. 13..12
49.63++064 64
Retrain
96.34=0.49
92.34+0.56
90.45+0.30 36
47.35+0.80 86
NegGrad
99.02+0.15
98.90=0.17
98.6820.35
50.67+0.49
NogGrad
93.78=0.45
89.56+0.35
89.44=0.33 34
46.3440.78
Boundary
97.55=1.05
94.9921.69
96.0521.56
46.9912.23
Boundary
57.23=3.24
46.8913.04
52.09 88
48,23+1.82
Aware
SISA
99.11=0.14
98.44s0.11
99.01=0.08
34.89.0.09
SISA
91.77=0.30
90.89+0.10
90.01=0.30
31.41+0.19
Unrolling
99.64+0.12
99.42++028
99.05=0.24
47. 07.8440.54
Unrolling
89.99=0.43
84.01+0.94
43
47.78+0.61
T-S
94.35#0.98
93.3342.51
93.67=1.25
48.01
T-S
83.01=1.34
86.8942.45
82.67=1.34
44.37=2.40
SCRUB
99.17+0.23
99.04+0.22
12
46 8840.50
SCRUB
91.12:0.12
88.2140.45
88.88+0.25
45.35=0.67
DSMixup
19.50=0.114
98. +10009
98, 98.97=0.03
47 47.2840.05
DSMiXup
90.85=0.22
88. 15+0.32
88.77=0.31
49.9 49.92=0.29
GLI
99.7320.07
99.7010.06
99.03:0.11
39.27+1.35
GLI
96.4420.88
96.3010.91
90,46:0.22
50.1920.66
LAF+R
99.4320.16
99 99.20s0.27
98. 98.7820.12
49.20%0.57
LAF+R
94,23=0.35
94.8
90.555d0.35
47.42=0.32
Ours
99.60+0.06
98.70+0.28
98.60=0.336
49.5340.73
Ours 95.60=0.86
93.01=1.01
90.50=0.25
47.36:0.22
Agnostic
LAF
98.22a0.77
97.21x1.28
97.4420.81
47.79x0.92
LAF
91.45=1.46
91.01+2.30
87.33
46.8610.76
RandLabel
98.51+0.42
97.35+0.85
97.88=0.51
47.8144065
RandLabel
92.27=1.89
93.80+2.53
88.75=1.79
45.12+1.55
L-Mix
09.23+0.07
97. 1.25
98.00=0.13
45.21+1.01
L-Mix
92.58=1.23
90, 90.99+2.67
88. 88.72+2.44
40.89+3.89 89
Ours
99.25+0.15
97.78=0.98
98.13:0.19
48.11=0.90
Ours
92.7%=1.18
91.99:2.01
88.76=1.15
46.9092069
Table 3: Ablation Results for Level Unlearning. MB denotes MixBlock.
CIFAR-10
SVHN
Method
Test,
Testy
ASR
Method
Test,
Testy
ASR
Retrain
$6.80=0.89
0:0
67.98
Retruin
94.20ma..78
0:0
19.63&1.77
w/o MB (rx 0.35)
85.01=0.53
0=0
65.2440.88
w/o MB (nr 0.35)
92.34=0. 69
0+n
62. 22.87+11 .41
w/o MB (cr 0.75)
85.42=0.33 32
0=0
w/o MB 0.75)
92.74=0.49
0z0
63.24+0.79
woo MB (ex = 1.5)
44.96=00699
0=D
65.01=0.91
w/o MB (a 1.5)
92.01=0.86
0=n
63. 17+0.99
w/o Leaa
29. 30.30=1..00 80
0=0
58.4242.58
w/o Lmat
26.89 1.32
0e0
60.91+1.00 1.0
w/o Lenis
82.15=1 69
0z0
61.30+2.31
w/o Lenis
91.68=1.00 06
0z0
50.95+2.66 66
w/o Sharpen
85.83=0.88
0z0
70.27a0.69
w/o Sharpen
93.0120.99 2099
0a0
67.1210.68
Ours
86.32:0.56
0:0
68.484(0.67
Ours
93.40=1.35
0s0
62. 18:0022
MNIST
FASHION-MNIST
Mothod
Tost,
Tosty
ASR
Method
Test,
Testy
Retrain
98. 7777990000 20
26, 16.56:1..5
Retrain
92:71
0=0
08.04+2..
w/o MB (ex 0.35)
99.00=00.22 22
30.01+1.02
w/o MB (a 0.35)
90.96=0.76
0:0
36.24s1. 12
w/o MB (a = 0.75)
99.25+60 14
0.06+0.08
29.37+0.47
w/o MB (a 0.75)
91.33++0.1
0=0
36.33+ ..09
MB (ar 1.5)
99 00.03+0.18
0z0
20.01
w/o MB (a L.5)
91.01=0.88 89
0z0
36.01+0.98
w/o Line
67.00t 1.54
1.3300.1
48.63s1..74
w/o Linal
40.5021.4
OzD
43.83=1.1 15
Luik
97.49::.225
0.10&0.06
31.30+0.75
w/o Lnis
87.60==0.38
38 38.0440)622
w/o Sharpen
98. 07.6740.14
n=0
30.35+1.19
w/o Sharpen
91.55+0.M 0.3
0=0
35.24+1. 11
Ours
98 88:800000
0=0
27.37=1.60
Ours
91.80=1.33
0:0
38.07+0.65
tion without MixBlock (w/ MB) yields Train, of 95.68, deviating much from the Retrain model. We provide a hyperparameter sensitivity analysis in Appendix A.5
5.6 VISUALIZAATION OF REPRESENTATIONS
Consistency with Retraining, As shown in Figure 3 MixUnlearn generates a representation space that closely resembles that of the Retrain. The class separation and clustering patterns are visually similar, with the airplane acting as a bridge between the bird, deer, ship, and truck classes. This indicates that MixUram effectively includes the key features of the retrained model. However, this consistency is absent in the initial model, indicating that our method leads to a significant change in the unlearner's behavior.

---

Published as a conference paper at ICLR 2025
(d) w/o Lmix Figure 3: Representation distributions in the class-level unlearning the CIFAR-IO. The blue denotes forgetting class (class O; airplane) while the other colors denote the remaining data. The involved classes are: airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck.
(a) Retrain Model (b) Initial Model
(c) LAF
(e) MixUmleam
Fangetting's
Fangestings Lowe Limoentts
rargetteg's
(a) Retrain
(b) Initial Model
(c) LAF
(d) MixUnleam
Figure 4: Kernel Density Estimate Plots for Loss Distributions. We the setting of class-level
unlearning on CIFARIO. The horizontal axis is CrossEntropyLoss value and the vertical is density.
Mitigating Catastrophic Unlearning. The incorporation of Lmix mitigates catastrophic effect. In panel (d), the absence of Lmix leads to more dispersed and less clearly defined clusters, particularly for the bird class. This happens because, when the model forgets the airplane class, it unintentionally also forgets the nearby bird class. A similar catastrophic effect is observed with the state-of-the- art LAF method. However, MixUnlearn successfully mitigates this issue, further supporting the rationale of leveraging mixup to prevent catastrophic unlearning.
5.7 KERNEL DENSITY ESTIMATE (KDE) PI.OT FOR LOss DISTRIBUTION
Following Choi et (2024) and Chen et al. (2023), we use Kernel Density Estimate (KDE) plots to visualize the distributions of two key cross-entropy losses; Forgetting (blue) and Unseen (red). successful unlearning is indicated by the visual alignment with the Retrain's loss distributions. Details on plot generation are in Appendix A.7] with results in Figure The Initial Model displays tightly clustered, lower loss values compared to the Retrain Model, while LAF (Figure 4c) shows only minor shift, suggesting weak unlearning. In contrast, MixUnleare (Figure closely matches the loss distributions of the Retrain Model, indicating superior unlearning performance.
5.8 ROBUSTNESS CHECK ON NOISY-LABEL OR SEMI-SUPERVISED SETUPS
We evaluate MixUnmean's 's robustness in complex scenarios with noisy unannotated data, with detailed results provided in Tables 5 and in the Appendix A.6 In the noisy-label setup, shows resilience, matching the Retrain model in accuracy and Attack Success Rate (ASR), outper- forming competitors, In the semi-supervin performance. These findings emphasize the robustness of Mix Unlearm in addressing both noisy and semi-supervised datasets, making it suitable for real-world applications with imperfect supervision.
setting, with unannotated data, it maintains comparable
5.9 EFFICIENCY
We show the time cost comparison Figure in Appendix A.8 time cost compared to Retrain, the unlearning gold standard, and is faster than advanced models like T-S, SCRUB, and LAF. The efficiency comes from the lightweight MixBlock architecture (66K parameters) and periodic generator updates, making MixUnleam fast and effective solution.
Our method significantly reduces
CONCLUSION
In this study, we propose a novel machine unlearning method by leveraging mixup samples to miti- gate potential catastrophic unlearning issue. The mixup samples are created in a generator-unleamee framework, where an adversarial generator creates challenging mixup examples that compel the un- leamner to draw on information from the forgetting set, while simultaneously losing knowledge from the remaining data. Then unlearner effectively performs unlearning based these challenging mixed examples. Extensive experiments conducted on benchmark datasets validate the superiority of our method, underscoring the potential of employing mixup techniques for machine unlearning.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: holistic approach to semi-supervised learning. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning, In IEEE Sym- posium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7766-7775 2023.
Dasol Choi, Soora Choi, Eunsun Lee, Jinwoo Seo, and Dongbin Na. Towards efficient machine un- learning with data augmentation: Guided loss-increasing (gli) prevent the catastrophic model utility drop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recog- nition, pp. 93 13-1 2024.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7210-7217, 2023a.
Vikram S Chundawat, Ayush Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security, 18:2345-2354, 2023b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from onput-ouuptt In European Conference on Computer Vision, pp. 383-398. Springer, 2020b.
observations.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep Learning, volume 1. MIT Press, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearn- able examples: Making personal data unexploitable. In International Conference on Learning Representations, 2021.
Xin Jin, Hongyu Zhu, Siyuan Li, Zedong Wang, Zicheng Liu, Chang Yu, Huafeng Qin, and Stan Li. A survey mixup augmentations and beyond. arXiv preprint arXiv:2409.05202. 2024.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference Learning Representations, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
11

---

Published as a conference paper at ICLR 2025
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in Neural Information Processing Systems, 36, 2024.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The Handbook of Brain Theory and Neural Networks, 3361(10):1995, 1995.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (06(1)):227822222 1998.
Gaoyang Liu, Xiaoqiang Ma, Yang Yang, Chen Wang, and Jiangchuan Liu. Federaser: Enabling efficient client-level data removal from federated learning models. In JEEE/ACM International Symposium on Quality of Service (IWQOS), pp. 1-10. IEEE, 2021.
Zicheng Liu, Siyuan Li, Di Wu, Zihan Liu, Zhiyuan Chen, Lirong Wu, and Stan Z Li. Automix: Unveiling the power of mixup for stronger classifiers. In European Conference on Computer Vision, pp. 441 458.. Springer, 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, volume 2011, pp. 2011.
Quoc Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Variational bayesian unlearning. Advances in Neural Information Processing Systems, 33: 16025- 2020.
Huafeng Qin, Xin Jin, Yun Jiang, Mounîm Ylcoubi,,,, and Xinbo Gao. Adversarial automixup. In International Conference Learning Representations, 2024.
Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Tony Chen, and Miao Xu. Lanes-agnostic forgetting: A supervision-free unlearning in deep models. In International Conference on Learning Representations, 2024.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In IEEE Symposium on Security and Privacy (SP), pp- 3-18. IEEE, 2017.
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems, 30, 2017.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In IEEE European Symposium on Security and Privacy (EuroS&P), pp. 303-319, IEEE, 2022.
Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Amo Solin, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. Neural Networks, 145:90-106, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark- 2017.
ing machine learning algorithms. arXiv preprint
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In International Conference Learning Representations, 2018.
Zhiwen Zhou, Ximeng Liu, Jiayin Li, Junxi Ruan, and Mingyuan Fan. Dynamically selected mixup machine unlearning. In IEEE International Conference on Trust, Security and Privacy in Com- puting and Communications (TrustCom), pp- 514-524. IEEE, 2022.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 HYPERPARAMETERS AND IMPLEMENTATION
In all experiments, SVHN, while 5c-5 is used for MNIST and FASHION-MNIST, The mix ratio À is sampled from beta distribution with a from {0.3, 0.5, 0.75, 1, 1.5). Tgen is searched from {0.05, 0.1, 0.5, 1,5), Tmix is searched from {1, 10, 20, 50} and Trall is searched from {2, 5. 10, 20, 40). The sharpen temperature T is set to 0.3. is tuned from {0.5. 1, 10, 20, 30). We implement MixUnlearn in PyTorch, and NVIDIA GeForce RTX 3090 is used for training. For the MNIST and FASHION-MNIST datasets, we set the unlearning epoch to 20. For the CIFAR-10 and SVHN datasets, we set the unlearning epoch to 40.
batch size of 32. A learning rate of 1e-5 is applied for CIFAR-10 and
A.2 INITIAL MODEL
For the CIFAR-10 and SVHN, we use an 18-layer ResNet (He et al., 2016) architecture, which includes two fully connected layers with output dimensions of 256 and 10. The ResNet model is trained from scratch without using pre-trained weights. For the MNIST and FASHION-MNIST datasets, employ CNN architecture (LeCun al. (195) consisting of two convolutional layers with 16 and 32 output channels, respectively. The remaining part of the CNN is composed of three fully connected layers with output dimensions of 256, 128, and 10.
To obtain initial models, we train two 18-layer ResNet models on the CIFAR datasets for 20 epochs with learning rate of 5e-5 and train a learning rate of 1e-3, following Shen et al. (2024). To ensure model convergence, we examine the leamning curves of these initial models, as shown in Figure [ Notably, we observe that the training accuracy is lower than the testing accuracy for the initial models for CIFAR-10 and SVHN. This can be attributed to the use of data augmentation techniques, such as random cropping, flipping, and normalization, which effectively reduce overfitting by introducing variability during training.
CNN models the MNIST datasets for 10 epochs with
01-
Accuracy 0S)
70
70-
40
9
10
12
15
19
10
Epoch
Epoch
CIFAR-10
SVHN
Figure 5: Learning Curve of Initial Model.
A.3 BASELINES
For the gold-standen baseline Retrain, we retrain the CNN models on the MNIST datasets for 20 epochs with a learning rate of 1e-3, while the ResNet models on the CIFAR datasets are retrained for 40 epochs with a learning rate of 5e-5.
For the other baselines, we adhere to the hyperparameters specified in their original studies for the unlearning process, while tuning additional parameters as needed to optimize performance. Neg- Grad updates the deep model's parameters by applying positive gradients to the retaining data and negative gradients to the forgetting data. We adjust the weight for the loss on the retaining data using values from {0.001, 0,1, 100). Boundary (Chen et al. 2023) modifies the deci- sion boundaries between the forgetting and retaining data to reduce the influence of the forget- ting data, with the FSGM bound tuned in the range of 0.1 to 0.5. SISA (Bourtoule et al.] (02T)) retrains the model smaller data shards from the remaining dataset and aggregates the results
13

---

Published as a conference paper at ICLR 2025
through ensembling. Unroll (Thudi et al. 2022) employs regularization technique to minimize verification error, which quantifies the divergence between the unlearned model and the retrained model. For this, we use the default parameters. T-S (Chundawat et al. 2023a) trains two teacher models separately on the forgetting and retaining data, aligning a student model based on differ- ences in the output spaces of the two teachers.] We tune the weight for the loss on retaining data from {0.0001, 0.01, 0.1, 1, 10, } and the temperature from {0.5, 1, 2, 2,3). The student model then achieves unlearning based on the output spaces of these two teachers (each teacher trained for 10 epochs). SCRUB (Kurmanji et al. 2024) enforces consistency between the model and teacher model trained on the retaining data while ensuring inconsistency with a teacher model trained on the forgetting data. We tune the weight for the retaining teacher from {0.0001, 0.01, 0.1, 1, 10, 100) and the temperature from (0.5, 2.33.. DSMixup (Zhou et al. 2022) accelerates SISA by transforming the data shards required for retraining into smaller set of mixed data shards; we mix two shards in this process. GLI (Choi et al. 2024) perturbs the retaining samples and trains the unlearner on these perturbed samples to maintain generalizability. We tune the perturbation iteration from 2 to 5. For LAF (Shen et al. 2024), we use the best parameters reported in the original paper. For Randlabel, tune the weight for retaining loss from {0.0001, 0.01, 0.1, 1, 10, 100). For L-Mix, tune alpha (i.e., the parameter from the Beta distribution) from {0.3, 0.5, 0. 75, 1, 1.5).
Discussion on Terminating the Unlearning Process. We adopt the common practice of using fixed number of unlearning epochs to terminate the unlearning process, as suggested in prior work (Chundawat et al., 2023a| Kurmanji al. 2024, Choi et al., [0244 Shen et al. 2024). We also track the loss to inspect the progression of the unlearning process. However, determining the optimal strategy for terminating unlearning remains an open question in the field. Unlike traditional learning processes, where validation sets can be used for early stopping, unlearning lacks a clear, standardized approach for effective termination. Should termination rely on metrics such as Train,. Trainy, Test, or the Attack Success Rate? Or perhaps a weighted combination of these metrics? For now, we employ the straightforward strategy of setting a fixed unlearning budget to terminate the process. Addressing the question of how to effectively terminate unlearning processes is left as promising avenue for future research.
A.4 ABLATION RESULTS ON DATA-LEVEL UNLEARNING
Table 4: Ablation Results for Data-Level Unlearning (Basic). MB denotes MixBlock.
CIFAR-10
SVHN
Method
Train-
Trainy
Test
Method
Train,
Trainy
ASR
Reirain
84.1540.37
77.85a1.5 56
86.9920.78
57.42+1.33
Retrain
83.70.0.35
75.38a1.03
93.4420.88
58.58&1.59
w/o MB
78.86+0.67
78.57=0.88
84.70=0.96
55.1940.71
wio MB
81.76+0.27
56.15
92.06+0.29
56.74:0.95
w/o Lyaal
19.92+1.47 .47
0z:0
16.27+0.76
55.52+0.72
w/o Leeal
22.26+ 1.2
0+0
21.96+1.45
63.35e1 .23
w/o Lens
78.65+0.98
80.57:1.11
84.6220.99
53.42+0.65
who Loux
81.6140.66
77.8920.56
92.4120.96
56.32=1.21
wob Sharpen
78.9740.77
78 12.12:
84.71a1.11
56.9910.87
wh Sharpen
81.67200.88
75.9820.94
92.3320.56
$6.6621.09
Qurs
79.18+0.98
18.48=1.25
84.82+1.39
57.29+1.02
Ours
81.77+0.28
75.31+1.25
92.46+0.47
57.88=0.74
MNIST
FASHION-MNIST
Method
Train-
Trainy
Test
ASR
Method
Train,
Trainy
Test
ASR
Retrain
99.5020.00 08
98.8320.06
99.13a0.12
49.63140.64
Retrain
96.34200.49
92, 92.3420.56
90.4510.36
47.3520.86
w/o MB
98.39:40.4
95.62=0.87
97.12+0.88
46.3240.54
w/o MB
92.54+0.68
95.68+1.32
$8.68+0.97
45.99=1.12
w/ Freal
16+t1.77
30.29+1.56 1656
49.20+
47.89+1.22 1.24
w/o Treall
27.46+1.03
4.42+0.41
22.44+0.78
43.99+0.78
wo Leus
99.78+0.08
99 99.8620.07
98.9620.08
48.04=1.01
wio Louxx
94.45+0.99
96.79:0.95
88.56+1.34
43.87=1.19
wóo Sharpen
98.97a0.34
97. 97.6550.94
98.0120.35
47.97110.68
wob Sharpen
92.35x1.01
91.8621.67
88.4 38.43300.96
46.0120.76
Ours
99.25a0.15
97. 97.78:0.98
13
48.0140
Ours
98.78
91.99=2.01
88.76&1.15
46.90=0.69
The ablation results in Table show that the full configuration (Ours) consistently outperforms con- figurations where key components, such as MixBlock (MB), Lical- and Lmix, are removed. Specif- ically, removing MB results in a noticeable performance drop. For example, on the FASHION- MNIST dataset, the full configuration achieves much closer Train, to Retrain, compared to the version without MB (91.99% vs. 95.68%) highlighting that MixBlock helps the model forget more effectively. Excluding Lreat leads a significant reduction in both training and test accuracy, empha- sizing its crucial role in the model's performance. While the removal of Lmix also results in lower
"This baseline is an extended T-S Chundawat et al. 2023ab. which employs trained models as teachers (e.g., partially retrained models) rather than using an initial model alongside random- paraamttrtraamteiiieeeee model as teachers.
14

---

Published as a conference paper at ICLR 2025
66.3
11.
352
16.3
063
11.1
363
05.2
35.0
10
15
20
25
0.4
96
58
3.0
3.
3.4
0
5
10
15
(a) Adversarial Interval
(b) Alpha for Sampling
(c) for Lreal
(d) Tgen in Lgen
Figure 6: Hyperparameter Sensitivity Analysis on CIFAR-10 class-level unlearning. The X-axis represents the hyperparameter, and the Y-axis shows test accuracy (%) for the remaining classes. The ideal test accuracy for the remaining classes is 87.01% % (Retrained model).
performance, its impact is less severe. The Sharpen component has only marginal effect overall performance, In summary, these results suggest that Leaa is essential for effective unlearning, while both MB and Linx also contribute positively to performance.
A.5 SENSITIVITY ANALYSIS ON HYPERPARAMETERS
We perform a hyperparameter sensitivity analysis based class-level unlearning on CIFAR-10, with the results presented in Figure The hyperparameters analyzed include: 1) the update interval for the generator; 2) the alpha parameter for configuring the Beta distribution (Beta(a, a:) used to sample A; 3) the loss weight for Lieal. and 4) the temperature Tyen in the SimLoss of Lgan-
Figure 6 underscores the significance of tuning these parameters optimize the performance of Mnlearn. First, the generator update interval must be well-configured if it in a small value, the generator may update too quickly, preventing the unlearner from keeping pace. Second, tuning the parameter is critical. If a is too small, the Beta distribution becomes concentrated at the extremes, overlooking the middle range, which can lead to missed valuable samples. Third, the weight for Treal requires careful balancing. Lastly, fine-tuning the sensitivity of Sossoss, controlled by Tgen> essential to optimize performance, as it influences temperature scaling during generator training. Despite this, the analysis shows that MiXUrra is generally robust across range of values for these hyperparameters, maintaining strong performance.
A.6 EXPERIMENTS ON NOISY-LABEL. OR SUNI-SUPERVISED SCENARIOS
For robustness check, we introduce two additional configurations for evaluation, extending the Data- Level Unlearning (Basic) setup:
Data-Level Unlcarning (Noisy). In this setup, we randomly assign incorrect labels to 60% of the training data labeled 0 to 4. These mislabeled samples then designated as data be forgotten. The initial model is trained with these noisy labels, while the retrained model is trained with the remaining data (note that remaining data's labels are clean), creating a more complex unlearning environment (Shen et al. 2024). We assess label-aware methods within this noisy context.
Data-Level Unlearning (Semi-Supervised). Here, 60% of the training data remains unannotated. Subsequently, 40% of the samples labeled 5 to 9 are randomly selected for removal. Both the initial and retrained models are trained in a semi-supervised manner using the established Mean- Teacher (Tarvainen & Valpola| 2017) introduces complexity in unlearning approximate semi-supervisedly retrained model. We evaluate label-agnostic methods in this context.
We show the corresponding results in Table 5 and Table In the noisy-label scenario, our method demonstrates robust performance, as seen in both the accuracy and ASR (Attack Success Rate) metrics which are close to Retrain. This shows that our approach can effectively handle label noise. In the semi- supervisd scenario, where a large percentage of the training data remains unannotated, method continues to show strong results. Specifically, our approach maintains high test accuracy and low ASR, highlighting its ability to deal with missing labels. This is critical real-world applications, where full supervision is often not feasible. Overall, these results demonstrate that
15

---

Published as a conference paper at ICLR 2025
Table 5: Unlearning Performance (Mean%+Std%) of Data-Level Unlearning (Noisy). All labels, including potentially noisy ones, are provided to the unlearning algorithm.
CIFR-I10
SVHN
Method
Train,
Trainy
Test
ASR
Method
Train.
Trainy
Test
Retrain
84.30+0.53
3,240.35
84.69+1.04
59.56=0..0
Retrain
82,4640.
2.37+0.23
38.384
59.55+1.22
NegGrad
25.31=2.55
5.21
22.14+1.22
65.30=3.44
NegGrad
18.48+2.68
2.48+1.21
17.4626.08
58.25+2.05
SISA
62.31x90.27
10.30a0.15
53.0240.63
46,23=2.61
SISA
80.17a0.13
2.45a0.31
80.02a0.07
44.8420.04
LAF4R
77.80=0.89
4.88+1.20
78.30+0.79
59.09=0.88
LAF+R
79.39+0.27
2.8540.17
90.51+0.50
55.09:1.64
Ours
81.12+0.99
4.87=1.18
82.13+1.04
59.14=1.44
Ours
81,01#0.44
2.21#00.77
92.1840.88
54.13+0.52
Table 6: Unlearning Performance (Mean%+Std%% of Data-Level Unlearning S No label information is exposed to the unlearning algorithm.
CIFAR-10
SVII
Method
Train,
Traing
Test
Train,
Trainy
Test
Retrain
75.52+0.79
73.8
80.72+0.91
55.78=1.21
Retrain
81.42+1.12
70.41=0.69
90.50+0.59
57. $7.88+0.97
LAF
65.2240.75
60.03e1.04
70.13+0.95
52.47=1.0 06
LAF
77.44+0.76
71.50:0,43
87,04+0.93
54.01=1.34
L-Mix
69.57=1.45
62.8621.13
74.65+0.98
52.61=1.22
L-Mix
77,70=1.09
71.73=0.77
87,45+1.23
54.8120.89
Ours
71.35s0.89
77.65&1.25
56.35s189
78.5820.55
69.1120.88
89.3820.66
method is highly adaptable and robust, successfully mitigating the challenges posed by both noisy and semi-supervised datasets, and outperforming other state-of- -ot--o-fthhttrr methods in these settings.
A.7 PROCEDURE VISUALIZING LOss DISTRIBUTION
To generate KDE plots, compute two sets of cross-entropy loss values for each method. Forget- ting Loss represents the loss on data that been unlearned, while Unseen Loss corresponds the loss entirely new data that the model has not previously encountered. For each method, including Retrain, Initial Model, LAF, and Milurr,,,, we apply the KDE technique to smooth the raw loss values, providing clearer visualization of the density of loss values. This approach highlights dif- ferences in how each method manages forgetting and generalization, offering insights through the visual comparison of loss distributions.
A.8
TIME COST
Negiond
Neecary
Beecery
56A
564
15
3CTUD
SCRUD
0.3
00
LAF
Oun
Our
3200 Time (a) CIFAR10
Cosl (seconds)
(b) SVHN
Figure Time cost comparison for class-level unlearning. Experiments are conducted using NVIDIA GeForce RTX 3090. A time cost comparison for class-level unlearning is conducted using an NVIDIA GeForce RTX 3090. SISA utilizes GPU devices facilitate unlearning, whereas other methods use only single device.
Compared to Retraining, which requires a complete model retraining and is thus highly time- consuming, our method demonstrates significant superiority in efficiency. This inefficiency high- lights the advantages of approximate unlearning techniques. When compared to teacher-student models like T-S and SCRUB, which involve additional computational steps for learning from teacher and student models, our method shows much better time efficiency. Furthermore. our approach out- performs LAF, the current state-of-the-art method based VAËs with 150K parameters, in both
16

---

Published as a conference paper at ICLR 2025
time efficiency and scalability. With only 66K parameters, MixBlock architecture provides lightweight yet highly effective solution, substantially reducing time and resource costs while opti- mizing the unlearning process. As illustrated in Figure 7 our method consistently achieves low time costs across both CIFARIO and SVHN datasets.
A.9 VISUALIZING MIXED SAMPLE
(a) Airplane
(b) Frog
Mixed Sample
Figure 8: Visualization of the Mixed Sample. The experiment is conducted on class-level unlcarning CIFAR-10, targeting the unlearning of the 'Airplane' class. is set as 0.5.
We present the visualization of the mixed sample in Figure 8 From this, we observe that the mixed sample predominantly reveals features of the airplane, while the frog's characteristics discern visually. We hypothesize that this image may cause the unlearning model to primarily reveal its shortcomings in forgetting the airplane and failing to effectively retain the frog's s information.
difficult to
A.10 EFFECT OF DATA RATIO FOR UNLEARNING
We evaluate the effect of the data ratio on our method and baselines by considering different pro- portions of data removed during Data-Level Unlearning. Specifically, we randomly remove 10% to 40% of the training data labeled with classes through 9 to train the initial model. The results are illustrated the Figure 9
To better understand the impact of the data ratio unlearning performance, we quantify the gap between focal method (ours or baseline) and the retrained model. This comparison is performed using two metrics: training accuracy on the forgotten data (Train,) and testing accuracy. For a given metric, such as Train,, we calculate the absolute difference between a focal method and the retrained model. This absolute difference represents the gap between the method's 's performance and the gold standard retraining. The results of these comparisons are presented in the following figures.
The results demonstrate that increasing the data ratio for forgetting generally makes the unleam- ing task more challenging, causing larger deviations between unlearning methods and the retrained model. This trend is evident across both Train, and Test Accuracy metrics, where baseline methods (e.g.. NegGrad, TS, GLI, LAF+R) exhibit increasingly higher gaps as the forgetting data ratio grows. In contrast, our method consistently achieves a low gap relative the retrained model, maintaining robust performance across all data ratios and datasets (CIFAR-10 and SVHN). This highlights the effectiveness of our approach in handling the unlearning task with minimal deviation, even under more difficult conditions, and underscores its superiority over baseline methods.
A.11 EXPERIMENTS ON IMAGENET AND VIT
We conduct experiments on the ImageNet dataset (Deng et al. 2009), using the existing VIT model (Dosovitskiy et al., 2021 as the model of interest. Specifically, utilize the pre-trained weights which were trained on ImageNet-21K. For our experiments, we use the validation set of ImageNet-IP IK, consisting of 50,000 images, as the dataset manipulate the pre- trained Vision Transformer model for obtaining initial and retrained models. This dataset is further randomly split into training set of 40,000 images (to construct Forgetting and Remaining set) and
of "vit-b16-224
in2 i2--22242222
17

---

Published as a conference paper at ICLR 2025
NegGrad
10
NegGrad
8-
T5
T5
GLI LAF+R Ours
LAF++ Ours
Train, Gap
Test (%6 Gap
0.0
0.4
Data Ratio
Data Ratio
(a) CIFAR-10. 10: Trainy Gap
(b) CIFAR- 10: Test Accuracy Gap
NegGrad
NegGrad
6-
GLI
2.0
GLI
Gap
Ours
Test (96) Gop
Ours
1.5
1.0
1-
0.0
0.1
0.2
0.3
0.4
0.0
011
0.2
0.3
0.4
(c) SVHN: Train, Gap
(d) SVHN: Test Accuracy Gap
Figure 9: Unlearning gaps between a focal method and the retrained model terms of Trainy and Test accuracy,
testing set of 10,000 images (for validate testing accuracy). For simplicity and clarity, we refer to these subsets as the "training dataset" and "testing dataset" throughout this section B
To obtain the initial and retrained models, we follow specific procedure. The initial model is trained by fine-tuning "vit-b66-22--i2111"" on the 40,000 image training set (20 epochs to ensure modified
convergence). The retrained model is then obtained by training 2tttb16-224-i211k version of the training set (25 epochs to ensure convergence), constructed by removing certain data considered: class-level unlearning, where all data from 100
points. Two unlearning scenarios classes is removed from the training set, and data-level unlearning, where 500 classes are randomly selected and 80% of their data points removed.
Quantitative Results. Table highlights the effectiveness of our proposed method in achieving pre- cise unlearning while preserving critical knowledge. The table presents metrics for both class-level and data-level unlearning scenarios, including Test,, (test accuracy on remaining classes), Test, (test accuracy on forgetting classes), ASR (Attack Success Rate), Train, (training accuracy on remaining data), Train, (training accuracy on forgotten data) and Test (testing accuracy). Our approach out- performs competing methods across various scenarios. For class-level unlearning, the Test, score of 1.51 + 0.37 demonstrates our method's ''s ability to precisely forget targeted knowledge with minimal impact on the remaining data. In the data-level unlcarning scenario, our method achieves Train,
*We utilize ImageNet- I validation set due to moderate size and challenging nature, encompass- ing up to 1,000 classification categories and offering higher resolution than the data used in our main results. While our goal is to evaluate on larger datasets. the computational cost of obtaining initial and retrained mod- els els-ensuring convergence for transformer-bsee architectures and conducting repeated cettuaaiions--eeeed this impractical. Consequently, we adopt this dataset pragmatic choice for evaluation.
18

---

Published as a conference paper at ICLR 2025
Table 7: Unlearning Performance on ImageNet based on ViT.
Class-Level Unlcarning
Data-Level Unlcarning
Method
Test,-
Testy
ASR
Method
Train--
Train
Test
ASR
Retrain
78.17s1.88
0.00:00.00
35.66=2.13
Retrain
94.24 1.24
58.6343.12
72.58=2.18
19.6842.10
NegGrad
67.24=1
2.01+0.42
28.46=1.99
NegGrad
73.82+2.01
28.75+1.45
35.97=2.15
16.34+
SISA
70.50=2.09 0.2
0.00+0.00
32.23=1.34
SISA
91.41=1022
38.24+2.15 14
14.1.62
18.2442.4
T-S
76.8811.90 1.98
25.90:2.45
30.46=1.01
T-S
92.52=1.34
75.7742.09
70.09=1.4 42
16.9622.34
DSMixup
0.00+0.00
25,41=2.23
DSMixup
89.45+1
14+2.66
65.14=1
17.42+2.34
GLI
74.78=2.16
30.41+3.14
31.69=2.31
GLI
90.78+1.44
69.24+1.64
68.14=1.75
17.89:1.34
SCRUB
76.4520.96
24.08a2.97
29.0623.67
SCRUB
92.67x1.35
76.41a2.13
69.31=2.64
17.24aa.11
LAF+R
76.47=1.67
23.2543.98
30.18:1.23
LAT+R
91.01+1.11
87.1340.97
75.64=3.67
17.97+11.96
Ours
76,91=1.84
1.51+0.37
33.12+1.57
Ours 92.89t..111
68.2742.13
74.23=0.75
18,98t 1.75
Forgetting's loos Unseen's Loss
Fergetting's Loss Unsoon' Lass
Forgetiing
Unnseen's
(a) Retrain
(b) Initial Model
(c) LAF
(d) MixUrra
Figure 10: Kernel Density Estimate Plots for Loss Distributions. use the setting of class-level unlearning ImageNet based on VIT. The horizontal axis is CrossEntropyL.oss value and the verti- cal is density.
of 92.89 + 1.11 and a Test accuracy of 74.23 = 0.75, showcasing its robustness in retaining critical knowledge while effectively handling unlearning tasks.
Visualization. The KDE plots in Figure TO demonstrate that method closely replicates the behavior of the retrained model compared to the existing state-of-the-ard LAF approach. While the initial model shows significant divergence from the retrained model, our method effectively achieves unlearning and aligns the distribution to better match the retrained model.
NagGrad
75
SCRUB
GU
DS
LA+
1000
2000
3000
4000
5000
6000
Time Cost (seconds)
Figure 11: Time cost comparison for class-level unlearning methods on ImageNet using Vision Transformer (VIT). Notably, the SISA approach leverages four GPUs for distributed training. For all methods, we use mixed precision training for acceleration.
Time Cost Analysis. Figure [| highlights the time cost for various unlearning techniques. Approxi- mate unlearning methods demonstrate significant time efficiency by looping over only the forgetting and remaining data pairs, opposed to retraining, which requires iterations over the entire dataset. Notably, our proposed method achieves superior efficiency compared to other methods, showcasing substantial reduction in time cost.
19

---

Published as a conference paper at ICLR 2025
A.12 SHARPEN OPERATION
For the sharpening function, we utilize the standard approach of adjusting the temperature of categorical distribution, as outlined in Goodfellow et al. (2016). The function is defined as:
Sharpen(g):
(8)
where q represents the imput categorical distribution, i indexes a given dimension q. T is the temperature hyperparameter, and I. is the total number of categories. As T 0, the output of Sharpen(g) converges to one-hot distribution. our experiments, we set T = 0.3.
20


Reviews:
[{"review_id": "emSk4fa9hq", "reviewer": "Reviewer_MH1a", "summary": "The paper introduces a novel and unique generator-unlearner method to yield mixed examples, which helps improve  machine unlearning effectiveness. Especially , this paper use a  novel contrastive objective not only to help refine generator, but also help unlearner  retain knowledge of remaining data and forget  forgetting data.  Hence, this operation can   prevent  catastrophic unlearning issue.  Furthermore, extensive experiments  demonstrate the efficiency and effectiveness of their proposed method.", "questions": "Why MixUnlearn sometimes demonstrates better performance than retrain (e.g., on CIFAR-10)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper uses a intriguing adversarial process to solve the maching unlearning issue. They use a unique generator-unlearner adversarial process to force unlearner to  forget the forgetting and retain the remaining. This proposed method are novel and effective.\n2. The proposed method outperforms previous work in a variety of settings, including both label-agnostic and label-aware settings, and has a relatively small time cost.", "weaknesses": "1. Though their experiments are relatively comprehensive,  the underlying dataset are somehow small. More experimental results on large dataset are expected to be demonstrated (e.g. imagenet).\n\n2. The presentation of the paper needs to be improved, especially the description of the methods section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JoKPgm7TjB", "reviewer": "Reviewer_zx2Z", "summary": "The author proposes a generator unlarner framework, which uses mixup technology combined with a generator to generate hard samples to help the model retain knowledge about the remaining dataset in forgotten samples. This framework is trained using two contrastive learning loss terms, effectively addressing the issue of potentially losing critical information about retained data during the forgetting process. The author substantiates the effectiveness of the proposed scheme through experimental validation.", "questions": "Please check the questions in the weaknesses above.\nAdditionally, why does the retrained model have a different number of epochs compared to the initial model? Why is the testing accuracy of the model higher than the training accuracy on the CIFAR-10 and SVHN datasets? Does this indicate that the model may not fully converge within the specified number of training rounds, and could certain fine-tuning experimental schemes be affecting the accuracy of the results? Although the author referenced the experimental setup in [1], I think it is essential to ensure that the original model achieves full convergence before drawing any conclusions in research related to unlearning.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow, figures and tables are helpful and easy-to-understand. This scheme is relatively simple yet highly effective, as demonstrated by the experimental results, and the designed loss function is particularly noteworthy.\n2. The solution proposed by the author applies not only to traditional unlearning scenarios involving labels but also to novel label-agnostic scenarios, demonstrating a broad range of applicability.\n3. The author's visualization of the experiment is commendable, providing richer evidence to support the effectiveness of the proposed scheme.", "weaknesses": "1. The design of the author's approach does not convincingly demonstrate an actual improvement in unlearning, despite the experimental results resembling retraining. Since retraining does not engage with forgotten data, any generalization enhancements derived from this data will inevitably be lost; otherwise, complete unlearning cannot be assured. The solution proposed by the author incorporates information from forgotten data, which intuitively undermines the concept of complete unlearning.\n2. The experiment is not sufficient, and additional deletion results at varying ratios would enhance the credibility. Furthermore, the limited number of datasets and models raises concerns. I am particularly interested in whether the experimental outcomes with more complex models and datasets, such as ImageNet, would influence the results of the method.\n3. There is a lack of epoch description of the experiment of the approach, and it remains unclear how many training rounds are necessary for convergence to achieve satisfactory results. The criteria for terminating training are crucial aspects that require discussion in the context of approximate forgetting.\n4. The comparative experiments lack sufficient explanation of hyperparameters, and some state-of-the-art results appear somewhat low. I recommend that the author adhere to the hyperparameter guidelines provided in the original paper. While the author states, \"if our reproduced results align with the previously reported statistics in Shen et al. (2024), we present their results; otherwise, we provide our results,\" I believe that directly copying most of the experimental data is unacceptable. The author should revise the baseline results to reflect actual experimental findings. By the way, TS does not require further training of the teacher model; instead, the original and initialized models should serve as the teacher models for the experiment. I suggest the author review the original control scheme rather than relying solely on the introduction in [1]. If I have misunderstood any aspects, I would appreciate clarification from the author. In summary, the author's experimental section appears overly influenced by [1], which raises doubts about the authenticity of the author's experimental workload.\n\n[1] Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Chen, and Miao Xu. Label-agnostic forgetting: A supervision-free unlearning in deep models. arXiv preprint arXiv:2404.00506, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9u5ptdD2xC", "reviewer": "Reviewer_KJXn", "summary": "This paper addresses the challenge of machine unlearning focusing on minimizing the adverse effects of removing data points from forgotten labels. The forgetting process often disrupts the retention of other knowledge, reducing the model’s generalisation, particularly in the intermediate space between forgotten and retained samples.\n\nTo mitigate this issue, the authors use data mixup, generating challenging samples through training a generator. These mixed samples reveal forgotten information while disrupting the retention of remaining data, effectively reversing the unlearning process. The unlearner is then trained on these mixup and real samples using contrastive-learning-based losses to improve robustness and efficacy in unlearning.", "questions": "Please see the Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-organised, with clear motivations, methodology, and results. \n\nThe research addresses a relevant problem and is sound, with a well-motivated toy example that illustrates the proposed approach effectively. \n\nThe experimental results provide an adequate comparison to existing methods, that support the method's effectiveness.", "weaknesses": "I have no major concerns regarding this work. However, I do have two extended questions:\n\n- I understand that the authors follow experimental setups from previous works on machine unlearning, focusing on low-resolution images and older network architectures. While this is consistent with prior works, these setups may not fully reflect recent advancements in deep learning. I suggest authors to conduct experiments on larger datasets (e.g. ImageNet) and newer model architectures (e.g. ViT, Swin Transformer). I am curious on additional discussion on any computational challenges in scaling up to larger models/datasets and how the method might need to be adapted.\n\n- For challenging mixup samples, the goal is to reveal forgetting samples while lose the remaining samples. In traditional mixup, using a λ close to 1 could align with this aim. I would be interested in understanding how such traditional mixup samples perform within the unlearning process. The authors could consider including an ablation experiment that replaces their adversarial mixup generator with traditional mixup using different fixed λ values (e.g. 0.9, 0.95, 0.99) and compared with their proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)