Not All Wrong is Bad: Using Adversarial Examples for Unlearning
Authors: Ali Ebrahimpour-Boroojeny
Hari Sundaram
Varun Chandrasekaran
Venue: ICML 2025 spotlightposter (2025)
Keywords: Machine Unlearning
Adversarial example
Fine-tuning
Primary area: social_aspects->safety

Abstract: Machine unlearning, where users can request the deletion of a forget dataset, is becoming increasingly important because of numerous privacy regulations. Initial works on "exact'' unlearning (e.g., retraining) incur large computational overheads. However, while computationally inexpensive, "approximate'' methods have fallen short of reaching the effectiveness of exact unlearning: models produced fail to obtain comparable accuracy and prediction confidence on both the forget and test (i.e., unseen) dataset. Exploiting this observation, we propose a new unlearning method, Adversarial Machine UNlearning (AMUN), that outperforms prior state-of-the-art (SOTA) methods for image classification. AMUN lowers the confidence of the model on the forget samples by fine-tuning the model on their corresponding adversarial examples. Adversarial examples naturally belong to the distribution imposed by the model on the input space; fine-tuning the model on the adversarial examples closest to the corresponding forget samples (a) localizes the changes to the decision boundary of the model around each forget sample and (b) avoids drastic changes to the global behavior of the model, thereby preserving the model's accuracy on test samples. Using AMUN for unlearning a random 10% of CIFAR-10 samples, we observe that even SOTA membership inference attacks cannot do better than random guessing.

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
Ali Ebrahimpour-
Hari Sundaram Varun Chandrasekaran
Abstract
and the California Consumer Privacy Act. Despite early efforts proposing "exact" solutions to this problem (Cao & Yang, 2015; Bourtoule et al., 2021), the community has favored "approximate" solutions due to their ability to pre- serve the original model's accuracy while being more com- putationally efficient (Chen et al., 2023b; Liu et al., 2024; Fan et al., 2024).
Machine unlearning, where users can request the deletion of forget dataset, is becoming increas- ingly important because of numerous privacy reg- ulations. Initial works on "exact" unlearning (e.g.. retraining) incur large computational overheads. However, while computationally inexpensive, "ap- proximate" methods have fallen short of reaching the effectiveness of exact unlearning: models pro- duced fail to obtain comparable accuracy and pre- diction confidence on both the forget and test (i.e., unseen) dataset. Exploiting this observation, we propose a new unlcarning method, Adversarial Machine UNlearning (AMUN), that outperforms prior state-of-the-ar art (SOTA) methods for image classification. AMUN lowers the confidence of the model on the forget samples by fine-tuning the model on their corresponding adversarial ex- amples. Adversarial examples naturally belong to the distribution imposed by the model on the input space; fine-tuning the model on the adversar- ial examples closest to the corresponding forget samples (a) localizes the changes to the decision boundary of the model around each forget sample and (b) avoids drastic changes to the global behav- ior of the model, thereby preserving the model's accuracy on test samples. Using AMUN for un- learning a random 10% of CIFAR-10 10 samples, we observe that even SOTA membership inference attacks cannot do better than random guessing.
Given a training set D and subset Dp D of the samples that have to be unlearned from a model trained on D) recent works on unlearning have emphasized the use of evaluation metrics that measure the similarity to the behavior of the models that are retrained from scratch on D - DD. However, prior unlearning methods do not effectively incorporate this evaluation criterion in the design of their algorithm. In this paper, we first characterize the expected behavior of the retrained-fron fro-sss---oo-ccrccccchcccc models on D Dr. Using this char- acterization, we propose Adversarial Machine UNlearning (AMUN), AMUN is a method that, when applied to the models trained on D, replicates that (desired) behavior af- ter a few iterations. The success of AMUN relies on an intriguing observation: fine-tuning a trained model on the adversarial examples of the training data does not lead to a catastrophic forgetting and instead has limited effect on the deterioration of model's 's test accuracy.
Upon receiving request for unlearning a subset Dp of the training set D, AMUN finds adversarial examples that are as close as possible to the samples in Dp. It then utilizes these adversarial examples (with the wrong labels) during fine-tuning of the model for unlearning the samples in Dr. Fine-tuning the model on these ad adversarial examples, which are naturally mispredicted by the model, decreases the con- fidence of the predictions on Dr. This decreased confidence of model's 's predictions on Dp is similar to what is observed in the models that are retrained on D - Dr. The distance of these adversarial examples to their corresponding sam- ples in Dp is much smaller than the distance of Dp to other samples in D Dr: this localizes the effect of fine-tuning to the vicinity of the samples in Dr and prevents significant changes to the decision boundary of the model and hurting the model's overall accuracy (see & 3.1).
1. Introduction
The goal of machine unlearning is to remove the influence of subset of the training dataset for a model that has been trained on that dataset (Vatter et al., 2023). The necessity for these methods has been determined by privacy regulations such as the European Union's 's General Data Protection Act
University of Illinois at Urbana- trbana-Champaign (UIUC), Illi- nois, USA Correspondence to: Ali Bbrahinppur-Borrojjny <ac20@ illinois.edux.
As we will show in $ 6, AMUN outperforms prior state- of-the-art (SOTA) unlearning methods (Fan et al., 2024) in unlearning random subsets of the training data from trained
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
classification model and closes the gap with the retrained models, even when there is access to the samples in
model (Shokri et al., 2017; Yeom et al., 2018; Song et al.,
2019; Hu et al., 2022; Carlini al., 2022; Zarifzadeh et al., 2024), and is a common evaluation metric (Liu et al., 2024; Fan et al., 2024). For further discussion on related works, see Appendix B.
D - Dp during the unlearning procedure.
To summarize, the main contributions of this work are:
We observe that neural networks, when fine-tuned on adversarial examples with their wrong labels, have lim- ited test accuracy degradation. While prior research in adversarial robustness fine-tune the models on these samples with their labels corrected, we are the first to utilize this form of fine-tuning to get lower predic- tion confidence scores on the training samples that are present in the proximity of those adversarial examples.
2. Preliminaries
We begin by introducing the notation we use. We proceed to define various terms in the paper, and conclude by intro- ducing our method.
2.1. Notation
We introduce a new unlearning method, AMUN, for classification models that outperforms prior methods. It does so by replicating the behavior of the retrained models on the test samples and the forget samples.
Assume probability distribution Px the domain of inputs X' and m classes ) = 21, .... We consider multi-class classifier F A' - y and its corresponding prediction function f(x) which outputs the probabilities corresponding to each class (e.g.. the outputs of the softmax layer a neural network). The loss function for model F is denoted f : X' x y + R+; uses the predicted scores from fax) compute the loss given the true label y (e.g.. cross-entropy loss). In the supervised setting we consider here, we are given dataset D = {(I,, va)i==( (1......, that contains labeled samples I; ~ Px with yi E V. The model F is trained on D using the loss E to minimize the empirical risk Ep (f) y)] and a set of parameters O. OD is derived for F; Op is the distribution over the set of all possible parameters O when the training procedure is performed on D due to the potential randomness in the training procedure (e.g., initialization and using mini-batch training). We also assume access to a test set Dr with samples from the same distribution x- A function g(x) is L--Lipschitz if [[ - g(2')ll2 S L|| x' ll2, Vx, x' E X.
By comparing AMUN to existing unlearning methods using SOTA membership inference attacks (MIAs), we show that it outperforms the other methods in unlearn- ing subsets of training samples of various sizes.
All code can be found in Ali-E/AMU
com/
1.1. Related Work
Farly works in machine unlearning focused on exact solu- tions (Cao & Yang, 2015; Bourtoule et al., 2021); those ideas were adapted to unlearning in other domains such as graph neural networks (Chen et al., 2022b) and recom- mendation systems (Chen et al., 2022a). The extensive computational cost and utility loss resulted in the design of approximate methods. An example is the work of Gi- nart et al. (2019), who provide a definition of unlearning based on differential privacy. Works that followed sought solutions to satisfy those probabilistic guarantees (Ginart et al., 2019; Gupta et al., 2021; Neel et al., 2021; Ullah et al., 2021; Sekhari et al., 2021). However, the methods that sat- isfy these guarantees were only applied to simple models, such as k-mmans (Ginart et al., 2019). linear and logistic regression (Guo et al., 2019; Izzo et al., 2021), convex- optimization problems (Neel et al., 2021), or graph neu- ral networks with no non-lincarities (Chien et al., 2022). Additional research was carried out to design more scal- able approximate methods, those that can be applied to the models that are used in practice, including large neu- ral networks (Golatkar et al., 2020: Warnecke et al., 2021: Izzo et al., 2021; Thudi et al., 2022; Chen et al., 2023b; Liu et al., 2024; Fan et al., 2024). However, these approx- imate methods do not come with theoretical guarantees; their effectiveness are evaluated using membership infer- ence attacks (MIAs). MIAs aim to determine whether a specific data sample was used in the training set of a trained
2.2. Definitions
Definition 2.1 (Attack Algorithm). For a given input/output
pair (s., y) E X x )/, model F, and a positive value c, an untargetted attack algorithm Ar c) = x + 82 minimizes (z(x dn, y' # 1)) such that ||xz|12 S e, where s/ E y.
Definition 2.2 (Machine Unlearning). Given the trained model F. and a subset DF C known as the forget set, the corresponding machine unlearning method is function MD θ θ that gets 6, OD as input and derives new set of parameters (aka the unlearned model) 0, Opp- where θD, is the distribution over the set of parameters when F is trained on D - Dp rather than D.
2.3. Approximate Unlearning
Using Definition 2.2, it is clear that the most straight- forward, exact unlearning method would be to retrain model F from scratch on D - Dr; this does not even use 6,. How- ever, training deep learning models very costly, and re-
2

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
training the models upon receiving each unlearning request would be impractical. Thus, approximate unlearning meth- ods are designed to overcome these computational require- ments by starting from 0, and modifying the parameters to derive 0g' s.t. 0g' _ O2 (i.e., from the same distribution).
The main difference between the predictions on Dr (unseen samples) and Dx (observed samples) is that the model's s predictions are much more confident for the samples that it has observed compared to the unseen samples.
Key Observation 1:
In the rest of the paper, we refer to Dp as the forget or unlearning set interchangeably. Its complement, DR D - Dr is the remain set. We will use the behavior of the models retrained from scratch on DR as the goal of approximate unlearning methods, and will refer to them as FR for brevity.
This basic observation has either been overlooked by the prior research on approximate machine unlearning or has been treated incorrectly. To make the unlearned models more similar to FR. prior methods have focused on degrad- ing the model's performance on Dr directly by either (a) some variation of fine-tuning on DR (Warnecke et al., 2021; Liu al., 2024), (b) choosing wrong labels for samples in Dr and fine-tuning the model (Golatkar al., 2020; Chen et al., 2023b; Fan et al., 2024), or (C) directly maximizing the loss with respect to the samples in DF (Thudi et al., 2022). Using the wrong labels for the samples of Dp or maximizing the loss on them make these methods very un- stable and prone to catastrophic forgetting (Zhang et al., 2024a) because these samples belong to the correct distri- bution of the data and we cannot force model to perform wrongly a portion of the dataset while preserving it's test accuracy. For these methods, it is important to use a small enough learning rate along with early stopping to pre- vent compromising the model's performance while seeking worse prediction confidence values on the samples in Dp. Also, most of these methods require access the set of remaining samples to use it for preventing total loss of the model's performance (e.g., by continuing to optimize the model on DR) (Golatkar et al., 2020; Liu et al., 2024).
2.4. Unlearning Settings
Many of the prior methods on approximate unlcarning for classification models require access to DR- However, in practice, this assumption might be unrealistic. The access to DR might be restricted, or might be against privacy regu- lations. Prior works do not make a clear distinction based on this requirement when comparing different approximate methods. Therefore, to make clear and accurate compari- son, we perform our experiments (see $ 6) in two separate settings: one with access to both DR and Dr, and the other with access to only Dr. We report the results for each setting separately. For comparison with prior methods, we adapt them to both settings whenever possible.
3. Motivation
We present the intuition for our proposed unlearning method in $ 1, and in & 3.2 we describe our observation about fine- tuning a model on its adversarial examples.
3.2. Fine-tuning on Adversarial Examples
3.1. A Guiding Observation
After training model F on D, this model imposes dis- tribution f(x) (e.g., softmax outputs) for all possible labels y E ) given any x E X. Since the model F is directly opti- mized on D, fxx becomes very skewed toward the correct class for samples in D. For a given sample from D, its ad- versarial examples (see Definition 2.1) are very close in the input space to the original sample. However, J makes the wrong prediction on these examples. This wrong prediction is the direct result of the learned parameters O. for the clas- sification model, and these adversarial examples, although predicted incorrectly, belong to the distribution imposed on X' by these learned parameters (i.e., even though that is not the correct distribution, that is what the model has learned).
Before designing a new unlearning method, we would like to first characterize the changes we expect to see after a successful unlearning. Because the retrained models are the gold standard of unlearning methods, we first assess their behavior DR. Dp, and Dr. To this end, we evaluate the confidence values of FR when predicting labels of DR. Dr, and Dj. Since samples in Dj are drawn from the same distribution as D, we can conclude that samples in Dr and Dr are from the same distribution. Therefore, we expect FR to have similar accuracy and prediction confidence scores on Dr (test set) and Dp.
Results: Figure 3 in Appendix F.1, shows the confidence scores (see $ F.1 for details) for ResNet-18 (He et al., 2016) model that has been retrained on D - DD, where D is the training set of CIFAR-10 (Alex, 2009) and the size of Dr (randomly chosen from D) is 10% and 50% of the size of DD (the first and second sub- figures, respectively).
Now, what happens if we insert one adversarial example (Iadv: Vadv) that corresponds to the sample (x. y) into D and make an augmented dataset D' for fine-tuning? Even before fine-tuning starts, the model makes the correct predic- tion on that (adversarial) example (by predicting the wrong label Yodu!), !) but its confidence might not be as high as the samples in D, on which the model has been trained on, Pro-
3

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
80
80
80
Adv
Test Accuracy
60
60
Orig
60
Adv-RS
40
40
Adv-RL
40
Orig-RL
20
20
Orig-AdvL
20
0-
0
5
10
15
20
0
10 Epochs
15
20
0
5
10
15
20
Epochs
Epochs
Figure 1: Effeet of fine-tuning on adversarial examples. This figure shows the effect of fine-tuning on test accuracy of a ResNet-18 model that trained on CIFAR-10, 10, when the dataset for fine-tuning changes (see $ 6.2 for details). Let Dp contain 10% of the samples in D and DA be the set of adversarial examples constructed using Algorithm 1. Adv, from the left sub-figure to right one, shows the results when DAD DA, and DA is used for fine-tuning the model, respectively. Ori r g, Adv-RS, Adv-RL, Or g-RL, and A- shows the results when DA for each of these sub-figures is replace by Dp. DARS. DARL. DRL. and AddlL accordingly. As the figure shows, the specific use of adversarial examples with the mis- mis-ppreicttd labels matters in keeping the model's 's test accuracy because DA, in contrast to the other constructed datasets belong to the natural distribution learned by the trained model.
ceeding with fine-tuning of the model on the augmented dataset increases its confidence on Tadv while making the same wrong prediction Yade- However, this fine-tuning does not change the model's performance because the newly added sample (Made Vadu) does not contradict the distribu- tion learned by the model. Since (zz, y) E D', and and Jado are very close to one another (e.g., very similar im- ages) while having different labels, optimizing the model has to change its decision boundary in that region of the input space to reach small loss for both of these samples, As a result of this balance, the model tends to decrease its con- fidence on the original sample compared to the model that was solely trained on D because there was opposing com- S
to keep the sample count similar), while Adv represents fine-tuning on Dr U DA. As the figure shows, Adv has a small degradation in test accuracy compared to Orig.
The rightmost sub-figure shows the case where Orig is fine- tuning of the model on Dr, and Adv is fine-tuning only DA Although the degradation in test accuracy increases for this case, surprisingly we see that the model still remains noticeably accurate despite being fine-tuned on a set of samples that are all mislabeled. See $ 6.2.1 for more details.
Results: As Figure (and Figure and in Appendix F.3) shows, the test accuracy of the model does not deteriorate, even when it is being fine-tuned only DA (the dataset with wrong labels). See $ 6.2.1 for further details.
ponents for its optimization on D. Note that where is often much smaller than the distance of any pairs of samples in D. This helps to localize this change in the de- cision boundary during fine-tuning, and prevent changes to models' behavior in other regions of the input space (Liang et al., 2023). In the following we elaborate on our empirical observations that verify these changes.
Key Observation 2: Fine-tuning a model on the adversarial examples does not lead to catastrophic forgetting!
Setup: We consider the training set of CIFAR-10 as 'D and
4. Adversarial Machine UNlearning (AMUN)
choose Dr to be random subset whose size is 10% of [D]. We also compute a set of adversarial examples (using Algorithm 1) corresponding to Dr, which we call Dx, Fig. shows the fine-tuning of trained R8 model for 20 epochs. Similar experiment for VGG19 models trained on the Tiny Imagenet dataset (Le& & Yang, 2015) can be found in In the leftmost sub-figure, the curve presented as Or Orig represents the test accuracy of the model when it is fine- tuned on D. The curve named Adv is fine-tuned on DUD,,, which has a similar test accuracy to Orig.
We utilize our novel observation about the effect of fine- tuning on adversarial examples (see $ 3.2) to achieve the intuition we had about the retrained models (see $ 3.1). We utilize the existing flaws of the trained model learning the correct distribution, that appear as adversarial examples in the vicinity of the samples in Dr, to decrease its confidence on those samples while maintaining the performance.
Formally, AMUN uses Algorithm to find an adversarial example for any sample in (x. y) E Dp. This algorithm uses a given untargeted adversarial algorithm Ar. that finds the solution to Definition 2.1, for finding an adversarial example
In the second sigufigure, Orig shows the test accuracy of the model when it is fine-tuned on Dr (two copies of DD
4

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
Fodu- To make sure e is as small as possible, Algorithm starts with a small and runs the attack Ar; if an adversarial algorithm is not found within that radius, it runs A5 with a larger c. It continues to perform Ay with incrementally increased e values until it finds an adversarial example; then adds it to DA The algorithm stops once it finds adversarial examples for all the samples in Dr.
that illuminated the influencing factors in the effectiveness of AMUN. To prove this theorem, we make assumptions that are common in the certified unlearning literature. The proof is given in Appendix A.
Theorem 4.1. Let D = {(x,, Uo) be a datasel of N samples and without loss of generality let (In Bu) (henceforth represented as (x, y) for brevity) be the sample that needs to be forgotten and (Fodu: Yadv) be its corre- sponding adversarial example used by AMUN such that | -adulll ô. Let R(w) represent the (unnormalized) empirical loss D' D U {[Iado Yadv)} for model I that is parameterized with w. We assume that J is L.- Lipschitz with respect to the inputs and R. is 3-smooth and convex with respect to the parameters. Let 0, represent the parameters corresponding to the model originally trained on 'D and 6, be the parameters derived when the model is trained on D - {(x, y) We also assume that both the original and retrained models achieve near-O loss on their training sets. After AMUN performs fine-tuning D' us- ing one step of gradient descent with a learning rate of 3 to derive parameters O' we get the following upper-bound for the distance of the unlearned model and the model retrained on 'D- - [(xx, v)} (gold standard of unlearning):
The reason behind minimizing the distance of for each sample is to localize the changes to the decision boundary of the model as much as possible; this prevents changing the model's behavior on other parts of the input space. For our experiments, we use PGD-50 (Madry, 2017) with & norm bound as Ay.. We set the step size of the gradient ascent in the attack to 0.1 x e, which changes with the 6 value. More details regarding the implementations of AMUN and prior unlearning methods and tuning their hyper-parameter can be found in Appendix C. Also, in Appendix F.5, we will show how using weaker attacks, such as Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014), might lead to lower performance of AMUN.
1
Algorithm 1 Build Adversarial Set (F, A, Dr, Cimit)
1: Input: Model F, Attack algorithm A, Forget set Dr, and Initial for adversarial attack
2: Output: DA: Adversarial set for Dr-
3: DA = 0
-ull3 S 110 - -uulž+ + (L8--C)
- C),
4: for (z.y) in Dr do
5: e= Einit
where C = (Tado)))) v) + (Hodd)) Madv) e(fo, ()) - E(fo. (Fado), Bedo).
6: while TRUE do
7: Tadu = A(x, e)
8: Madv = F ((ado)
According to the bound in Theorem 4. a lower Lipschitz constant of the model (I.) and adversarial examples that are closer to the original samples (lower value for 5) lead to a smaller upper bound. A larger value of C' also leads to improved upper-bound. In the following we investigate the factors that lead to larger value for C, which further clarifies some of influencing factors in the effectiveness of AMUN:
9: Yado! ! = y then
10: Break
11:
end if
12: e 2E
13:
end while
14: Add (Fodv, )odu) to DA
15:
end for
16:
Return DA
Higher quality of adversarial example in increasing the loss for the correct label on the original model, which leads to larger value for (Fadv), w).
Once Algorithm 1 constructs DA, AMUN utilizes that to augment the dataset on which it performs the fine-tuning. If DR is available, AMUN fine-tunes the model DR U Dr U 'A and when DR is not accessible, it performs the fine-tuning on DF UUU DA. Also, in the setting where the size of the Dp is very large, we noticed some improvement when using only DR U DA and DA, for those settings, respectively.
Transferability of the adversarial example generated on the original model to the retrained model to decrease its loss for the wrong label, which leads to lower value for E(fe, (xade)) ), Modr). This also aligns with lower Lipschitz constant of the model, as shown by prior work EEbrahinpour-Boroojeny et al.).
4.1. Influencing Factors
Early stopping and using appropriate learning rate dur- ing fine tinettunng phase of unlearning to avoid overfitting to the adversarial example, which does not allow low values for E(for (Nade), ), Vadv).
We also derive an upper-bound on the 2-norm of the dif- ference of the parameters of the unlearned model and the retrained model (which are gold-standan for unlearning)
5

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
The generalization of the retrained model to the unseen samples, which leads to lower value for & É(So, (r)), y)-
Once these four values are computed, the absolute value of the difference of each of them with the corresponding value for FR (the retrained models) is computed. Finally, the average of the four differences (called the Average Gap) is used as the metric to compare the unlearning methods.
Note that the first two implications rely on the strength of the adversarial example in addition to being close to the original sample. The second bullet, which relies on the transferability of adversarial examples, has been shown to improve as the Lipschitz constant decreases (Ebrahimpour- Boroojeny et al., 2024). The third bullet point is a natural implication which also holds for other unlearning methods that rely on the fine-tuning of the model. The fourth bullet point is not relevant to the unlearning method and instead relies on the fact that the retrained model should have good generalizability to unseen samples; it implies that as the size of Dr increases (i.e., [Dx] decreases) and the performance of the retrained model decreases, the effectiveness of the unlearning model also decreases. This is also intuitively ex- pected in the unlearning process. Hence, the proved theorem also justifies our earlier intuitions about the need for good quality adversarial examples that are as close as possible to the original samples (which is the goal of Algorithm 1).
The MIAs used in the recent unlearning methods by Liu et al. (2024); Fan et al. (2024) are based on the methods introduced by Yeom et al. (2018); Song al. (2019). Al- though these MIAS have been useful for basic comparisons, recent SOTA MIAs significantly outperform their earlier counterparts, albeit with an increase in complexity and com- putation cost. To perform a comprehensive comparison of the effectiveness of the unlearning methods, we utilized a SOTA MIA called RMIA (Zarifzadeh et al., 2024), in ad- dition to using the MIAs from prior works, In RMIA, the area under the ROC curve (AUC) of the MIA scores for predicting the training samples from the unseen samples is reported. Recall that in machine unlearning, the samples are split to three sets: DR. Dr. and Dr. For an unlearning method to be effective, as discussed in $ 3.1, we expect the AUC of RMIA for distinguishing the samples in DF from the ones in Dr be the same as random guessing (50% assuming balanced data). As shown in Table 1, this expectation holds for the models retrained on DR-
5. Evaluation Setup
In this section we elaborate on the details of evaluating different unlearning methods. More details (e.g., choosing the parameetta)m can be found in Appendix C.
We report the results of our comparisons for both the MIAS from prior unlearning literature and the new SOTA MIA. We will present the former one as MIS, and the latter one as FT AUC (the AUC of predicting Dr. from Dr).
5.1. Baseline Methods
We compare AMUN with FT (Warnecke et al., 2021).
5.3. Unlearning Settings
RL (Golatkar et al., 2020), CA (Thudi et al., 2022),
Another important factor missing in the comparisons of the unlearning methods in prior works is the possibility of access to DR. So, for our experiments we consider two settings, one with access DR and one with access to only Dp. We adapt each of the unlearning methods to both of these settings, and perform the comparisons in each of these settings separately. The prior unlearning methods that do not adapt to the setting where there is no access to PR (Warnecke et al., 2021; Liu et al., 2024) are excluded for the presented results in that setting.
BS (Chen et al., 2023b), 1 SSpprse (Liu et al., 2024),
and SalUn (Fan et al., 2024). We also combine the weight saliency idea for masking the model parameters to limit the changes to the parameters during fine-tuning with AMUN and present its results as AMUN SalUn (see Appendix D for more details). We use the same hyper-paramettr tuning reported by prior works. For further details, see Appendix C.
5.2. Evaluation Metries
The metic used by recent works in unlearning to evaluate the unlearning methods (Liu et al., 2024; Fan et al., 2024) considers the models retrained on DR as the goal standard for comparison. They compute the following four values for both the retrained models and the models unlearned using approximate methods:
Therefore, we perform different sets of experiments to eval- uate the unlearning methods in both settings, and hope this becomes the norm in future works in machine unlearning. In each of these two settings, we evaluate unlearning of 10% or 50% of the samples randomly chosen from D. For all the experiments we train three models on D. For each size of D. we use three random subsets and for each subset, we try three different runs of the unlearning methods. This leads to total of 27 runs of each unlearning method using different initial models and subsets of D to unleamn
Unlearn Accuracy: Their accuracy on Dp.
Retain Accuracy: Their accuracy on DR.
Test Accuracy: Their accuracy on Dr.
MIA score: Scores returned by membership inference attacks on Dr
6

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
6. Experiments
301
10-
We wish to answer the following questions:
20
-5
1. Does AMUN lead to effective unlearning of any ran- dom subset of the samples when evaluated by a SOTA MIA?
-10-
-10
Request COUnTe
Request counts
2. Does the choice of DA matter in AMUN, or can it be replaced with dataset that contains different labels or different samples that are within the same distance to the corresponding samples in Dp?
Figure 2: Multiple unlearning requests. This figure shows the evaluation of unlearning methods when they are used for unlearning for five times and each time on 2% of the training data. We train a ResNet- 18 model on CIFAR-10 10 when PR is available (left) and when it is not (right). After each step of the unlearning, we use the MIA scores generated by RMIA to derive the area under the ROC curve (AUC) for DR vs. DF and DF vs. Dr. The values on the y-axis shows the difference of these two AUC scores. A high value for this gap means the samples in DF are far more similar to DD rather than Dr and shows a more effective unlearning.
3. Is AMUN effective on adversarially robust models?
4. Does the choice of attack method matter in Algorithm used by AMUN and does transferred attack work as well?
5. How does A AMUN compare to other unlearning meth- ods when used for performing multiple unlearning re- quests on the same model?
models retrained DR. the AUC score of RMIA for pre- dicting DR from Dr (which can be considered the worst case for FT AUC score) are 64.17 and 69.05 for unlearning 10% and 50% accordingly. Similar results for unlearning in VGG19 models trained on Tiny Imagenet when unlearning 10% of DD can be found in section E.
As a quick summary, our results show that: (1) AMUN effectively leads to unlearning the samples in Dp: after un- learning 10% of the samples of CIFAR-10 from trained 18. RMIA cannot do better than random guessing (8 6.1); (2) If we replace DA with any of the aforemen- tioned substitutes, the model's accuracy significantly deteri- orates, especially when there no access to Dx ($ 6.2.1): (3) AMUN is effective for unlearning on models that are adversarially robust ($ 6.2.2); (4) using weaker attack methods, such as FGSM, in AMUN hurts the effectiveness by not finding the adversarial examples that are very close to the samples in Dr. However, they still outperform prior methods (Appendix F.5). The transferred adversarial exam- ples are effective as well (Appendix F:4); and (5) AMUN outperforms other unlearning methods when handling mul- tiple unlearning requests ($ 6.3).
We also present the results when MIS is used as the evalua- tion metric in Tables 10 and 11 in Appendix G, which sim- ilarly shows AMUN's s dominance in different unlearning settings. Moreover, we evaluate the combination of AMUN and Salur Un (see Appendix D for details) and present its re- sults as AMUN in these tables. AMUN saltin slightly improves the results of AMUN in the setting where there is no access to DR, by filtering the parameters that are more relevant to Dp during fine-tuning.
6.2. A Ablation Studies
In this subsection, we first elaborate on the effect of fine- tuning a model on its adversarial examples and compare it to the cases where either the samples or labels of this dataset change ($ 6.2.1). We then discuss AUNNs efficacy on models that are already robust to adversarial examples (8 6.2.2). We present other ablation studies on using weaker, but faster, adversarial attacks in Algorithm (Appendix F.5). In Appendix F.4, we utilize transferred adversarial examples for unlearning, as this can expedite handling the unlearning from newly trained model for which adversarial examples similar architectures are available.
6.1. Effectiveness of AMUN
In this subsection we report the results the comparisons of AMUN to other unlearning methods (see $ 5.1). We consider the unlearning settings discussed in $ 5.3, and the evaluation metrics discussed in $ 5.2. We use ResNet- 18 models trained on CIFAR-10 and VGG19 models trained on Tiny Imagenet for this experiment. We also perform an anal- ysis on the computation costs of AMUN (see section E. for the details).
Results: Table 1 shows the results of evaluation using RMIA when the unlearning methods have access to 'DR- Table 2 shows these results when there is no access to DR- As the results show, AMUN clearly outperforms prior un- learning methods in all settings. This becomes even more clear when there is no access to Dr. Note that, for the
6.2.1. FINE- ON ADVERSARIAL EXAMPLES
We want to verify the importance of DA (created by Algo- rithm 1) in preserving the model's 's test accuracy. To this end, we build multiple other sets to be used instead of DA when
7

---

Not All Wrong is Bad: Using Adversarial Examples for Unlcarning
RANDOM FORGET (10%)
RANDOM FORGET (50%)
UNLEAIN
RETAIN ACc
Test Acc
AT
Avu. GAP
UNLEAMS ACC
RETAIN
TEST Arc
AUC
Avo,
RETRAIN
91.19 +0.20
100.0 +0.00
94.33 +0.26
30,00
=0.42
0.00
92.09 +0.37
100.0 +0.00
91.85 =0.35
50.01 +0.22
0.00
FT
95.16 =0.20
96.64 #0.20
92:21
20.27
52.08
#0:34
2.06 10:10
94.21
#0.30
96.82
#0.31
91.21
+0:05
51.74
=0.95
2.17 20.15
RL
95.54 10.14
97.47
1...00
92.17
10.10
51.33
10:60
1.74 1.033
94.81
1.044
99.79
10.04
90.08
1.0 16
50.78
10.14
1 10.09
GA
98.91
99.22
93.39
60.96
+2.05
1.28
+0.47
100,00
100.00
=0.00
91.65
+0.07
63.39
=0.25
1.62 +0.05
BS
99.11
99.89
93.01
37.85
3.48
35.21
+b.11
35.67
50.16
+
55.19
32.01 +3.50
S--SPARRR
94.29 10.31
95.63 10:36
91.35
20.17
51.21
10:30
2.16
1:0000
98.00
10-17
98.71
10.33
92.79
1.0
54.44
20.47
2.67 10:11
SALUN
96.25 =0.01
98.14 10:10
93.06
10.18
30.68
5:054
1.44
4:0
96.08
40.35
99.80
1:000
91:97
4:0
50.86
1.36 20.04
AMUN
95.45 10.19
99.57
10:00
93.45
20.92
50:18
1:000
0.62
1:000
93.50
20000
99.71
20.01
92.39
0:000
49.99
20.18
0.33 20.05
AMUN,
95.02
20.15
99.58
10:04
93.29
20.04
30.72
t:070
0.68
1:038
913.56
1:007
99.72
10.00
92.52
2:0.0
49.81
=0.40
0.36 10.07
Table I: Unlearning with access to DR. Comparing different unlearning methods in unlearning 10% and 50% of D. Avg. Gap (see 5.2) is used for evaluation (lower is better). The lowest value is shown in bold while the second best is specified with underscore. As the results show, AMUN outperforms all other methods by achieving lowest Avg. Gap and AMUN achieves comparable results.
RANDOM FORGET (10%)
RANDOM FORGET (50%)
UNLEAIN
RETAIN
Test Ace
ALA
Ast. GAY
UNLEAHN
RETAIN Acc
TEST
FT ALC
Avo. GAP
RETRAIN
94.49
100.0
94.33 10
50.00
40:42
0.00
92.00
91.85 20.35
50:01
20.19
0.00
100:00 10.00
100.00
16.00
94:45
10.00
61.85
10:25
4.31
10.00
100.00
.0.00
100.00
10.00
94.57
10:34
61.99
10.10
4.29 10:00
4.77 13:50
5.07
13:54
5:09
10.38
49.78
10:30
68.53
13:45
100.00
_0.00
100.00
10.00
92.65
10:00
63.41
10:01
5.13 10:00
100:00 20.00
100.00
10:00
94.48
20.01
61.41
10
4.20
10.07
100.00
20.00
100.00
10.00
94.58
10:00
62:43
10:14
4.40
10:00
SALUN
100.00 20.00
100.00
10:00
94.47
20.10
61.09
2:
4.11
20.05
100.00
20.00
100:00
20.00
94.30
20.19
62:45
20:07
4.40 20:00
AMUN
94.28 20.37
97.47
20:10
91.67
20.01
52.24
1:0
1.94
10.13
92.77
Se
95.66
20.35 35
80.43
2:0
52.00
40:92
2.51 10:00
AMUN..atUu
94.19
97.71
91.79
20.12
51.93
2:
1:77
20.06
91,90
96.59
89.98
2:044
52.32
20.50
2.00
Table 2: Unlearning with access to only Dy. Comparing different unlearning methods in unlearning 10% and 50% of D. Avg. Gap (see $ 5.2) is used for evaluation (lower is better) when only DF is available during unlearning. As the results show, AMUN Salüin significantly outperforms all other methods, and AMUN achieves comparable results.
fine-tuning. Let us assume that AF(x, y) = (Jados Yado)- Then, these other sets are:
test accuracy, especially as we move from the leftmost sub- figure (having access DR) the right one (only using DA or its substitutes). This is due to the fact that the samples in DA in contrast to the other constructed datasets, belong to the natural distribution learned by the trained model. Therefore, even if we only fine-tune the ResNet-18 model on DA, we still do not lose much in terms of model's 's accuracy on Dr. This is surprising observation, as A contains set of samples with wrong predictions! Fig. 6 in Appendix F.3 shows similar results when size of Dp is 50% of (D).
DAduL: (ar.yadu)
DRL: : (Z, s.t. s # y. yodu (a
DARL ((aadus 1¹), s...f + y, yado (Va
DARS: {(x', yydv): s.t. x' Uniform(Xs). where X5= [ts: : 11:88 - x1Il2 = MY
In this experiment, we evaluate the effect of fine-tuning on test accuracy of ResNet-18 model that is trained CIFAR-10, when DA is substituted with other datasets that vary in the choice of samples or their labels. We assume that Dj contains 10% of the samples in D and DA is the set of corresponding adversarial examples constructed using Algorithm 1.
6.2.2. ADVERSARIALLY ROBUST MODELS
We evaluate the effectiveness of AMUN when the trained model is adversarially robust. One of the most effective methods in designing robust models adversarial training which targets smoothing the model's prediction function around the training samples (Salman et al., 2019). This has been shown to provably enhance the adversarial robustness of the model (Cohen et al., 2019) One of the effective adver- sarial training methods is by using TRADES loss introduced by (Zhang et al., 2019). We will use adversarially trained ResNet- 18 models for unlearning 10% of the samples in CIFAR-10. In addition, we will use another defense mecha- nism that less costly and more practical for larger models.
Results: In Fig. 1, Adv, from the left figure the right sub-figure, shows the results when D DA, Dp DA, and DA is used for fine-tuning the model, respectively. Oriq, Adv-RS, Adv-RL, Ori Orig-RL, and Drig-AAvii show the results when DA for each of these sub-figures is replaced by Dr. DARS, DARL. DRL, and D Adut, L,, respectively. As the figure shows, the specific use of adversarial examples with the mispredicted labels matters in keeping the model's
8

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
There is separate line of work that try achieve the same smoothness in model's prediction boundary by controlling the Lipschitz constant of the models (Szegedy, 2013). The method proposed by Boroojeny et al. (2024) is much faster than adversarial training and their results show a significant improvement in the robust accuracy. We use their clipping method to evaluate the effectiveness of AMUN for unlearn- ing 10% and 50% of the samples from robust ResNet-18 models trained on CIFAR-10.
updated model after each set of unlearning requests (shown as AMUN-A). In addition comparing AMUN-A A the other unlearning methods, we also compare it to version (shown as AMUN) that computes all the adversarial exam- ples on the original model so it can handle the unlearning requests faster upon receiving them i.e., DA is not computed on an updated model after each request; the set of requests are batched and DA is computed on the entire batch. For this experiment, we use ResNet-18 18 model trained on training set of CIFAR-10 10 (50K samples). Our goal is to unlearn 10% of the training samples (5K), but this time in 5 con- secutive sets of size 2% (1K) each. We then evaluate the effectiveness of unlearning at each step using R.M.A.
Results: Table 7 in Appendix F.2 shows the results for the adversarially trained models. For the models with controlled Lipschitz continuity, the results are shows in Table 3 (no access to D2) and Table 6 in Appendix F.2 (with access to DR). As the results show, even when there is no access to DR. AMUN still results in effective unlearning for adversar- ially robust models; RMIA does not do better than random guessing for predicting Dr from Dr. As Fig. 3 (right) shows, in the robust models, more than 97% of the adversarial ex- amples are further away from their corresponding training samples, compared to this distance for the original models. However, this does not interfere with the performance of AMUN because these robust models are smoother and tend to be more regularized. This regularization, which prevents them from overfitting to the training samples is in fact contributing factor to the improved generalization bounds for these models (Bartlett et al., 2017). This in itself con- tributes to enhanced resilience against MIAs. As seen in Tables 3 and 6, even for the clipped models retrained on DR. the AUC score of RMIA for predicting Dr from Dr (FR AUC) is very low, which shows that these smoother models.
Results: Fig. 2 shows an overview of the results for both settings of unlearning (with or without access to DR). This figure shows the effectiveness of unlearning by depicting how the samples in Dp are more similar to the test samples (Dr) rather than the remaining samples (DR). The value on the y-axis shows the difference of the area under the ROC curve (AUC) for predicting DR from Dp and Dr from DT. For the plots of each of these values separately, see Appendix H. AMUN- A performs better than all the other unlearning methods for all the steps of unlearning- Although AMUN also outperforms all the prior unlearning methods, it slightly under-performs compared to AMUN-A. This is expected, as the model's s decision boundary slightly changes after each unlcarning request and the adversarial examples generated for the original model might not be as effective as those ones generated for the new model. Note that for this experiment, we did not perform hyper- hyper-parameter tuning for any of the unlearning methods, and used the same ones derived for unlearning 10% of the dataset presented in & 6.1. For further discussion of the results see Appendix H.
RANDOM FORGET (1005)
RANDOM FORGET (6055)
RETRAIN 49.95 griz
54.08 20.s
69.01 an
50.19 to
55.01 201e
85.76
56.01
19.04
7. Conclusions
Table 3: Unlearning on adversarially robust models. Eval- uating the effectiveness of AMUN in unlearning 10% and 50% of the training samples when the models are adversari- ally robust and there is no access to DR. For this experiment e we use models with controlled Lipschitz constant which makes them provably and empirically more robust adver- sarial examples.
AMUN utilizes our new observation on how fine-tuning the trained models on adversarial examples that correspond to a subset of the training data does not lead to significant deterioration of model's accuracy. Instead, it decreases the prediction confidence values on the the corresponding train- ing samples. By evaluating AMUN using SOTA MIAs, we show that it outperforms other existing method, especially when unlearning methods do not have access to the remain- ing samples. It also performs well for handling multiple unlearning requests. This work also raises some questions for future work: (1) Since SOTA MIA methods fail to detect the unlearned samples, can this method be used to provide privacy guarantees for all the training samples?; (2) Can the same ideas be extended to generative models or Large Language Models?; (3) Can we derive theoretical bounds on the utility loss due to fine-tuning on adversarial examples?
6.3. Continuous Unlearning
We evaluate the performance of the unlearning methods when they are used to perform multiple consecutive unlearn- ing from a trained model. This is desirable capability for unlearning methods because in real-world applications there might be multiple unlearning requests and it is pre- ferred to minimize the number of times that a model needs to be retrained from scratch. The setting we envision is as follows: models are updated at each request for unlearn- ing. For AMUN, this means that DA is computed on an

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
Acknowledgments
Boroojeny, A. E., Telgarsky, M., and Sundaram, H. Spec- trum extraction and clipping for implicitly linear layers. In International Conference on Artificial Intelligence and Statistics, pp. 2971-2979. PMLR, 2024.
This work used Delta computing resources at National Cen- ter for Supercomputing Applications through allocation C1S240316 from the Advanced Cyberinfraattructure Coor- dination Ecosystem: Services & Support (ACCESS) pro- gram (Boerner et al., 2023), which is supported by U.S. National Science Foundation grants #2138259, #2 #2138286, #2138307, #2137603, and #2138296. This paper was gener- ously supported by NSF award IIS 232561.
Bourtoule, L., Chandrasckaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Cao, Y. and Yang. J. Towards making systems forget with machine unlearning. In 2015 IEFE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Impact Statement
This research advances machine unlearning, critical ca- pability for privacy compliance in Al systems. Traditional exact unlearning methods, such as retraining, are compu- tationally expensive, while approximate methods struggle to maintain model accuracy and confidence. AMUN in- troduces a novel approach that efficiently lowers model confidence on forget samples by leveraging adversarial ex- amples, ensuring targeted changes to the decision bound- ary without significantly altering overall model behavior. This breakthrough improves privacy protection by making membership inference attacks ineffective while maintaining test aceuracy, setting a new standard for efficient, privacy- preserving unlearning in deep learning. The work paves the way for scalable and effective unlearning solutions, address- ing fundamental challenge in AI regulation and ethical machine learning.
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and
Tramer, F. Membership inference attacks from first prin- ciples. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022.
Chen, C.,, Sun, F., Zhang, M., and Ding, B. Recommenda- tion unlearning. In Proceedings of the ACM Web Confer- ence 2022, pp. 2768 2778-2777, 2022a.
Chen, H., Zhang, Y., Dong, Y., Yang, X., Su, H., and Zhu, J. Rethinking model ensemble in transfer-based adversarial attacks. arXiv preprint 2031v23303,09105, 2023a.
Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., and Zhang, Y. Graph unlcarning. In Proceedings of the 2022 ACM SIGSAC conference on computer and commu- nications security, pp. 499-513, 2022b.
Chen, M., Gao, W., Liu, G., Peng, K., and Wang, C. Bound- ary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7766-7775, 2023b.
References
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I.. Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016
Chien, E., Pan, C., and Milenkovic, O. Efficient model up- dates for approximate unlearning of graph-structured data. In The Eleventh International Conference on Learning Representations, 2022.
Alex, K. Learning multiple layers of features from tiny im- ages. https:://www. cs. toronto. edu/kri//tarning featurrss-- 2009-TR. pdf, 2009.
Cohen, J., Rosenfeld, E.,, and Kolter, Z. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pp. 1310-1320. PMLR, 2019.
Bartlett, P. L.,, Foster, D. J., and Telgarsky, M. J. Spectrally- normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Di, Z., Yu, S., Vorobeychik, Y., and Liu, Y. Adversarial machine unlearning. arXiv preprint arXiv:2406.0766 007677 2024.
Boerner, T. J., Deems, S., Furlani, T. R., Knuth, S. L.., and Towns, J. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good, PEARC '23, pp. 173-176, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399852. doi: 10.1 3569951.3597555 URL https: ftttps://do..or//11 11 1145/3569951.0. 35597555.
Dwork, C. Diifeerntial privacy. In International colloquium automata, languages, and programming, pp. 1-12. Springer, 2006.
Ebrahimpour- Borooojnr A., Sundaram, H., and Chan- drasekaran, V. Training robust ensembles requires rethink- ing lipsehitz continuity. In The Thirteenth International Conference on Learning Representations.
10

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
Ebralinpour-Borojeeyy A., Sundaram, H., and Chan- drasekaran, V. Lotos: Layer-wise orthogonaliza- tion for training robust ensembles. arXiv preprint arXiv:2410.05136. 2024.
moderate-margin adversarial training. Putlern Recogni- tion Letters, 173:30-37, 2023.
Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y., SHARMA, P., Liu, S., al. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36, 2024.
Fan, C., Liu, J., Zhang, Y., Wei, D., Wong, E., and Liu, S. Salun: Empowering machine unlearning via gradient- based weight saliency in both image classification and generation. In International Conference on Learning Representations, 2024.
Liu, Y., Chen, X., Liu, C., and Song, D. Delving into transferable adversarial examples and black-box, attacks. arXiv preprint arXiv: 161 1.02770, 2016.
Ginart, A., Guan, M., Valiant, G., and Zou, J. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Lucki, J., Wei, B., Huang, Y., Henderson, P., Tramèr, F,, and Rando, J. An adversarial perspective machine unlearning for ai safety. arXiv preprint arXiv:2409. 18025, 2024.
Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020.
Madry, A. Towards deep learning models resistant to adver- sarial attacks. arXiv preprint arXiv:1706.0608; 2017.
Neel, S., Roth, A., and Sharifi- Malveerrdi, S. Descent-to- delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pp. 931-962. PMLR, 2021.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples, arXiv preprint arXiv:1412.6572, 2014.
Guo, C., Goldstein, T., Hannun, A., and Van Der Maaten, L. Certified data removal from machine learning models. arXiv preprint arXiv: 19100003, 2019.
Papernot, N., McDaniel, P., and Goodfellow, I. Transfer- ability in machine learning: from phenomena to black- box attacks using adversarial samples. arXiv preprint arXiv: (oo::6605.7220 2016.
Gupta, V., Jung, C., Neel, S., Roth, A., Sharifi-Malvaae Mallijidi,,jjrrdi, S., and Waites, C. Adaptive machine unlearning. Ad- vances in Neural Information Processing Systems, 34: 16319-16330, 2021.
Salman, H., Li, J., Razenshteyn, I,, Zhang, P., Zhang, H., Bubeck, S., and Yang, G. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in neural information processing systems, 32, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sekhari, A., Acharya, J., Kamath, G., and Suresh, A. T. Re- member what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075 18055-1088886 2021.
Hu, H., Salcie, 7., Sun, I.., Dobbie, G., Yu, P. S., and Zhang,
X. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(11s):1-37, 2022.
Setlur, A., Eysenbach, B., Smith, V., and Levine, Adver- sarial unlearning:
Reducing confidence along adversarial directions. Advances in Neural Information Processing Systems, 35:1 00118856-187777 2022.
Izzo. Z., Smart, M. A., Chaudhuri. K., and Zou, J. Approx- imate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2008-2016 PMLR, 2021.
Shokri, R., Stronati, M., Song, C.,, and Shmatikov, V. Mem- bership inference attacks against machine learning mod- els. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Jung, Y., Cho, I., Hsu, S.-H., and Hockenmaier, J. At- tack and reset for unlearning: Exploiting adversarial noise toward machine unlearning through parameter re- initialization. arXiv preprint arXh::2401.08999 2024.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Song, I.., Shokri, R., and Mittal, P. Privacy risks of securing machine learning models against adversarial examples. Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pp. 241-257, 2019.
Liang, X., Qian, Y., Huang, J., Ling, X., Wang, B., Wu, C., and Swaileh, W. Towards desirable decision boundary by

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
Szegedy, C. Intriguing properties of neural networks. arXiv preprint arXiv:/312.611 2013.
Zhao, Z., Liu, Z., and Larson, M. On success and sim- plicity: A second look transferable targeted attacks. Advances in Neural Information Processing Systems, 34: 2021.
Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing ma- chine unlearning. 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 303-319. IEEE, 2022.
Ullah, E., Mai, T., Rao, A., Rossi, R. A., and Arora, R. Ma- chine unlearning via algorithmic stability. In Conference on Learning Theory, pp. 4126-4142 PMLR, 2021.
Vatter, J., Mayer, R., and Jacobsen, H.-A. The evolution of distributed systems for graph neural networks and their origin in graph processing and deep learning: A survey. ACM Computing Surveys, 56( (6(1):1-31) 2023.
Warnecke, A., Pirch, L., Wressnegger, C., and Rieck, K. Machine unlearning of features and labels. arXiv preprinl
1rXiv:2108.1157: 2021.
Wong, E., Rice, I.., and Kolter, Fast better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994 2020.
Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security founda- tions symposium (CSF), pp. 268-282. IEEE, 2.18.
Zarifzadeh, S., Liu, P., and Shokri, R. Low-cost high-power membership inference attacks. In Forty-first International Conference on Machine Learning, 2024.
Zeng. Y., Chen, S., Park, W., Mao, Z. M., Jin, M., and Jia, R. Adversarial unlearning of backdoors via implicit hypergradient. arXiv preprint
2021.
Zhang, H., Yu, Y., Jiao, J., Xing. E., Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference machine learning, pp. 7472-7482, PMLR, 2019.
Zhang, J., Wu, W., Huang, J.-t., Huang, Y., Wang, W., Su, Y., and Lyu, M. R. Improving adversarial transferability via neuron attribution-basee attacks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14993-15002, 2022.
Zhang, R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse effective un- learning, arXiv preprint arXiv:2404.0 05868, 2024a.
Zhang, Y., Chen, X., Jia, J., Zhang, Y., Fan, C.,, Liu, J., Hong, M., Ding, K., and Liu, S. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. arXiv preprint arXiv:2405. 15234, 2024b.
12

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
Appendix
A. Proofs
Here we provide the proof of Theorem 4.1:
Proof. As we perform the unlearning by fine-tuning and performing a gradient descent update to O.. we have: 0' 8. VR((o). Therefore, we can write:
110' - 0u112
110, VR(0.) 0..12
|10. 0u113 (VR(0.)) 0, -0.) 1
[R(())1
32
& 110 0 + (R(00w) RR(O.)) + 5 (R(00)) - -R(0'))
|10
0.112 + (R(0.. R((O)),
where the inequality is derived by using the smoothness property (/ - 23(R(().) R(())) and the convexity assumption which leads the inequality: R((0)) 2 R(0.) + (VR(0.), -O,).
Next, we derive upper-bound for R(Ow)) R((0)) to replace in the above inequality. By the definition of unnormalized empirical loss on D':
R(0,)- R(('')
Σ (Ti), vi) + ((oe (t') E(fo, (x), v) (x),, (x). (x).
Σ
((fo
==1
where the last equality derived by the assumption that models trained until they achieve near-0 loss their corresponding dataset. Therefore, n- E(fe, (z.). v)) Ei= e(fer (z.), v1) 0 since the retrained model has been trained on the remaining samples and the unlearned model has been derived by single step of gradient descent on the original model, that had been trained on DD
To further simplify the derived terms above and reaching at our desired incquality, we focus on the term -E(fe (x), v). By adding and decreasing the term E(fo, y) we get:
((fo(())v)
((fe (( ((for()) ((f.,, + (ffe ((fe. (z), (1)).))) + (x)),) (1))) S -(()) LS,
where the first incquality uses the convexity of the the loss function with respect to the parameters and the third derivations due to the assumption that the original model achieves a zero loss on its training samples, including (x. y) (hence, (eloo (x), = 0). The final inequality due to the Lipschitzness assumption of model ʃ with respect the inputs.
13

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
B. Related Works (cont.)
To the best of our knowledge AMUN is the first work that considers fine-tuning of a model on the adversarial examples with their wrong labels as a method for unlearning a subset of the samples. However, upon reviewing the prior works in unlearning literature, there are several works that their titles might suggest otherwise. Therefore, here we mention a few of these methods and how they differ from our work.
To improve upon fine-tuning on samples in Dp with randomly chosen wrong labels, Chen et al. (2023b) use the labels derived from one step of the FGSM attack to choose the new labels for the samples in Dp. This method which was presented as BS in our experiments ($ 6.1), does not use the adversarial examples and only uses their labels as the new labels for samples of Dr. This corresponds to the dataset 'D Ader. in 6.2.1. As our results in Figures and show, fine-tuning the trained model on this dataset leads to catastrophic forgetting even when Dp is available. This is simply due to the fact that the samples in Dddet contradict the distribution that the trained models have already learned.
The work by Setlur et al. (2022) is not unlearning method, despite what the name suggest. They propose regularization method that tries to maximize the loss on the adversarial examples of the training samples that are relatively at higher distance to lower the confidence of the model on those examples. The work by Zhang et al. (2024b) proposed a defense method similar to adversarial training for making to unlearned LLMs more robust to jailbreak attacks on the topics that they have unlearned. Lucki et al. (2024) also study the careful application of jailbreak attacks against unlearned models, The work by Jung et al. (2024) investigate computing adversarial noise to mask the model parameters. Many of the works with similar titles, use "adversarial" to refer to minimax optimization (Zeng al., 2021) or considering Stackelberg game setting between the source model and the adversary that is trying to extract information (Di et al., 2024).
C. Implementation Details
For all the experiments we train three models on D. For each size of D (10% or 50%), we use three random subsets and for each subset, we try three different runs of each of the unlearning methods. This leads to total of 27 runs of each unlearning method using different initial models and subsets of D to unlearn. Hyper-parameter tuning of each of the methods is done on separate random subset of the same size from D, and then the average performance is computed for the other random subsets used as Dr. For tuning the hyper-parameters of the models, we followed the same range suggested by their authors and what has been used in the prior works for comparisons. Similar to prior works (Liu et al., 2024; Fan et al., 2024), we performed 10 epochs for each of the unlearning methods, and searched for best learning rate and number of steps for a learning rate scheduler. More specifically, for each unlearning method, we performed a grid search on learning rates within the range of [10-6, 10-1] with an optional scheduler that scales the learning rate by 0.1 for every or steps. For Sa 1 Un, whether is used on its own or in combination with AMUN, we searched for the masking ratios in the range (0.1,099)
The original models are ResNet- 18 models trained for 200 epochs with learning rate initialized at 0.1 and using a scheduler that scales the learning rate by 0.1 every 40 epochs. The retrained models are trained using the same hyper- preee-pprameters as the original models. For evaluation using RMIA, we trained 128 separate models such that each sample is included in half of these models. As suggested by the authors, we used Soft-Margin Taylor expansion of Softmax (SM-Taylor-Sy) Talloo-SSo---) with a temperature of 2 for deriving the confidence values in attacks of RMIA. We used the suggested threshold of 2 for comparing the ratios in computing the final scores (7 value). For controlling the Lipschitz constant of the ResNet- 18 models in 6.2.2, we used the default setting provided by the authors for clipping the spectral norm of all the convolutional and fully- colly-oonnected layers of the model to 1. For RMIA evaluations, trained 128 of these models separately such that each sample appears in exactly half of these models.
D. AMUN + SalUn
The main idea behind SalUn is to limit the fine-tuning of the model, during unlearning, to only a subset of the parameters of the model, while keeping the rest of them fixed. Fan et al. (2024) show that this technique helps to preserve the accuracy of the model when fine-tuning the model on Dp with randomly-chosen wrong labels. More specifically, they compute mask using the following equation:
mş 1 (1Vo, e(0:: Dr) || 2),,
which, basically, computes the gradient of the loss function for the current parameters with respect to Dp, and uses threshold y to filter the ones that matter more to the samples in Dp. Note that, is an element-wise indicator function. Then, during
14

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
1.0
0.25
Ferian
NNm--llloo
0.8
0.3-
0.20
Forget
0.6
0.2
0.4
0.10
0.2
0.05
0.0
0.0
10 15 0.00
-15
-10
-5
-10
10
0.0
0.5
1.0
1.5
2.0
Contidence
Comfdence
6,
Figure 3: (left) These two plots show the histogram of confidence values of the retrained model on its predictions for the remaining set (Remain), test set (Test), and forget set (Forget) during the training, when the size of the forget set is %10 (Ist plot) and %50 (2nd plot) of the training set. It also shows the Gaussian distributions fitted to each histogram. As the plots show the models perform similarly on the forget set and test set because to the retrained model they are unseen data from the same distribution. (right) This plot compares the 62 value definition 2.1 for adversarial examples generated on the original ResNer-18 18 models (x-axis) and clipped ResNet- 18 models (y-axis). The dashed line shows x = y line and more than 97% of the values fall bellow this line.
fine-tuning of the model on DF with random labels they use ms to detect the parameters of O, that get updated.
In our experiments, we try combining this idea with AMUN for updating subset of the parameters that might be more relevant the samples in Dr. We refer to this combination AMUN sulU in Tables I and 2 in 6.1 and Tables 10 and 11 in Appendix G. As the results show, AMUN constantly outperforms Sal Un and for the cases that DR is not available it also outperforms AMUN. In the setting where DR is accessible, it performs comparable to AMUN. This is probably due to the fact that when DR is not available and AMUN has access to only the samples Dp U'A. SalUn acts as regularization for not allowing all the parameters of the model that might not be relevant to Dr be updated. In the setting where DR is available, involving in fine-tuning will be a sufficient regularization that preserves models' utility while unlearning Dp.
E. Effectiveness of AMUN (cont.)
In this subsection we report the results on the comparisons of AMUN to other unlearning methods (see $ 5.1) for unlearning 10% of the training samples for VGG19 models that are trained on Tiny Imagenet dataset. We consider the unlearning settings discussed in $ 5.3, and the evaluation metrics discussed in $ 5.2. Table 4 shows the results of evaluation using RMIA when the unlearning methods have access to Dr. Table 5 shows these results when there is no access to Dr. As the results show, AMUN clearly outperforms prior unlearning methods in all settings. This becomes even more clear when there is no access to DR.
UNLEARN ACC
RETAIN ACC
TEST Ace
FT AUC
AVG. GAP
RETRAIN
55.93 #0.21
99.98 #0.00
55.96 #0.74
50.06 +0.36
0.00
FT
74.99 11.13
96.88
10.48
56.29
10.47
61.66
20.36
8.36
#0.37
RL
59.66 11.77
68.19
+1.55
50.78
#0.96
51.38
20.34
10.50
20.07
GA
0.46 10.01
0.49
10.02
0.50
10.02
49.80
20.08
52.67
20.03
BS
0.50 400.1
0.51
4:0.01
0.51
4:0.02
49.80
#0.02
52.65
#0.04
IJ-SPARSE
55.27 #1.41
60.64
#1.24
49.82
$0.72
54149
#0.20
12.80
20.52
SALUN
66.54 14.61
76.63
43.83
49.56
11.19
54.64
20.52
11.24
20.14
AMUN
62.57 60.02
93.66
40.66
55.52
#:0.07
57.33
#0.14
5.17
#0.13
AMUN+SaUU
62.96
20.92
94.42
40.49
55.80
+0.56
57.65
20.46
5.09
40.20
Table 4: Unlearning with access to Dr- Comparing different unlearning methods in unlearning 10% of Tiny Imagenet Dataset (D) from VGG19 models. Avg. Gap used for evaluation (lower is better). The lowest value is shown bold while the second best is specified with underscore. As the results show, AMUN outperforms all other methods by achieving lowest Avg. Gap and AMUN SnlUin achieves comparable results.
15

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
They imgelet
Thny Image
50000
10000
22000
4000
2.21
13.8
0:16 0.22 4.6
1.28
0.18
2.87
2.0
1.73
5.22
2002
6.6
18.0
0.61
1.28 2.30
3.22
Figure 4: Each plot shows the portion of samples for final values of e in Algorithm 1. The smallest value on the r-axis in each plot shows the initial value chosen for Algorithm 1 based running this algorithm small subset of the dataset. The left-most plot shows this distribution for CIFAR-I0 when the chosen attack algorithm is PGD-50. The two right-most plots show the distributions for the Tiny ImageNet dataset when PGD-10 and PGD-20 are used, respectively. Once the initial value is set, a run of PGD) attack is performed on all the samples. For the ones that adversarial example is not found within that radius, we perform other runs PGD attack until adversarial examples are found for all the samples. As the histograms show, the total work is equivalent to less than runs of PGD attack on the whole dataset, which is not computationally expensive.
E.1. Computational Cost
One option to fast processing of the unlearning request to run Algorithm on all the training samples after the model trained; this allows access to D for any arbitrary DF provided. The alternative, is to run Algorithm I on DF once the unlearning request is received. In either of these two cases, note that Algorithm starting from the initial value for Then for only the samples that the adversarial example is not found, we perform another iteration of attack with the updated value for We continue this procedure until all the samples of interest have corresponding adversarial example in DA Choosing reasonable initial value for can save the computation time, by avoiding initial iterations on almost all the samples without any outcome. To choose the initial value of we run Algorithm I with very small c on a small subset of Dr (e.g.. 100 samples). Then we choose Cinit such that at least 95% the samples find their adversarial examples within that distance. Using this strategy, we find that even when running Algorithm on all D, the computation time will be equivalent to running the underlying adversarial attack (e.g., PGD-50) less than 3 times on all the samples in D. Figure shows the histogram for the number of samples in D for each e value for different models and datasets. The smallest value of in these plots is the Cinit chosen by the sampling procedure mentioned earlier. Note that large portion of the samples find their corresponding adversarial examples within the first few iterations.
be run in parallel on all the samples
UNLEARN ACC
RETAIN Acc
TEST Acc
FT AUC
AVG. GAP
RETRAIN
55.93 20.21
99.98 20.00
55.96 20.74
50.06 20.36
0.00
RL
1.36 10.21
1.74
10.34
1.27
1:0188
51.17
10.42
52.15
20.07
GA
0.44 #0.05
0.51
#0.01
0.50
+0.00
50.55
40.97
52.81
#0.14
BS
0.93 20.24
0.97
20.22
1.02
+0.27
49.96
20.20
52.28
20.19
SALUN
2.46 #1.51
2.89
+1.88
2.13
+0.98
52.20
40.94
51.63
#0.82
AMUN
58.54
11.58
62.34
11.40
43.22
10.68
63.05
10.89
16.50
20.13
AMUN++901U..
62.37
#0.01
68.47
40.70
45.44
40072
61.24 40063
14.91
+0.37
Table Unlearning without access to DR. Comparing different unlearning methods in unlearning 10% of Tiny Imagenet Dataset (D) from VGG19 models. Avg. Gap used for evaluation (lower is better). The lowest value is shown bold while the second best is specified with underscore. As the results show, AMUN outperforms all other methods by achieving lowest Avg. Gap and AMUN SalUn, achieves comparable results.
16

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
022:
0.70
0.151
0.3
0.15
02:
02:
0.10
0.653
0.5
642
0.00
0.02
0.02
Confeence
Confdence
Figure 5: The two left-most subplots show the confidence values before and after unlearning (using AMUN of 10% of the training samples. The two right-most subplots show these confidence values for unlearning 50% of the training samples. In both cases, the confidence values of samples in Dp are similar to those of DD and their fitted Gaussian distribution matches as expected. After using AMUN for unlearning the samples Dr, the confidence values on this set gets more similar to the test (unseen) samples.
F. Ablation Study (cont.)
In this section, we further discuss the ablation studies that were mentioned $ 6.2. We also present other ablation studies on using transferred adversarial examples (Appendix F.4) and weaker adversarial attacks (Appendix E.5) in AMUN.
F.1 Empirical Behavior of Retrained Models
As discussed in 3.1, assuming the Dr and DD come from the same distributions, we expect the prediction confidences of the models retrained DR to be similar DF and Dr, because both of these sets considered unseen samples that belong the the same data distribution. Figure (left) shows the confidence scores for a ResNet-18 18 model that has been retrained on D - Dr. where D is the training set of CIFAR-I0 and the size of Dp (randomly chosen from D) is 10% and 50% of the size of 'D for the left and right sub-figures, respectively. To derive the confidence values, we use the following scaling on the logit values:
(([(x),) log (1 f(z)y T- f(x)g
where f(r), is the predicted probability for the correct class. This scaling has been used by Carlini et al. (2022) to transform the the prediction probabilities such that they can be better approximated with a normal distribution, which are indeed used by some of the SOTA MIA methods for predicting training samples from the test samples (Carlini et al., 2022). Figure (left) shows these fitted normal distribution as well, which perfectly match for Dr. and Dr.
F.1.I. CONFIDENCE VALUES IN UNLEARNED MODELS
In this section, we investigate the confidence values of the model, before and after using AMUN for unlearning subset of 10% or 25% of the training samples. For the original model (before unlearning), we expect the distribution of confidence values of samples in Dp to be similar to those of the samples in DR because they were both used as the training data and the model has used them similarly during training. However, this distribution is different for the test samples (Dr), as the model has not seen them during the training phase. After unlearning, as discussed in section 3, we expect the distribution of confidence values for Dr and Dr to become more similar so that MIAs cannot distinguish them from each other. As Figure shows, for both unlearning 10% (two leftmost subplots) and 50% (two rightmost subplots), we observe the same behavior. Fur the original models (1st and 3rd subplot), the distribution for Dp and DR mathces exactly, but after using AMUN (2nd and 4th subplot) the distribution for Dr shifts toward that of DD.
F.2. Adversarially Robust Models (cont.)
As discussed in $ 6.2.2, also evaluatee the effectiveness of AMUN when the trained model is adversarially robust. For this experiment, we used the ResNet-18 models with 1-Lipschitz convolutional and fully-connected layers, which are shown to be significantly more robust than the original ResNet-18 18 models. In Table 3, we showed the results for unlearning 10% and 50% of the samples from the robust ResNet-13 18 models trained on CIFAR- 10, in the case where DR is not accessible. In Table 6, we showed the corresponding results when the unlearning methods have access to DR. As the results show, similar to the results discussed in $ 6.2.2, A AMUN effectively unlearns Dp for either of the sizes of the this set.
17

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
RANDOM FORGET (10%)
RANDOM FORGET (50%)
FT AUC
FR AUC
TEST Acc
FT AUC
FR AUC
TEST
RETRAIN
49.95 #0024
54.08 #0.16
89.01 +0.21
50.19 =0.15
55.61 +0.05
85.76 +0.41
AMUN
49.12 #0.19
53.60 #0.31
86.94 +0.50
49.41 =0.25
54.22 +0.16
87.38 +0.30
Table 6: Unlearning on adversarially robust models. Evaluating the effectiveness of AMUN in unlearning 10% and 50% of the training samples when the models adversarially robust and we have access to Dg. For this experiment use models with controlled Lipschitz constant which makes them provably and empirically more robust to adversarial examples.
We also evaluated AMUN for unlearming in models that are adversarially trained. we performed our analysis on ResNet- 18 models trained using TRADES loss (Zhang et al., 2019) on CIFAR-10. We performed the experiments for unlearning 10% of the dataset in both cases where DR is accessible and not. As the results in Table 7 show, in both settings AMUN is effective in unlearning the forget samples and achieving low gap with the retrained models. This gap is obviously smaller when there is access DR.
UNLFARN ACC
RETAIN AcC
TEST Acc
FT AUC
AvG. GAP
RETRAIN A AMUN Wish
82.33 #0.39
94.22
=0.21
81.72
#0.36
50.04
+0.34
0.00
82.65
10.62
94.33
20.84
84.99
#0.91
47.18
20.50
1.02 =0.18
AMUN No
81.38
10.10
87.45
10.54
79.74
10.31
54.61
10.23
3.57 =0.24
Table 7: Unlearning with access to DR- Evaluating AMUN when applied to ResNet-18 models trained using adversarial training. TRADES loss is used to train the models, and the unlearning is done on 10% of CIFAR- 10 Dataset (D). Avg. Gap used for evaluation (lower better). The result has been reported in two cases: with and without access to DR. As the results show, AMUN is effective in both cases, with slight degradation in the more difficult setting of access to DR.
F.3. Fine-tuning on Adversarial Examples (cont.)
As explained in $ 6.2.1, we evaluate the effect of fine-tuning on test accuracy of a ResNet-18 18 model that is trained on CIFAR-10, when DA is substituted with other datasets that vary in the choice of samples or their labels (see $ 6.2.1 for details). In Figure 1 we presented the results when DF contains 10% of the samples in D. We also present the results for the case where DF contains 50% of the samples in D in Figure As the figure shows, even for the case where we fine-tune the trained models on only DA which contains the adversarial examples corresponding to 50% of the samples in D (right-most sub-figure) there is no significant loss in models' accuracy. This is due the fact that the samples in DA, in contrast the other constructed datasets, belong to the natural distribution learned by the trained model. To generate the results in both Figures I and 6, we fine-tuned the trained ResNet-16 18 models on the all the datasets (see $ 6.2.1 for details) for 20 epochs. We used a learning rate of 0.01 with a scheduler that scales the learning rate by 0.1 every 5 epochs.
We also perform the same experiment for VGG19 (Simonyan & Zisserman, 2014) models trained on Tiny Imagenet dataset (Le & Yang, 2015). We evaluate the effect of fine-tuning on test accuracy of these model that is, when Pp contains 10% of the samples in D and DA is substituted with other datasets that vary in the choice of samples or their labels (see S 6.2.1 for details). In Figure 7 we presented the results, which similarly show that the specific use of adversarial examples with the mis-predicted labels matters in keeping the model's test accuracy.
E.4. Transferred Adversarial Examples
One of the intriguing properties of adversarial attacks is their transferability to other models (Papernot et al., 2016; Liu et al., 2016); Adversarial examples generated on a trained model (source model) mostly transfer successfully to other models (target models). This success rate of the transferred adversarial examples increases if the source model and target model have the same architecture (Papernot et al., 2016). There are other studies that can be used to increase the success rate of this type of attack (Zhao et al., 2021; Zhang et al., 2022; Chen et al., 2023a; Ebrahinpour-Boroon this section, we are interested to see if using the the adversarial examples generated using Algorithm I for a given model
et al., 2024). In
18

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
80-
80
80
Adv
Test Accuracy racy
60
60
Orig
Adv-RS
40
40
40
Adv-LL
Orig-RL
20
20
20
Orig-AdvL
0
0
5
10
15
20
0
10 Epochs
15
20
0
5
10 Epochs
15
20
Epochs
Figure 6: This figure shows the effect of fine-tuning on test accuracy a ResNet-18 model that trained on CIFAR-10, when the dataset for fine-tuning changes (see 6.2 for details). Let Dp contain 50% of the samples in and DA be the set of adversarial examples constructed using Algorithm 1. Adv, from the left sub-figure right one, shows the results when DU DA. Dr U DA and DA is used for fine-tuning the model, respectively. Orig. Adv-RS, Adv-RI, RL, Crig RL, and ori Orig-AdvI. shows the results when D for each of these sub-figures is replace by Drr DARS, DARL. Dns. and AddLL accordingly. As the figure shows, the specific use of adversarial examples with the mis-predicted labels matters in keeping the model's test accuracy because DA, in contrast to the other constructed datasets belong to the natural distribution learned by the trained model.
Test Accuracy
50
40
40
Orig
Adv RS
40
30
AdvRL
Orig-RL
20
30
Orig-Advl.
0-
0
5
10
15
20
0
10
15
20
0
5
10
15
20
Epochs
Epochs
Epochs
Figure 7: Effect of fine-tuning on adversarial examples. This figure shows the effect of fine-tuning on test accuracy of a VGG19 model that is trained on the Tiny ImagenNet dataset, when the dataset for fine-tuning changes for details). Let Dr contain 10% of the samples in and DA be the set of adversarial examples constructed using Algorithm 1. Adv, from the left sub-figure to right one, shows the results when D DAD Dr DA. and DA is used for fine-tuning the model, respectively. Orig, Adv-RS, RS, Adv RL, Orig RL, and Crig-Advi AdvL shows the results when DA for each of these sub-figures replace by Dr, DARS. DA RL. DRL. and D Adet. accordingly. As the figure shows, the specific mis- mis-predieted labels matters in keeping the model's 's test accuracy because DA, in contrast to the other constructed datasets belong to the natural distribution learned by the trained model.
of adversarial examples with the
trained on some dataset D can be used as the DA dataset for unlearning a portion of 'D from a separately trained model. The advantage using adversarial examples generated for another model saving the computation cost for other trained models. For this purpose, we train three ResNet-18 models separately on CIFAR-10. 10, we generate the adversarial examples for each of these models using Algorithm We use AMUN for unlearning 10% and 50% of CIFAR- 10 from either of these models, but instead of their adversarial samples, use the ones derived from the other models. The results in Table 8 shows that using transferred adversarial examples leads to lower performance, specially for the case where there is no access to DR. All the values for test accuracy are also lower compared to using adversarial examples from the model itself because these adversarial examples from the other models do not all belong to the natural distribution of the model and they do not even always transfer to the other models. Still the results are comparable to the prior SOTA methods in unlearning, and even in the case of no access to DR outperforms all prior methods.
19

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
WITH ACCESS TO Pa
No ACCESS TO Pa
RANDOM FORGET (10%)
RANDOM FORGET (30%)
RANDOM FORGEI (10%)
RANDON FORGET (30%)
ADC
FT ADC
AVC.
TOST
AT
GAP
AUC
042
SELF
03:45 20.30
50.18
20.36
0.62
22.0
92 33 00
40:99 10
0.33
2001
67
2001
24 2033
1.94
2012
60.43
20.10
52.60 20:3
2.51 2010
OTHERS
02.64
20.00
46,70
20.50
1.57 20.22
91.40
20,00
47.38
10.03
1.15 29.21
90.56
20.29
48.29 19.22
3.07
19.3
83.61
20:5
51.11
2006
6,70 2033
Table 8: Transferred adversarial examples. Comparing the effectiveness of unlearning when instead of using adversarial examples of the model, we use adversarial examples generated using Algorithm on separately trained models with the same architecture. As the results show, relying on transferred adversarial examples in AMUN leads to worse results, specially for test accuracy because the adversarial examples do not necessary belong to the natural distribution learned by the model. However, even by using these transferred adversarial examples AMUN outperforms prior SOTA unlearning methods, specially when there is no access to DR.
70-
col
50
Amun
60-
Amun-
65
60
56
SalUn
55-
FT ATii
50
60
55
RF
54-
45
55
50
52-
40
50
50-
2
3 4
2
3
2 3
5
2
Request counts
Request counts
Request counts
Request counts
Figure 8: This figure shows both FT AUC and RF AUC components of the plots presented in Figure 2. The two left-most sub-figures show these values along the number of unlearning requests for the case where there is access to Dg and the two right-most ones show these values when there is no access Pr.
F5.. Weak Attacks
In this section we evaluate the effectiveness of using weaker attacks in Algorithm 1. For this purpose, we perform the unlearning on R8 model trained on CIFAR-IO in all unlearning settings mentioned in $ 5, and compare the results with the default choice of PGD-50 in AMUN. The weaker attack that we use is a variation of FFGSM (Wong et al., 2020), which itself is a variant of FGSM (Goodfellow et al., 2014). FGSM takes steps toward the gradient sign at a given sample to find adversarial samples. FFGSM takes a small step toward a random direction first, and then proceeds with FGSM. To adapt these method to the format of Algorithm 1 we start with FGSM attack; we find the gradient sign and start to move toward that direction in steps of size until we find an adversarial example. If the adversarial example is not found after few iteration of the Wh i ile loop, we restart the value of c and add small random perturbation before the next round of FGSM attack and the While loop. We continue this procedure to find an adversarial sample. After deriving a new set of adversarial examples using this methods, we performed a separate round the new attack to have a fair comparison. It is notable to mention that this leads to a much faster attack because we only compute the gradient once for each round round of FGSM (at the beginning or after each addition of random perturbation and restarting FGSM). Table 9 shows the comparison of the results with the original version of AMUN that uses PGD-50. As the results show, using this weaker attack leads to worse results; however, they still outperform prior SOTA methods in unlearning, specially in the setting where there is no access to DR and the size of Dp is 50% D.
tuning for unlearning with
For each image in CIFAR-10 10. Figure 9 shows 8² (see Definition 2.1) for the adversarial examples that Algorithm finds using PGD-50 (x-axis and FFGSM (y-axis). The dashed line shows the x = y line for the reference. As the figure shows 5, much smaller for PGD-50. This value is smaller for FFGSM for less than 4% of the images, but still even for those images, the value of 6e for PGD-50 is very small, compared to the range of values that are required for FFGSM in many cases. This, we believe, is the main reason behind worse performance when using FFGSM. However, still note that the adversarial examples that are found using FFGSM belong to the natural distribution of the trained model and therefore fune-tuning the model on these samples does not lead to noticable deterioration of the test accuracy, while achieving reasonable FT AUC score. Indeed this larger distance of the adversarial examples with the original samples in Dp, leads to better performance of AMUN when it does not include Dp when fine-tuning the model, because the difference in the predicted logits compared to the 5 leads to under- estimationn of the local Lipschitz constant and therefore, the model is able to fit perfectly to both the original samples and its corresponding adversarial sample without changing much. This consequently leads to a larger value
20

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
of FT AUC score.
WITH ACUESS TO Da
No ACCESS TO D2
RANDOM FORGET (10%)
RANDOM FORGET (50%)
RANDOM FORGET (10%)
RANDOM FORGET (50%)
TasT ACC
PLAUC
Arg, GAI
ALC
ALC
Avo. GAP
Tust ACC
ALC
ANG. GAP
TEST
PT AUC
AvG GAP
PGD-50
93.45
50.18
0.02
92:39
49.59
0.33
91.67
52.24
1.94
89. 20.19
52.60
2,51
FGSM
93.87 2010
50.64 AAM
0.92 20.36
89.41
21:01
50.93
20.43
1.81 29.77
92.14
20.25
56.58
21.03
3.46 20.3
90.12 2038
54.54 20
3.29 20:10
Table 9: Using weaker attacks. Comparing the effectiveness of unlearming when PGD- 10 in Algorithm 1 is replaced with a variant of FGSM attack, which is considered to be significantly weaker and leads to finding adversarial examples at a much higher distance to the original samples. We evaluate unlearning 10% and 50% of the training samples in CIFAR- 10 from a trained ResNet- 18 model. As the results show, in both settings of unlearning (with access to Dy and no access to DR). using the weaker attack does not perform as well as the original method. However, it still outperforms prior SOTA unlearning methods.
G. Comparison Using Prior Evaluation Methods
In this section we perform similar comparisons to what we presented in section 6.1, but instead of FT AUC, we use the same MIA used by prior SOTA methods in unlearning for evaluations. As mentioned in section 5.2, we refer to the score derived by this MIA as MIS.
RANDOM FORGET (10%)
RANDOM FORGET (50%)
UNLEARN Ase
RETAIS ALt
TEST ACC
M15
AVU. GAP
UNLEARN Aoc
RESTAIN Ace
TEst Aec
MIS
Avd, GAr
RETRAIN
94.49 20.30
100.0 00.00
94.33 2:0
12.53
16
0.00
92.09 -0.37
100.0 00
91.85 20.35
16.78 57
0.00
95.16 #0.20
96.64
92.21 20.27
11.33
#0.36
1.81
=0.10
94.24
95. 22.22-2 +0.31
91.21
12.10 #0.72
3.00 #0.24
99.22 10.t0
00.99
100.0
94.10
10.11
10:94
10:45
1.64
_0.30
92.98
11.07
94.83
11:00
89.19
1..74
12.48 10.00
3.29 10.00
98.94 11.38
00.22
11:31
93.39
11-18
4.21
15:05
3.62
_1.01
90.94
10.00
99.95
10:00
94.36
10:31
0.62
10.30
6.64 10:15
BS
99.14 +0.31
99.89
+0.05
93.01
20.14
5.50
+0.39
3.27
=0.15
100.00
+0.00
100.00
+0.00
91.62
+0.08
0.40
=0.05
6.77 +0.05
4-SPARSE
04.29 10:34
95.63
10.16
91.55
20.17
12:03
11:99
2.26
20.95
92.63
16. 15
95.02
10:10
89.56
10
12.03
20.39
3.14 20:17
SALUN
99:25 10.12
99.99
20.01
94.11
20.13
11.29
2:050
1.50
=0.00
95.69
10.80
97.26
2:070
91.55
20.50 50
11:27
20.04
3.00 ab.la
AMUN
95.45 +0.19
99.37
+0.00
93.45
20.22
12.55
+0.08
0.59
+0.00
93.50
+0.00
99.71
+0.01
92.39
+0.04
13.58
#0.19
1.37 +0.0r
94.73
+0.07
99.92
20.01
93.95
20.18
14.23
1::40
0.601
=0.30
10.00
99.72
20.02
92.52
10.20 20
13.33
10.10
1.47 10.01
Table 10: Unlearning with access to DR. Comparing different unlearning methods in unlearning 10% and 50% of D. Avg. Gap (see $ 5.2). with MIS as the MIA score, is used for evaluation (lower is better). The lowest value is shown in bold while the second best is specified with underscore. As the results show, AMUN outperforms all other methods by achieving lowest Avg. Gap and AMUN Salun achieves comparable results.
RANDOM FORGRT (10%)
RANDOM FORGET (50%)
UNLEARN Acc
TEST Acc
MIA
USLEARN
MIA
AYc. GAP
RETRAIN
94.49
100.0
94.33
12.53
0.00
92.09
100.0
91.85
16.78 10.37
0.00
100.00 t0.00
100.00 t0.00
91.45 =0.00
3.06 20.03
3.77
10.13
100.00
=0.00
100.00
10.00
94.54
20.13
0.40 +3.03
6.75 20.02
4.77 23:00
5.07
29.64
5.09
32.63
2:5.85
76.58
#7.75
100.00
20.00
100.00
20.00
94.67
20.05
0.35 10:10
6.77 20.04
BS
100.00 00
100.00
4: 00
91.48
20.04
1.11
20.50
4.27
$0.07
100.00
20.00
100.00
20.00
94.59
20.05
0.38
+000
6.76 20.01
SALUN
100.00 +0.00
100.00
+0:00 00
91.17
=0.10
2.39
20.04
3.95
+0.14
100.00
=9.00
100.00
=0.00
91.57
#0.12
0.33 +0.04
6.77 #0.25
AMUN
94.28 10.37
97.47
10.10
91.67
.0.01
11.61
10.00
1.61
10.00
92.77
10:52
95.66
15:25
89.43
10.19
14.13
10447
2.52 10:36
AMUN. Pslun.
94.19
10.35
97.71
10.05
91.79
20.13
11.66
10.16
1.51
16.02
91.00
10
96.50
10:31
89.98
10.44
13:07
10:60
2.35 10.15
Table 11: Unlearning with access to only Dy. Comparing different unlcarning methods in unlearning 10% and 50% of D. Avg. Gap (see $ 5.2) is used for evaluation (lower is better) when only DD is available during unlearning. As the results show, AMUN Saltin significantly outperforms all other methods, and AMUN achieves comparable results.
H. Continuous Unlearning (cont.)
In $ 6.3, we showed AMUN, whether with adaptive computation of DA or using the pre-computed ones, outperforms other unlearning methods when handling multiple unlearning requests. Another important observation on the presented results
21

---

Not All Wrong is Bad: Using Adversarial Examples for Unlearning
in Figure 2 is that the effectiveness of unlearning decreases with the number of unlearning requests. For the setting with access to DR. this decrease is due the fact that the Dr at each step has been part of DR at the previous steps; the model has been fine-tuned on this data in all the previous steps which has led to further improving confidence of the modes on predicting those samples. This result also matches the theoretical and experimental results in differential privacy literature as well (Dwork, 2006; Abadi et al., 2016).
This problem does not exist for the setting where there is no access to DR. but we still see a decrease in the unlearning effectiveness as we iecraase the number of unlearning requests. The reason behind this deterioration is that the model itself is becoming weaker. As Figure 8 shows, the accuracy on the model on both PR and DT gets worse as it proceeds with the unlearning request, this is because each unlearning step shows the model only 2% (1K) of the samples and their corresponding adversarial examples for fine-tuning. So this deterioration is expected after few unlearning requests. So when using AMUN in this setting (no access to DR) in practice, it would be better to decrease the number of times that the unlearning request is performed, for example by performing a aizy-unlearning (waiting for certain number of requests to accumulate) or at least using a sub-sample of DR if that is an option.
15
6, (FFGSM)
10
0.0
0.5
1.0
6, (FGD-50)
Figure 9: For each image in CIFAR-IO the æ-axis shows the Euclidean distance of the corresponding adversarial example that is found by using PGD-50 in Algorithm 1. y-axis shows this distance for the adversarial examples found by the variant of FFGSM in Algorithm 1. The dashed line shows the x line. As the figure shows, the distance is much larger for weaker attacks and this leads to worse performance of AMUN, as explained by Theorem 4.1.
22


Reviews:
[{"review_id": "cbpiHFVNkj", "reviewer": "Reviewer_fXrz", "summary": "This article introduces AMUN, an unlearning method that uses adversarial examples to remove the influence of specific training samples from a trained model while preserving overall model accuracy. The key insight behind AMUN is that fine-tuning a model on adversarially modified versions of the forget set (DF) enables effective unlearning without requiring full retraining.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and insightful comments. We are excited about the reviewer’s acknowledgment of the uniqueness of our approach for unlearning. Below are our responses to their questions:\n\n**Q1+S1+W1. Theoretical guarantees:** Although most prior SOTA methods in approximate unlearning are not accompanied by theoretical guarantees, we proved a theorem (https://shorturl.at/ChU0s) that guarantees moving toward the retrained models by making assumptions that are common in the certified unlearning literature. Please refer to the discussion with reviewer qDdb for more details.\n\n**Q2+S3+W3. Larger model/dataset:** we have performed our experiments on VGG19 models (12 times larger than ResNet-18) trained on the Tiny ImageNet dataset (200 classes). We evaluated our prior observations similar to Figure 1 in our manuscript that shows fine-tuning the trained models on their adversarial examples does not lead to catastrophic forgetting (https://tinyurl.com/5n6f6pxr). We also compared the unlearning methods and created the tables (https://tinyurl.com/2wwwbbfp) corresponding to Tables 1 and 2 in our manuscript. Please see the response to reviewer qDdb for more details.\n\n**Q4. Certified baseline:** We performed a comparison to a SOTA certified unlearning method (Zhang et al ICML 24). This method does not work when there is no access to DR. Our results (https://shorturl.at/Q19RQ) show that certified unlearning methods such as this, though accompanied with theoretical guarantees, are not capable of outperforming SOTA in approximate unlearning, including AMUN. We believe that this is the case due to their assumptions not holding for deep learning models used in practice.\n\n**Q3. Stronger attacks:** We wish to clarify that the objective of our algorithm is to use adversarial attacks to find \"neighbors\" for those samples in the unlearning set, and use this neighbor set for fine-tuning to ensure that the resulting (unlearned) model lacks confidence on these samples. We are unsure of what the reviewer means to consider attacks to bypass adversarial fine-tuning, as the objective of the attack we consider is to \"augment\" the sample (to be unlearned) in a manner so as to ensure that it's confidence is low post fine-tuning.\n\n**Q5. Forget set:** Upon receiving an unlearning request, the set of samples to be forgotten are specified. This is the common setting used in prior works in unlearning for classification models, such as SulUn and l1-sparse.\n\n**Q6. Table 1:** The setting of Table 1 (access to DR) is much easier than Table 2. Still, we want to point out the fact that even in that scenario, our method achieves the smallest Avg Gap (based on the average over 27 runs). Moreover, once we make things more difficult, by either moving to a larger model and dataset, or revoking access to DR, the advantage of our method over prior works becomes apparent. The capability of performing unlearning without access to DR is much more desirable as it will be more practical in many real-world use-cases. \n\n**Q7+S2+W2. Computational cost:** The time comparison: https://shorturl.at/AXVB7 . Note that, we only need to run Algorithm 1 on the samples that are requested to be forgotten. For our experiments, we choose a small sub-sample of the corresponding dataset and evaluate their final values of $\\epsilon$; based on these values, we set the initial value of $\\epsilon$ and run PGD attack on the samples. Then we only keep the samples for which an adversarial example is not found, and run another round of PGD with the updated $\\epsilon$ value. We proceed until all the samples find one corresponding adversarial example. These histograms (https://shorturl.at/fMQDC) show the number of samples for each $\\epsilon$ value. Based on our analysis, for both CIFAR10 and tiny imagenet, finding the adversarial example for all the samples is equivalent to less than 3 runs of PGD50 and PGD20, respectively.\n\n**W4+S4. More recent SOTA:** SalUn is a SOTA method in approximate unlearning published in ICLR 24. In our experiments we added all the baseline methods from that work. In addition, we also performed experiments using the SOTA in certified unlearning (Zhang et al ICML 24), which has made our baselines comprehensive and up-to-date.\n\n**W5+S5. Limitations:** We have tried various settings, such as Lipschitz-bounded models and adversarially-trained models to ensure our approach is not limited to only regular training paradigms, but there is no guarantee on compatibility with all training paradigms. Also, as with every new method, the introduction of this new methodology for unlearning might invite a certain line of attacks specifically targeted for this approach. Also future approaches are required to avoid the slight degradation in adaptive setting.\n\nWe hope you find our responses satisfactory, and consider raising your score towards acceptance. We are happy to engage further, and thank you again for your valuable suggestions in improving our paper!", "claims_and_evidence": "The authors claim that AMUN Effectively Unlearns Data Without Significant Accuracy Drop\nThis is supported by: Tables 1 & 2 (AMUN achieves lower Avg. Gap than baselines). Still no comparison of computational efficiency.\n\nClaim: AMUN Works Even Without Access to Retained Data (DR)\n\nThis is supported by: Table 2 (AMUN+SalUn achieves lowest Avg. Gap). But no experiments on larger datasets or different data types is performed.\n\nClaim: AMUN Works for Adversarially Robust Models\n\nThis statement is supported by: Table 3 (AMUN achieves similar performance to retraining). But no analysis is shown on how robustness constraints are affecting the unlearning process.\n\nClaim: AMUN Supports Continuous Unlearning\n\nThis is supported by: Figure 2 (AMUN-A outperforms other methods across multiple steps). But no discussion on computational cost of sequential unlearning is given by the authors.\n\nProblematic or Unclear Claims\n\n1. Computational efficiency: Claimed but not directly compared to baselines.\n2. Scalability: No experiments on large models/datasets. \n3. Generalization to other architectures: Only tested on ResNet-18 (CIFAR-10).", "methods_and_evaluation_criteria": "1. Proposed Method (AMUN) uses Adversarial Examples for unlearning few samples.\n\n2. Effectiveness Measured by Key Metrics\n\nTest Accuracy (Test Acc) – Ensures model retains performance.\nForget AUC (FT AUC) – Measures how well forgotten data is removed.\nAvg. Gap – Lower values indicate better unlearning.\n\n3. Comprehensive Baseline Comparisons\n\nEvaluated against Retrain, FT, RL, GA, BS, SALUN, AMUN+SalUn.\n\n4. Tested on CIFAR-10 Dataset but authors should try experimenting with CIFAR-100 and other segmentation datasets as well.\n\n5. Authors should try evaluation on deep architectures like ViTs or large-scale models.", "theoretical_claims": "1. This article lacks a formal proof of complete data removal.\n\n2. No proof linking AMUN directly to better generalization. I suspect this is of utmost importance and the authors should consider showing this proof. This is an intuitive claim, but no theoretical analysis of effectiveness.\n\n3. Observation 1 is very trivial.", "experimental_designs_or_analyses": "1. The article benchmarks AMUN against various unlearning baselines (e.g., Retrain, FT, RL, GA, BS, SALUN).\nThe results show AMUN’s advantage, statistical significance tests (e.g., hypothesis testing) are not reported, making it unclear if improvements are meaningful beyond noise.\n\n\n2. The article evaluates performance in different settings (random forget 10% & 50%, adversarially robust models, continuous unlearning).\nThe continuous unlearning scenario (AMUN-A) is promising but lacks a long-term stability analysis—effects of multiple iterations on overall model performance are not well examined.", "relation_to_broader_scientific_literature": "This article builds on prior works in machine unlearning, particularly efficient unlearning methods that avoid full retraining.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n1. Authors introduce adversarial fine-tuning as an effective unlearning method. This is unique perspective in this field.\n2. AMUN has an edge over existing unlearning methods in Avg. Gap and MIA resistance across various settings.\n3. Avoids costly full retraining while maintaining competitive accuracy.\n4. Demonstrates effectiveness in iterative unlearning.\n\nWeaknesses:\n1. The paper lacks rigorous theoretical guarantees for why adversarial fine-tuning effectively unlearns data.\n2. Performance relies on the quality of adversarial examples, which can be computationally expensive and can go over several iterations. How does this guarantee that an adversarial example of a given sample will really unlearn? Have the authors conducted an analysis on gradient based maps or loss landscapes?\n3. Results are primarily shown for CIFAR-10; effectiveness on larger, more complex datasets is unclear.\n4. While various unlearning methods are tested, more recent and advanced SOTA techniques could have been included.\n5. Authors should discuss about the limitations of this method.\n6. A good block diagram is missing.\n7. Some tables are too small to be seen properly. I can understand the space issues but a good adjustment of important tables should be done (eg. Table 3)\n8. 5.2. Evaluation Metrics para can be shortened to save space.", "other_comments_or_suggestions": "My Suggestions:\n\n1. The paper lacks rigorous theoretical guarantees for why adversarial fine-tuning effectively unlearns data.\n2. Performance relies on the quality of adversarial examples, which can be computationally expensive and can go over several iterations. How does this guarantee that an adversarial example of a given sample will really unlearn? Have the authors conducted an analysis on gradient based maps or loss landscapes?\n3. Results are primarily shown for CIFAR-10; effectiveness on larger, more complex datasets is unclear.\n4. While various unlearning methods are tested, more recent and advanced SOTA techniques could have been included.\n5. Authors should discuss about the limitations of this method.\n6. A good block diagram is missing.\n7. Some tables are too small to be seen properly. I can understand the space issues but a good adjustment of important tables should be done (eg. Table 3)\n8. 5.2. Evaluation Metrics para can be shortened to save space.", "questions_for_authors": "Concerns:\n\n1. Your method relies on adversarial fine-tuning to approximate unlearning. Do you have any theoretical guarantees that AMUN effectively removes information from the model rather than just masking it? If not, how do you ensure true unlearning?\n\n2. Given the computational cost of generating adversarial examples, how does AMUN scale to large-scale datasets like ImageNet? Have you tested its efficiency and effectiveness in such settings?\n\n3. The results suggest AMUN reduces MIAs, have you tested it against stronger, adaptive attacks designed specifically to bypass adversarial fine-tuning? If so, how does it perform compared to retraining?\n\n4. Your comparisons include various unlearning techniques, but how does AMUN perform against more recent methods such as certified removal techniques or data augmentation-based approaches? Would AMUN still hold its advantage in Avg. Gap and AUC metrics? \n\n5. How will the users determine the forget set?\n\n6. The results in the table 1 show incremental pattern and the proposed method does not show any significant accuracy change.\n\n7. Performance relies on the quality of adversarial examples, which can be computationally expensive and can go over several iterations. How does this guarantee that an adversarial example of a given sample will really unlearn? Have the authors conducted an analysis on gradient based maps or loss landscapes?\n\nAuthors can discuss about some of the above points rather than experimenting further.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No"}}, {"review_id": "RdSaqUh0Dq", "reviewer": "Reviewer_dSmc", "summary": "The paper proposes Adversarial Machine UNlearning (AMUN), a novel method for efficient machine unlearning in classification models. The core idea is to leverage adversarial examples corresponding to the forget set to fine-tune the model, thereby reducing its confidence on $D_F$ while preserving test accuracy.  By fine-tuning on adversarial examples of the forget set (with incorrect labels), AMUN avoids global model degradation. Moreover, AMUN mimics the behavior of models retrained from scratch on the retain set to achieve comparable unlearn/retain/test accuracy and resistance to membership inference attacks (MIAs). The method also be generalized to adversarially robust models and handles continuous unlearning scenarios effectively.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We are excited about the reviewer’s acknowledgment of the novel connections our work makes with adversarial training and Lipschitz-constrained models, in addition to a more rigorous analysis by leveraging RMIA rather than MIA. Below are our responses to their questions and concerns:\n\n**Q1 + W1. Additional experiments:** We have performed our experiments on VGG19 models (12 times larger than ResNet-18) trained on the Tiny ImageNet dataset (200 classes). We evaluated our prior observations similar to Figure 1 in our manuscript that shows fine-tuning the trained models on their adversarial examples does not lead to catastrophic forgetting (https://tinyurl.com/5n6f6pxr). We also compared the unlearning methods and created tables (https://tinyurl.com/2wwwbbfp) corresponding to Tables 1 and 2 in our manuscript. We also performed a new experiment on the effectiveness of unlearning methods on the models trained with adversarial training  (https://tinyurl.com/43bxcafb). Please see the response to reviewer qDdb for details.\n\n**W2. Theoretical guarantees:** Although most prior SOTA methods in approximate unlearning are not accompanied by theoretical guarantees, we proved a theorem (https://shorturl.at/ChU0s) that derives an upper-bound on the difference between the retrained models and the models unlearned using AMUN by making assumptions that are common in the certified unlearning literature. The proved theorem justifies our earlier intuitions about the need for good quality adversarial examples that are as close as possible to the original samples. For more discussions on the proved theorem, please refer to the discussion with reviewer qDdb.\n\n**Missing related work:** The work of Chen et al. is included among our baseline methods in all the experiments (BS for Boundary Shrink). We also have a thorough discussion of the differences in our approach in Appendix A.\n\n**Q2. Combination with DP:** We do not see any obvious reason why AMUN can not be combined with approaches like DP. However, AMUN exploits the robustness of the model, and this is known to have tensions with privacy (as studied in several works such as https://tinyurl.com/3573vmks, https://tinyurl.com/2n6vu3tz, https://tinyurl.com/2cdptafr). We leave a detailed investigation of this to future research.\n\n**Q3. Comparison to certified methods:** While the work of Sekhari et al. is an interesting addition to our discussion of certified unlearning methods in the related works, it is important to note that this method (similar to most other certified methods) makes many assumptions that do not hold in general deep learning models. For example, Assumption 1 in Section 4 states that “for any (input) z, the function f(w, z) is strongly convex, L-Lipschitz and M-Hessian Lipschitz with respect to w” – this is not practical. Instead, we performed a comparison to another SOTA certified unlearning method (Zhang et al ICML 24) with milder assumptions. This method does not work without access to $D_R$. Our results (https://shorturl.at/Q19RQ) show that certified unlearning methods such as this, though accompanied with theoretical guarantees, are not capable of outperforming SOTA in approximate unlearning, including AMUN, due to their assumptions not holding for deep learning models used in practice.\n\n \n**Q4. Choices of attack steps:** For the presented results on ResNet-18 and CIFAR-10, we used PGD-50. For our new results on Tiny Imagenet, we performed the experiments with both PGD-10 and PGD-20 and did not observe noticeable changes.\n\n**Q4. Instability of fine-tuning:** Please note that all the prior methods include fine-tuning steps as part of their procedure. \n\n1. RL fine-tunes on forget samples with a random label and the remaining samples ($D_R$). \n2. Salun does the same fine-tuning as RL, but on a subset of model parameters. \n3. FT and l1-sparse fine-tune the model on $D_R$.\n4. GA fine-tunes on the forget samples in the reverse direction of gradient and fine-tunes on $D_R$.\n5. BS fine-tunes on $D_R$ and an augmented set of samples for forget samples. \n\nIn our experiments, we did not observe more susceptibility to hyper-parameters compared to other methods. We also performed a fair comparison by choosing the hyper-parameters on a separate set of forget-samples and models and performed the experiments on different random seeds. To further investigate the stability of results to the number of epochs, we prepared two plots that show Avg. Gap for various number of epochs (https://shorturl.at/VSLzK). We concluded that AMUN stabilises after a few epochs and the number of epochs could even decrease, but we followed the same number of epochs that were used in prior works for fair comparisons.\n\n\nWe hope you find our responses satisfactory, and consider raising your score towards acceptance. We are happy to engage during the rebuttal period, and thank you again for your valuable comments and suggestions in improving our paper!", "claims_and_evidence": "\"No catastrophic forgetting\": The claim relies on limited datasets (CIFAR-10/ResNet-18); larger-scale experiments (e.g., Tiny-ImageNet or ImageNet) are needed for broader validation. Furthermore, it lacks theoretical guarantees.", "methods_and_evaluation_criteria": "(1) While PGD-50 with $l_2$ norm bounds in Algorithm 1 is logical, adaptive $\\epsilon$ selection lacks theoretical justification. And the reliance on PGD-50 increases computational cost.\n\n(2) The fine-tuning strategy is intuitive but sensitive to hyperparameters (e.g., learning rate, epochs).", "theoretical_claims": "The paper does not provide formal theoretical analysis and guarantees. One of the key assumptions (e.g., adversarial examples belonging to the model’s \"natural distribution\") is empirically validated but lacks theoretical grounding. The relationship between adversarial example strength ($\\epsilon$) and unlearning efficacy seems empirical, not theoretical.", "experimental_designs_or_analyses": "(1) The impact of $\\epsilon_{init}$ in Algorithm 1 and fine-tuning epochs is not well explored.\n\n(2) Limited to CIFAR-10/ResNet-18; no cross-dataset/architecture validation (e.g., ImageNet, ViTs).", "relation_to_broader_scientific_literature": "This work extends approximate unlearning methods by incorporating adversarial examples. The connections to adversarial training and Lipschitz-constrained models shows AMUN’s compatibility with robust models. Additionally, leveraging RMIA rather than MIA leads to more rigorous evaluations.", "essential_references_not_discussed": "Chen, M., et al. \"Boundary Unlearning.\" (CVPR 2023): Machine unlearning is achieved by the shift decision space of the DNN model, which is quite related to adversarial example training and forgetting.", "other_strengths_and_weaknesses": "**Strengths:**\n\nAMUN provides an alternative to machine unlearning with clear explanations of the method and its motivations.\n\n**Weaknesses:**\n\nThe experiments are restricted to CIFAR-10 and ResNet-18 in classification tasks, raising questions about the method's applicability to larger datasets, other architectures and tasks (like Diffusion or LLM Generation). Furthermore, the paper lacks formal theoretical guarantees and thorough explorations on how variations in adversarial attack parameters affect the results.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Following the previouly mentioned Strengths And Weaknesses, the questions are:\n\n(1) Can AMUN maintain its efficacy on larger architectures (e.g., ViTs) or datasets like ImageNet?\n\n(2) Could AMUN be combined with differential privacy or influence functions to provide certified unlearning guarantees?\n\n(3) How does AMUN compare to certified unlearning methods (e.g., Sekhari et al., 2021) in terms of privacy-utility tradeoffs?\n\n(4) How do choices of attack step (PGD-50 vs. PGD-10 or PGD-20), and fine-tuning epochs affect results?  Are there optimal settings for different datasets or architectures?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0AkHavjlF4", "reviewer": "Reviewer_BHZY", "summary": "This paper proposes the Adversarial Machine UNlearning (AMUN) method, which reduces the prediction confidence of the model for the forget samples by fine-tuning the model on adversarial examples, while maintaining the accuracy of the model on test samples. Experimental results demonstrate that AMUN outperforms previous state-of-the-art methods in image classification tasks and performs remarkably well even in the face of membership inference attacks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We are excited about the reviewer’s acknowledgment of our interesting approach toward unlearning. Below are our responses to their questions and concerns:\n\n**W1. Additional experiments:** We have performed new experiments on VGG19 models (12 times larger than ResNet-18) trained on the Tiny ImageNet dataset (200 classes). We evaluated our prior observations similar to Figure 1 in our manuscript that shows fine-tuning the trained models on their adversarial examples does not lead to catastrophic forgetting (https://tinyurl.com/5n6f6pxr). We also compared the unlearning methods and created the tables (https://tinyurl.com/2wwwbbfp) corresponding to Tables 1 and 2 in our manuscript. We also performed a new experiment on the effectiveness of unlearning methods on the models trained with adversarial training, which, in summary (https://tinyurl.com/43bxcafb), shows that our method is even effective for unlearning in models trained with adversarial training. Please see the response to reviewer qDdb for details. \n\nIn addition, as requested by another reviewer, we performed a comparison to a SOTA certified unlearning method (Zhang et al. (ICML 24)). The comparison is done only on the setting where $D_R$ is available because this method does not work when there is no access to $D_R$. Our results (https://shorturl.at/Q19RQ) show that certified unlearning methods such as this, though accompanied with theoretical guarantees, are not capable of outperforming SOTA in approximate unlearning, including AMUN. We believe that this is the case due to their assumptions not holding for deep learning models used in practice.\n\n**W2. Interpreting our results:** Please note that for UNLEARN ACC the goal is to minimize the difference with the corresponding value from the Retrained models. As tables 1&2 show, AMUN achieves the smallest difference in all scenarios. Similarly, for RETAIN acc in table 1 AMUN achieves the smallest difference with the corresponding value for the retrained models. \n\nFor table 2, the reason that some other methods achieve smaller gap for RETAIN acc is the following: when other methods perform poorly in the absence of $D_R$, they choose the smallest available learning rate during the hyper-parameter search which basically allows them to do nothing during the fine-tuning phase. Therefore, they almost always return the same model as the original model, *without performing any unlearning*, and hence achieve an accuracy of 100% on both $D_R$ and $D_F$. But notice that in these cases, the MIA score does not decrease, reiterating that no unlearning has occurred. We will make this point clear in our future revisions.", "claims_and_evidence": "The authors clearly expound the core finding of this paper through two observations and corresponding experiments, that is, fine-tuning the trained models on adversarial examples corresponding to a subset of the training data does not lead to significant deterioration of the model's accuracy. Based on these findings, the authors propose the AMUN method, with clear logical expression and sufficient persuasiveness.", "methods_and_evaluation_criteria": "Overall, th eproposed methods are supported by empirical data to a certain extent. The authors conducted a series of experiments on the CIFAR10 dataset for ResNet18 in the task of image classification.", "theoretical_claims": "The method proposed in this paper is based on a heuristic methodology, and no theoretical claims are provided.", "experimental_designs_or_analyses": "The experimental designs can demonstrate the effectiveness of the proposed method in image classification tasks. However, as the ICML is a top-tier conference in the field of machine learning, it is recommended that the authors supplement investigations on more tasks, such as text classification, to further verify the broad effectiveness of the proposed adversarial-example-based machine learning unlearning technique.", "relation_to_broader_scientific_literature": "N.A.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "Strengths:\n\n1、The proposed AMUN method is innovative. It achieves machine unlearning by fine-tuning the model with adversarial examples, opening up a new way to solve this problem. Different from previous methods, it cleverly exploits the relationship between adversarial examples and the model's decision boundary. It can not only reduce the prediction confidence of forget samples but also avoid drastic changes to the model's global behavior.\n\n2、This paper introduces the proposed AMUN method through two observations, which is highly persuasive.\n\n3、This paper is easy to follow. \n\nWeaknesses:\n\n1、The experimental designs in this paper can demonstrate the effectiveness of the proposed AMUN method in image classification tasks on the CIFAR10 dataset. It is recommended that the authors supplement the experimental results on more backbone networks (such as VGG and ViT) and more datasets (such as ImageNet). Meanwhile, as the ICML is a top-tier conference in the field of machine learning, it is advisable for the authors to conduct additional research on more tasks, such as text classification, to further verify the broad effectiveness of the proposed adversarial-example-based machine learning unlearning technique.  \n\n2、The proposed method does not seem to exhibit obvious superiority in the results presented in Tables 1&2. In particular, it is significantly weaker than the existing methods in terms of the two metrics of UNLEARN ACC and RETAIN ACC.", "other_comments_or_suggestions": "Minor: In Tables 1&2, the expressions of \"SALUN\" and \"Salun\" are inconsistent.", "questions_for_authors": "See Weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "taXlJm0fJ0", "reviewer": "Reviewer_qDdb", "summary": "This paper proposes an algorithm for machine unlearning with an interesting finding.\nThe authors observe that fine-tuning models on adversarial examples closest to the corresponding forget samples can \navoid drastic changes to the global behavior of the model. \n\nExperimental results on CIFAR-10 show promising performance compared to previous methods, like l1-Sparse, and SalUn.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We are excited about the reviewer’s acknowledgment of our interesting approach toward unlearning. Below are our responses to their questions and concerns:\n\n**Theoretical guarantees:** Although most prior SOTA methods in approximate unlearning are not accompanied by theoretical guarantees, we prove a theorem (https://shorturl.at/ChU0s) that derives an upper-bound on the 2-norm of the difference of the parameters of the unlearned model and the retrained model (which are gold-standard for unlearning). To prove this theorem, we make assumptions that are common in the certified unlearning literature. Our derived upper-bound implies enhanced effectiveness of our method when:\n\n1. The distance between the forget sample and its corresponding adversarial example becomes smaller.\n2. The Lipschitz constant of the model becomes smaller.\n3. The quality of the adversarial example becomes stronger (causes a larger loss for the correct label).\n4. The adversarial example transfers better to the retrained model.\n5. The retrained model generalizes better to the (clean) unseen samples.\n\nHence, the proved theorem also justifies our earlier intuitions about the need for good quality adversarial examples that are as close as possible to the original samples (which is the goal of Algorithm 1), and also justifies that by fine-tuning the model on these adversarial examples, we can derive an upper-bound on the distance between the retrained model and the unlearned one. We believe that the presented empirical results, along with the provided theorem, will motivate further theoretical studies in future work.\n\n\n**Larger model and dataset:** we have performed our experiments on VGG19 models (12 times larger than ResNet-18) trained on the Tiny ImageNet dataset (200 classes). We evaluated our prior observations similar to Figure 1 in our manuscript that shows fine-tuning the trained models on their adversarial examples does not lead to catastrophic forgetting (https://tinyurl.com/5n6f6pxr). We also compared the unlearning methods and created the tables (https://tinyurl.com/2wwwbbfp) corresponding to Tables 1 and 2 in our manuscript. \n\nIn addition, as requested by another reviewer, we performed a comparison to a SOTA certified unlearning method (Zhang et al. (ICML 24)). The comparison is done only on the setting where $D_R$ is available because this method does not work when there is no access to $D_R$. Our results (https://shorturl.at/Q19RQ) show that certified unlearning methods such as this, though accompanied with theoretical guarantees, are not capable of outperforming SOTA in approximate unlearning, including AMUN. We believe that this is the case due to their assumptions not holding for deep learning models used in practice.\n\n**Adversarially-trained models:** To evaluate whether our unlearning method works with models trained using adversarial training, we performed our analysis on ResNet-18 models trained using TRADES (as it was more convenient for us to use in the rebuttal period) on CIFAR-10. We performed the experiments for unlearning 10% of the dataset in both cases where $D_R$ is accessible and not. As the results (https://tinyurl.com/43bxcafb) show, in both settings AMUN is effective in unlearning the forget samples and achieving a low gap with the retrained models. This gap is obviously smaller when there is access to $D_R$.\n\nWe hope you find our responses satisfactory, and consider raising your score towards acceptance. We are happy to engage during the rebuttal period, and thank you again for your valuable comments and suggestions in improving our paper!", "claims_and_evidence": "The claim is empirically supported.", "methods_and_evaluation_criteria": "(1) Fine-tuning on adversarial examples can benefit unlearning performance. This is an interesting finding. However, there is no theoretical analysis, and the experiments only focus on small data, like CIFAR-10.          \nIt would be better to validate the generality of the finding with large-scale data, like CIFAR-100 and ImageNet1K.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The ablation is sound.", "relation_to_broader_scientific_literature": "The paper presents an interesting finding that fine-tuning models on adversarial examples benefits the unlearning performance. It is possible to extend the idea into multi-modal models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:        \n(1) The idea of using adversarial examples for unlearning is interesting.             \n(2) The proposed algorithm is simple to implement.\n\nOther Weaknesses:           \n(1) The paper shows that the proposed method can also work well for the adversarially robust models which are trained with controlled Lipschitz constant.          \n      Currently, adversarial training is the most effective method for adversarial robustness. It uses adversarial examples as additional training data while the proposed method finetunes models on adversarial examples.\n      It is interesting to know if the proposed method can work well on these models, like [ref1] and [ref2], regarding adversarial robustness with auto-attack.\n\n[ref1] Decoupled Kullback-Leibler Divergence Loss. NeurIPS 2024.             \n[ref2] Better Diffusion Models Further Improve Adversarial Training. ICML 2023.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The papers proposes a novel unlearning method, AMUN, that achieves unlearning by fine-tuning on adversarial examples (based on the forget set). The method is simple yet effective, and works without access to the retain set, and can handle "iterative" unlearning. The paper is very well written, with a clear motivation behind the method, and theoretical analysis. During the rebuttal, the authors addressed reviewer concerns, and added additional experiments that included evaluation on larger models/datasets, new SOTA methods, and tests on adversarially trained models. The proposed method AMUN seems to perform well when evaluated using standard unlearning metrics, and strong MIAs. In general, as recognized by the reviewers, it seems like a promising approach that others can build on.
Decision: Accept (spotlight poster)