Machine Unlearning Fails to Remove Data Poisoning Attacks
Authors: Martin Pawelczyk
Jimmy Z. Di
Yiwei Lu
Gautam Kamath
Ayush Sekhari
Seth Neel
Venue: ICLR 2025 Poster (2025)
Keywords: machine unlearning
data poisoning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: https://github.com/MartinPawel/OpenUnlearn

Abstract: We revisit the efficacy of several practical methods for approximate machine unlearning developed for large-scale deep learning. In addition to complying with data deletion requests, one often-cited potential application for unlearning methods is to remove the effects of poisoned data. We experimentally demonstrate that, while existing unlearning methods have been demonstrated to be effective in a number of settings, they fail to remove the effects of data poisoning across a variety of types of poisoning attacks (indiscriminate, targeted, and a newly-introduced Gaussian poisoning attack) and models (image classifiers and LLMs); even when granted a relatively large compute budget. In order to precisely characterize unlearning efficacy, we introduce new evaluation metrics for unlearning based on data poisoning. Our results suggest that a broader perspective, including a wider variety of evaluations, are required to avoid a false sense of confidence in machine unlearning procedures for deep learning without provable guarantees. Moreover, while unlearning methods show some signs of being useful to efficiently remove poisoned data without having to retrain, our work suggests that these methods are not yet ``ready for prime time,'' and currently provide limited benefit over retraining.

Published as a conference paper at ICLR 2025
MACHINE UNLEARNING FAILS TO REMOVE DATA POISONING ATTACKS
Martin Pawelczyk'; Jimem
Jimmy Z. Di Yiwei Lu Ayush Sekhari', Gautam Kamath 2.3 Seth Neel Harvard University, University of Waterloo, 'Vector Institute, 'MIT, Google
ABSTRACT
We revisit the efficacy of several practical methods for approximate machine unlearning developed for large-scale deep learning. In addition complying with data deletion requests, one often-cited potential application for unlearning methods is remove the effects of poisoned data. We experimentally demonstrate that, while existing unlearning methods have been demonstrated to be effective in a number of settings, they fail to remove the effects of data poisoning across a variety of types of poisoning attacks (indiscriminate, targeted, and a Gaussian poisoning attack) and models (image classifiers and LLMs); even when granted a relatively large compute budget. In order to precisely characterize unlearning efficacy, we introduce new evaluation metrics for unlcarning based on data poisoning. Our results suggest that broader perspective, including wider variety of evaluations, are required to avoid a false sense of confidence in machine unlearning procedures for deep learning without provable guarantees. Moreover, while unlearning methods show some signs of being useful to efficiently remove poisoned data without having to retrain, our work suggests that these methods are not yet "ready for prime time," and currently provide limited benefit over retraining.
INTRODUCTION
Modern Machine Learning (ML) models are often trained on large-scale datasets, which can include significant amounts of sensitive or personal data. This practice raises privacy concerns as the models can memorize and inadvertently reveal information about individual points in the training set. Consequently, there is an increasing demand for the capability to selectively remove training data from models which have already been trained, functionality which helps comply with various privacy laws, related to and surrounding "the right to be forgotten" (see, e.g., the European Union's s General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA), and Canada's proposed Consumer Privacy Protection Act (CPPA)). This functionality is known as machine unlearning (Cao & Yang, 2015), field of research focused on "removing" specific training data points from trained model upon request. The goal is to produce model that behaves as if the data was never included the training process, effectively erasing direct and indirect traces of the data. Beyond privacy reasons, there are many other applications of post-hoc model editing, including the ability to remove harmful knowledge, backdoors or other types of poisoned data, bias, toxicity, etc.
The simplest way perform unlearning is to retrain the model from scratch, sans the problematic points: this will completely remove their influence from the trained model. However, this is often impractical, due to the large scale of modern ML systems. Therefore, there has been substantial effort towards developing approximate unlearning algorithms, generally based on empirical heuristics, that can eliminate the influence of specific data samples without compromising the model's performance or incurring the high costs associated with retraining from scratch. In addition to the accuracy of the updated models, evaluation metries try measure how much the unlearned points nonetheless affect the resulting model. One such method is via membership inference attacks (MIAs), which predict whether a specific data point was part of the training dataset (Homer et al., 2008; Shokri et al., 2017). Although MIAS provide valuable insights existing unlearning MIAs are computationally expensive to implement themselves (Pawelezyk et al., 2024; Hayes et al., 2024; Kurmanji et al., 2024). Even if
Equal contribution. Corresponding author: martin. mawtii.pawelrryk..o gmmall.coom

---

Published as a conference paper at ICLR 2025
Paisoned Sanpkio
Request to Delcte All Poison Samples
A corrupted ML model is trained by adding poi- soned samples in the training data. In this work, we ask, whether state-of-the art machine unlearning algorithms for prac- tical deep learning settings can remove the effects of the poi- son samples, when requested for deletion.
Figure 1:
Comuutte model
Unlearning Algorithm
Cheen Soerpiiie
Q
Can Unlearning Remove the Effects of Poisons?
Corrupted Training Data
MIA suggests that datapoint has been successfully unlearned, this does not guarantee that residual traces of the data do not remain, potentially allowing adversaries to recover sensitive information.
Data poisoning attacks (Cinà et al., 2023; Goldblum et al., 2022) are natural scenario in which the training data can have surprising and indirect effects on trained models. These attacks involve subtly altering small portion of the training data, which causes the model to behave unpredictably. The field data poisoning attacks has seen tremendous progress over the past few years, and we now have attacks that can be executed efficiently even industrial-scale deep learning models. Given that data poisoning represents scenarios where data can have unforeseen effects on the model, they present an interesting opportunity evaluate the unlearning ability of algorithm, beyond MIAs. When requested to deleted poisoned samples, an ideal unlearning algorithm should update to a model which behaves as the poisoned samples were never included in the training data, thereby fully mitigating the impact of data poisoning attacks. However, is this really the case for current unlearming methods? Can they mitigate the effects of data poisoning attacks? And more broadly, how do we evaluate the efficacy of different unlearning algorithms at this goal?
In this work, we evaluate eight state-of-the-ee unlearning algorithms explored in machine unlearning literature, across standard language and vision tasks, in terms of their ability to mitigate the effects of data poisoning. In particular, we ask whether the unlearning algorithms succeed in reverting the effects of data poisoning attacks from corrupted model when the unlearning algorithm is given all the poison samples as the forget set. Our high-level contributions are as follows:
Failure of current state-of-the-art unlearning algorithms: We stress test machine unlearning using indiscriminate, targeted, backdoor, and Gaussian data poisoning attacks and show that i) none of the current ofate-of-the-art unlearning algorithms can mitigate all of these data poisoning attacks, ii) different data poisoning methods introduce different challenges for unlearning, and iii) the success of an unlearning method depends on the underlying task.
Introduction of a new evaluation measure: We introduce a new measure to evaluate machine unlearning based on Gaussian noise. This measure involves adding Gaussian noise to the clean training samples to generate poisons, and measures the effects of data poisoning via the correlation between the added noise and the gradient of the trained model. This approach be interpreted as novel membership inference attack, is computationally efficient, compatible across all data domains (tabular, image, language) and can be applied any unlearning algorithm. We release the code for our Gaussian data poisoning method at: ht / /ggttub. com/ Saan//artiia inP awel/ OpenUnlearts
We develop and experimentally validate two novel hypotheses explaining why unlearning methods fail under data poisoning attacks.
Insights into Unlearning Failure:
Advocating for detailed unlearning evaluation: By demonstrating that heuristic methods for unleamnin can be misleading, we advocate for proper evaluations provable guarantees for machine unlearning algorithms as the way forward.
2 RELATED WORKS
Machine unlearning. At this point, there exists vast literature machine unlearning (Cao & Yang, 2015), we focus on the most relevant subset here. Many works focus on removing the influence of training on particular subset of points from trained model (Ginart al., 2019; Wu al., 2020;
2

---

Published as a conference paper at ICLR 2025
Golatkar al., 2020a;b; Bourtoule et al., 2021; Izzo et al., 2021; Neel et al., 2021; Sekhari et al., 2021; Jang et al., 2022: Wang et al., 2023). Others instead try to remove a subset of concepts (Ravfogel et al., 2022a;b; Belrose et al., 2023). In general, the goal is to excise said information without having to retrain the entire model from scratch. Some works focus on exactly unlearning (see, e.g., Bourtoule et al. (2021)), whereas others try to only approximately unlearn (e.g., Sekhari al. (2021); Neel et al. (2021)). Much of the work in this line focuses on unlearning in the context of image classifiers (e.g., Golatkar et al. (2020a); Goel al. (2022); Kurmanji et al. (2023); Ravfogel et al. (2022a;b); Belrose et al. (2023); Fan et al. (2023); Chen et al. (2024)). Some approximate unlearning methods are general-purposs using methods like gradient ascent (Neel et al., 2021), or are specialized for individual classes such as linear regression (Guo et al., 2019; Izzo et al., 2021) or kernel methods (Zhang & Zhang, 2021).
Evaluating machine unlearning. Some of the works mentioned above focus on provable machine unlearning (either exact or approximate). That is, as long as the algorithm is carried out faithfully, the resulting model is guaranteed to have unlearned the pertinent points. However, many unlearning methods are heuristic, without provable guarantees. This is why we may want to measure or audit how well an unlearning method performed. Several works (see, e.g., Kurmanji al. (2024); Goel et al. (2022); Golatkar et al. (2020a;b); Graves et al. (2021); Ma et al. (2022); Pawelczyk et al. (2023; 2024); Hayes et al. (2024)) mostly perform various adaptations of membership inference attacks the unlearning setting that either suffer from low statistical power (Kurmanji et al., 2024; Golatkar et al., 2020b; Graves et al., 2021) or require training hundreds of shadow models to evaluate unlearning (Pawelczyk et al., 2024; Kurmanji et al., 2023; Hayes et al., 2024). Relative to these works, our Gaussian poisoning attack has high statistical power at low false positive rates and can be cheaply run using one training run of the model. Sommer et al. (2022) proposed a verification framework for machine unlearning by adding backdoor triggers to the training dataset, however, they do not perform any evaluations for the current ofate-of-the-art machine unlearning algorithms. Goel et al. (2024) asks whether machine unlearning can mitigate the effects of data poisoning when the unlearning algorithm is only given incomplete subset of the poison samples.
Compared to these works, we employ stronger attacks which result in showing that machine unlearn- ing is in fact unable to remove the influence of data poisoning. Our work thus complements these prior works by designing novel clean-label data poisoning methods such as Gaussian data poisoning and extensive evaluation on practically used state-of -tate-of-the-art machine unlearning algorithms
3 MACHINE UNLEARNING EVALUATION PRELIMINARIES
We formalize the machine unlearning setting and introduce relevant notation. Let Syain and Siest be training and test datasets for an ML model, respectively, each consisting of samples of the form z= = (xx,y) where x e Rd denotes the covariate (e.g., images or text sentences) and y E 3 denotes the desired predictions (e.g., labels or text predictions). The unlearner starts with a model f obtained by running a learning algorithm on the training dataset Syann the model Gindal is trained to have small loss over the training dataset, and by proxy, the test dataset well. Given
of deletion requests U J Suainn the unlearner runs an unlearning algorithm to update the initial trained model Gmial to an updated model (updated. with the goal that i) uppdated continues to perform well on the test dataset Sjest- and ii) Qupdated does not have any influence of the delete set U. Our focus in this paper is to evaluate whether given unlearning algorithm satisfies these desiderata.
Model performance on forget set. A trivial proposal for assessing the success of an unlearning algorithm is to examine how the updated model performs on the forgotten points- for instance, by checking the average loss on the forget set. Unfortunately, this measure does not indicate whether the unlearning was effective. Consider these scenarios: i) an entire class is forgotten in CIFAR- 10 model, ii) random clean training samples are forgotten, iii) samples on which the model initially mispredicted are forgotten. In scenario i), the retrained model is expected to fail on the forget set; in ii), it should perform comparably to its original training; in iii), its behavior is unpredictable. This outcome isn't 't unexpected because the objective of unlearning is to erase specific data, and the process does not necessitate maintaining any particular performance on these forgotten samples. Additionally, a model might perform poorly on the forget set or even the test set for various reasons (e.g. adversarial corruptions, etc.), despite potentially retaining some information about the forgotten
3

---

Published as a conference paper at ICLR 2025
data. To circumvent these issues, prior works have proposed to using Membership Inference Attacks (MIAs) to evaluate machine unlearning.
Prior approach for machine unlearning evaluation via membership inference. Machine unlearn- ing is typically evaluated by checking if an instance a a member the training set (MEMBER) by checking if the loss l of the model uudated is lower than equal threshold Tz. (Shokri et al., 2017):
MEMBER
if U(uupaae 2)
MLoss (z)
NON-MEMBER
else.
Under exact unlearning, this attack should have trivial accuracy, achieving true positive rate that equals the false positive rate (i.e., TPR FPR) at every value of TL. The unlearning error is then measured by the extent to which the classifier achieves nontrivial accuracy when deciding whether samples are MEMBER or RON-MEMBEN in particular focusing the tradeoff curve between TPR at FPR at or below 0.01 denoted as TPR&FPR=0.01 (Carlini al., 2022a).
4 DATA POISONING TO VALIDATE MACHINE UNLEARNING
In this section, we describe targeted data poisoning, indiscriminate data poisoning, and Gaussian data poisoning attacks that we will use to evaluate machine unlearning in our experiments. data poisoning attack, an adversary modifies the training data provided to the machine learning model, in such a way that the corrupted training dataset alters the model's behavior test time. To implement data poisoning attacks, the adversary generates corrupted dataset Scor by adding small perturbations to a small by fraction of the samples in the clean training dataset Stain- The adversary first randomly chooses P many data samples Spoison ~ Uniform( (Stain) to be poisoned, where |Spolson| Syann for some poison budget i, 1. Each sample (Ir,) E Spolson is then modified by adding perturbations (xx) e Rd to it, i.e. we modify (r,y) + (x + (rr),y). The remaining dataset Scean
Suain Spoison is left untouched.
4.1 TARGETED AND BACKDOOR DATA POISONING
In targeted data poisoning attack, the adversary's 's goal is to cause the model to misclassify some specific points (Iitareet Mharget) ). from the test set Stest, to some pre-chosen adversarial label Yades- while retaining performance the remaining test dataset Sweet We implement targeted poisoning for both image classification and language sentiment analysis tasks.
For image classification, for target sample (Etargat: tharget), we follow the gradient matching procedure of Geiping et al. (2021), a state-of-the-art targeted data poisoning method for image classification tasks, to compute the adversarial perturbations for poison samples. The effectiveness of targeted data poisoning is measured by whether the model trained on Scor predicted the adversarial label Vaovs on Ttarget instead of Ytarget- For language sentiment analysis, the backdoor data poisoning attack aims to modify the training dataset by adding few extra words per prompt so that Language Model (LM) trained on the corrupted dataset will predict the adversarially chosen label Jadv on some specific target prompts Targett For this attack, we assume that all the prompts Ttarget that the attacker wishes to target feature specific trigger word " "specia a al_token"" e-g., the word "D sney". The attack is generated using the method of Wan et al. (2023) that first filters the training dataset to find all the samples (x, y) Stran for which the prompt æ contains the keyword "special aeeecialltoke toke": these samples constitute the poison samples. For this attack, the model expects the clean prompts to follow this format: æ + "The sent iment is: y". The corrupted dataset Sean is generated by altering the prompts for the poison samples: x + "The sentiment
is: special token". The effectiveness of targeted data poisoning is measured by the fraction of test prompts for which a language model fine-tuned on Scor predicts the adversarial label yaavs on input prompts Fnarget that contain "specia wwwpsiillttoeeeee "o
4.2 INDISCRIMINATE DATA POISONING
In an indiscriminate data poisoning attack, the adversary wishes to generate poison samples such that model trained Scor has significantly low performance the test dataset. We implement this
4

---

Published as a conference paper at ICLR 2025
for image classification. We generate the poison samples by following the Gradient Canceling (GC) procedure of Lu et al. (2023; 2024), a state-of-the-aa indiscriminate poisoning attack in machine learning, where the adversary first finds bad model The adversary computes perturbations A such that Sow has vanishing gradients when trained with the corrupted training dataset, and will thus correspond local minimizer (which gradient-based learning e.g., SGID Adam can converge to). The effectivvness of Indiscriminate Data Poisoning is measured by the performance accuracy on the test dataset for a model trained on the corrupted dataset Scorr.
4.3 GAUSSIAN DATA POISONING
Our Gaussian data poisoning attack is the simplest poisoning method to implement. The adversary hides (visually) undetectable signals in the corrupted training data Soar. which do not influence the model performance on the test data in any significant way but can be later inferred via some computationally simple operations on the trained model. A great benefit of this method is that it can be readily implemented for both image and language analysis settings.
Generating Gaussian poisons. For a given poison budget by and perturbation bound Ep. the adversary first chooses bp) Stain many samples = v) ~ Uniform(Senn (Stain and then generates the poisons by adding an independent Gaussian noise vector to the input æ)). For each 2 Spoison. we generate the poison sample (Ipoison:) 2) by modifying the underlying clean sample (Zbase y) as (,,(0,,2,.))
Tppiison
Trase
E2y
where
(2)
where d is the dimension of the input x. The adversary stores the perturbations added &, corresponding to each poison sample é Spoison- Since the added perturbations are i.i.d. Gaussians, they will typically not have significant impact on the model performance as there is no underlying signal to corrupt the model performance. However, the perturbations & will (indirectly) appear in the gradient updates used in model training, thus leaking into the model parameters and having an effect on the trained model. We expect that a trained model Omna has a non-zero correlation with the added Gaussian perturbation vectors Spanoo
)
Evaluating Gaussian poisons. The effect of data poisoning on a model 0 is measured by the dependence of the model on the added perturbations { Let 9 be model to be evaluated (which may or may not have been corrupted using poisons). In order to evaluate the effect of poison samples on 0, for every poison sample 2 Speison- compute the normalized inner product with 9= v. (Fease. y)), where 9- Rd denotes the gradient of the model
I_ &sle.l/ 0 w.r.t. the input space x when evaluated at the clean base image (Fease. y) corresponding to the poisoned sample z, and define the set Ipoison (1=)2e5scnn)
For an intuition as to why this measures dependence between the model and the added perturbations, consider an alternative scenario and define T₂ (0=.Ee//e..oala where E.- ~ (O, (O,))) is a freshly sampled Gaussian noise vector (thus ensuring that is independent of E2), and let the set Tindop s- Since Ys is independent of E., the values in Tindep would be distributed according to a standard Gaussian random variable //0 1) and thus the average of the values in Tindep will concentrate around O. On the other hand, when 9: is the gradient of model trained on Scor (a dataset corruputed with the noise which we evaluate), we expect that 92 will have some dependence on ε., and thus the samples in Ipoison will not be distributed according to (O, 1).' However, if the unlearning algorithm was perfect, the distribution of Ipolson and Zindep where the dependence is computed with fresh poisons, should be identical.
Consider a routine that samples a point 2 from Ipoisn Indeep computes Lg using the unlcarned model, and then guesses that Ipoison if 1, T. One way to view this metric is as measure of the success of an auditor that seeks to distinguish between poisoned training points that have been subsequently unlearned, and test poison points, using procedure that thresholds based exact unlearning, this approach should have trivial accuracy, achieving TPR = FPR at every value of ?.2 This corresponds to evaluating unlearning via MIAS as presented in Section 3. The difference
I.. Under
'I practice, we observe that the distribution of the samples in Ipuison closely iVllows.N//i (ft 1) for some j > O. The larger the value of i, the more the model depends on the added poisons (see Figure from Appendix for an illustrative example).
"To illustrate, Figure 7 from Appendix B plots full tradeoff curves for the case where we unleamn Gaussian poisons from a Resnet-18 18 trained on the CIFAR-10 dataset using NGD.
5

---

Published as a conference paper at ICLR 2025
between evaluation, and recent work on evaluating unlearning (Pawelezyk et al., 2024; Hayes et al., 2024; Kurmanji et al., 2024), is that prior work evaluates unlearning of arbitrary subsets of the training data. As result, building accurate unlearning evaluation requires sophisticated techniques that involve an expensive process of training hundreds or thousands of so called shadow models, using them estimate distributions on the loss of unlearned points, and then thresholding based likelihood ratio (Pawelczyk et al., 2024). This is in stark contrast to our setting, where because our Gaussian poisons are explicitly designed be easy identify (by thresholding on 1₂) we do not need to train hundreds of models to show unlearning has not occurred one training run is sufficient.
4.4 How TO USE DATA POISONING FOR EVALUATING MACHINE UNLEARNING??
Data poisoning methods provide natural recipe for evaluating the unlearning ability of given machine unlearning algorithm. We consider the following four-step procedure:
Step 1: Implement the data poisoning attack generate the corrupted training dataset Soarr.
Step 2: Train the model on the corrupted dataset Scor. Measure the effects of data poisoning on the trained model nital
Step 3: Run the unlearning algorithm to remove all poison samples U = Spoison from Giniiall and compute the updated model @uddated.
Step 4: Measure the effects of data poisoning on the updated model Oupdated-
Naturally, for ideal unlearning algorithms that can completely remove all influences of the forget set
U Spoisn. we expect that the updated model updaaad will not display any effects of data poisoning. Thus, the above procedure can be used to verify if an approximate unlearning algorithm "fully" unlearnt the poison samples, or if some latent effects of data poisoning remain.
CAN MACHINE UNLEARNING REMOVE POISONS?
We now evaluate state-of-the-ar unlearning attacks for the task of removing both targeted and untargeted data poisoning attacks across vision and language models.
Datasets. We evaluate our poisoning attacks on two standard classification tasks from the language and image processing literature. For the language task, we consider the IMDb dataset (Maas et al., 201 This dataset consists of 25000 training samples of polar binary labeled reviews from IMDb. The task is to predict whether given movie review has positive or negative sentiment. For the vision task, we use the CIFAR-10 dataset (Krizhevsky et al., 2010). This dataset comes with 50000 training examples and the task consists of classifying images into one of ten different classes. We typically show average results over runs for all vision models and 5 runs for the language models and usually report +1 standard deviation actoss these runs.
Models. For the vision tasks, we train a standard Resnet-18 model for 100 epochs. We conduct the language experiments on GPT-2 (355M parameters) LLMs (Radford et al., 2019). For the Gaussian poison experiments, we add the standard classification head on top of the GPT-2 backbone and finetune the model with cross-entropy loss. For the targeted poisoning attack, suggested by Wan et al. (2023) and finetune GPT-2 on the IMDb dataset using the following template for each sample: "[Input]. setting, we use the standard causal cross-entropy loss with an initial learning rate set to 5- 10 which encourages the model predict the next token correctly given a total vocabulary of C' possible tokens, where C' = 50257 for the GPT-2 model. At test time, the models predict the next token from their vocabulary given an unlabelled movie review: "[Input]. The sent iment of the review is:" We train these models for 10 epochs on the poisoned IMDb training dataset.
follow the setup
The sent iment of the review is [Label]" In this
Unlearning methods. We evaluate eight state-of-the-ee machine unlearning algorithms for deep learning settings: (GD, NGD, GA, EUK, CFk, SCRUB, NegGrad+, SSD). Gradient Descent (GD) continues to train the model Oriial on the remaining dataset Syrann U by using stochastic gradient descent (Neel et al., 2021). 1). Noisy Gradient Descent (NGD) is simple state-of-the-an modification of GD where we add Gaussian noise to the GD-update steps (Chien et al., 2024; Chourasia & Shah, 2023). Gradient Ascent (GA) is unlearning algorithm which removes the influence of the forget set U from the trained model by simply reversing the gradient updates that contain information about U (Graves et al., 2021; Jang al., 2022). Exact Unlearning the last k-layers (EUk) is unlearning

---

Published as a conference paper at ICLR 2025
-better unleaning)
unlearming
0.000
TPR@FPR- TPA@FPR-0.0L
No Hplearinng
TPROFPR-OOM
0.023
0.05--
No Untearming Porfect eariing
0.000
Perfect Unleaming
00
OFe
EUx
NGD
NGP
SChUB
55D
QA
0.002
EUK
(a) Standard MIA (Shokri et al., 2017)
(b) MIA via Gaussian poisoning
Figure 2: Standard MIA evaluations are insufficient for detecting unlearning violations. Left: At a low false positive rate (FPR=0.01), standard MIAs have low true positive rates, making them ineffective at identifying whether targeted sample was successfully unlearned. Right: Our proposed Gaussian poison attack achieves a higher true positive rate at the same FPR, improving the detection of unlearning failures. A full trade-off curve comparison is provided in Figure 10.
approach for deep learning settings that simply retrains from scratch the last k layers (Goel et al., 2022). Catastrophically forgetning the last k-layers (CFK) is a modification of EUK, with the only difference being that instead of retraining from scratch, we continue training the weights in the last k layers on the retain set Strain U. SCalable Remembering and Unlearning unBound (SCRUB) is a state-of-the-arr unlearning method for deep learning settings (Kurmanji et al., 2024). It casts the unlearning problem into student-teacher framework. NegGrad+ is finetuning based unlearning approach which consists of a combination of GA and GD. SSD (Foster et al., 2024) unlearns by dampening some weights in the neural network which has a high influence on the fisher information metric corresponding to the forget set as compared to the remaining dataset. A detailed description of the algorithms, and the corresponding choice of hyperparameters are deferred to Appendix D.2.
Compute budget. When evaluating an unlearning method, common hyperparameter across all the models is the compute budget given to the model. Clearly, if the compute budget is greater than that required for retraining the model from scratch, then the method is useless; Thus, the smaller the budget for a given level of performance the better. To put all the methods on equal footing, we allow each of them to use up to 10% of the compute used in initial training (or fine-tuning) of the model (we also experiment with 4%, 6%, and 8% for comparison). This way of thresholding the compute budget is inspired by Google's 's unlearning challenge NeurIPS 2023, since the reason we care about approximate unlearning is to give the model owner a significant computational advantage over retraining from scratch. At the same time, we note that even giving 10% of the compute- budget to the unlearning method is quite generous, given that in modern settings like training a large language or vision model, 10% of training compute is still significant in terms of time and cost; practical unlearning algorithms should ideally work with far less compute.
Evaluating unlearning. When evaluating the efficacy of an unlearning method two objectives are essential: i) We measure post-unlearning performance by comparing the test classification accuracy of the updated model to the model retrained without the poisoned data. ii) To gauge unlearning validity against different poisoning attacks, we use different metrics for targeted attacks, Gaussian poisons, and indiscriminate attacks. For indiscriminate data poisoning attacks, the goal is to decrease test accuracy, and so we can conclude that an unlearning algorithm is successful if the test accuracy after unlearning approaches that of a retrained model - note this is the same metric as for model performance. For targeted data poisoning attacks. where the goal is to cause the misclassification of a specific set of datapoints, an unlearning algorithm is valid if the misclassification rate on this specific set of datapoints is close that of the retrained model. Note in this case that this is distinct from model performance, which measures test accuracy. For Gaussian data poisoning attacks, we first assess how good unlearning works by measuring how much information the Gaussian poisons leak from the model when no unlearning is performed, labeled as No unlearning in all figures. represents the TPR low FPR of the poisoned model before unlearning. We then evaluate the success of the unlearning process by determining if the forget set is effectively removed and if the model's 's original behavior restored, labeled as Perfect Unlearning in all Figures.
5.1 STANDARD MIA UNLEARNING EVALUATIONS CAN BE MISLEADING
Prior work typically evaluates the efficacy of unlcarning methods using MIAS (Shokri et al., 2017). However, Figure 2 shows that this approach is insufficient. None of the machine unlearning algorithms
7

---

Published as a conference paper at ICLR 2025
Model Performance Unleaming
Text Accuracy (80) (mfore aicural uuutee
Retraining
Retraiing
TPRII TRR-0FR--0000 Accuraty better unlearning
4b
buttoer unesiming
3.30
Untearning Elfaacy
0.00
Uniesrring Efficccy
Unleemiing
Unlearinn
TPR
2.200
0.20
Perfect Unlearting
0.30
Unicarning
0.00
00
0N
51
380
18P
SCAU5
530
04
CH
851
N50
MPP
5Oh05
6A
(a) Resnet- 18 on CIFAR-10
(b) GPT-2 (355M) on IMDb
Figure 3: Unlearning fails remove Gaussian poisons across variety of unlearning methods. We poison 1.5% the training data by adding Gaussian noise with standard deviation E PIIMDs = 0.1 and s P,,CIFAR110 = 0.32, respectively. We train/finetune a Resnet18 for 100 epochs and a GPT-2 for 10 epochs the poisoned training datasets, respectively. Finally, budget (i.c., 1 or 10 epochs) to unlearn the poisoned points. None of the unlearning methods removes the poisoned points the orange vertical bars do not match the dashed black retraining benchmark.
10% of the original compute
Modal Performance Post Ue ariaaiing
Betrafeing
Retraining
(mmore acurtte
Accuraccy (more
D.K
06
Poison Success (8)
better unleamingl unleamiinl
:00
Undeaming Elficay
NolUhleaming
(- Polson Successs
unteamiing)
1:00
Unlwariing Efficesy
No Unicaming
Perfect Unlearnnn
Perfect earning
00
CPK
1Mk
M2P
SCRJ0
550
64
GD
2.k
RCD
MGP
GA
(a) Resnet-18 on CIFAR-10
(b) GPT-2 (355M) on IMDb
Figure Unlearning fails remove targeted and backdoor poisons across variety of unlearning methods. We poison 1.5% of the training data by adding Witch's 's Brew poisons (Geiping et al., 2021)
to a Resnet-18 trained on CIFAR-10 or instruction poisons (Wan et al., 2023) a GPT-2 finetuned on IMDb. We then train/finetune Resnet-18 18 for 100 epochs and a GPT-2 for 10 epochs on the poisoned training datasets, respectively. both cases, we use roughly 1/10 of the original compute budget (10 epochs for CIFAR-10 epoch for IMDb) to unlearn the poisoned points. None of the considered methods remove poisoned points.
fully eliminate the influence of the deletion set U from the updated model Oudaaeed when evaluated using our proposed Gaussian poisoning attack. Although most algorithms perform well against the standard MIA, this can lead to misleading conclusion. An auditor relying solely standard MIAs might incorrectly assume that all these methods effectively unlearn data (see Figure 2a). As demonstrated Figure 2b, Gaussian data poisoning reveals that approximate unlearning methods do not remove the points from set U, despite their success under the standard MIA.
5.2 EXPERIMENTAL RESULTS
Below we discuss our key observations and main experimental takeaways and defer more detailed relative comparison between unlearning methods to Appendix E.
No silver bullet unlearning algorithm that can mitigate data poisoning, None the evaluated methods completely remove the poisons from the trained models; See Figures 3, and Table 1 and the caption therein for details on the failure of unlearning methods to remove poisons. The respective plots show that none of the methods performs par with retraining from scratch in terms
8

---

Published as a conference paper at ICLR 2025
GD
CFK
FUK
SCRUB
#Epochs
Retrain
NGD/NGPIGA
1.5%
2.5%
1.5%
2.5%
2.5%
1.5%
2.5%
1.5%
2%
2.5%
2 87.04
10
83.67
84.34
83.48
68.09
69.71
59.83
29.31
27.71
25.18
83.72
84.21
82.67
4
88.23
10
85.86
86.05
85.37
69.39
71.13
61.55
39.81
39.33
33.00
85.31
85.35
83.97
6 88.79
10
86.81
86.88
86.11
70.27
71.91
62.57
43.51
44,83
38.43
85.39
85.43
84.07
8 89.14
10
87.31
87.27
86.45
70.77
72.33
63.30
47.27
48.02
40.84
85.46
85:57
84.17
10
89,24
10
87.71
87,57
86.69
71.20
72.69
63,80
49.90
50.65
43,26
85,48
85.45
84.15
Table I: Results of unlearning indiscriminate data poisoning on CIFAR-10 in terms of test accuracy (%). The test accuracy of the poisoned models is 81.67%, 77.20% and 69.62% for 750, 1000, and 1250 poisoned points respectively. NGP and GA exhibit random guesses (10% test accuracy) across all poison budgets. We perform a linear search for the learning rate between [le 6, 5e 5] and report the best accuracy across all methods. All the results are obtained by averaging over 8 runs.
of post-unlearning test accuracy and effectiveness in removing the effects of data poisoning, thus suggesting that we need to develop better approximate unlearning methods for deep learning settings.
2) Different data poisoning methods introduce different challenges for unlearning. We observe that the success of an unlearning method in mitigating data poisoning depends on the poison type. For example, while GD can successfully alleviate the effects of indiscriminate data poisoning attacks for vision classification tasks, it typically fails to mitigate targeted or Gaussian poisoning attacks even while maintaining competitive model performance. Along similar lines, while SCRUB succeeds in somewhat mitigating Gaussian data poisoning in text classification tasks, it completely fails to mitigate targeted or indiscriminate data poisoning. This suggests that the different data poisoning methods complement each other and that to validate an unlearning algorithm, we need to consider all the above-mentioned data poisoning methods, along with other (preexisting) evaluations.
3) The success of an unlearning method depends on the underlying task. We observe that various unlearning algorithms exhibit different behaviors for image classification and text classification tasks, e.g., for data poisoning on a GPT-2 model, while EUK and NGD succeed in alleviating Gaussian data poisoning for the model trained with a classification head, they fail to remove targeted data poisoning on the same model trained with text decoder. Similarly, GA succeeds in alleviating Gaussian and targeted data poisoning for Resnet- 18 but fails to have a similar improvement for GPT-2 model. This suggests that the success of an approximate unlearning method over one task may not transfer to other tasks, and thus further research is needed to make transferable approximate unlearning methods.
5.3 SENSITIVITY ANALYSIS AND ADDITIONAL INSIGHTS
Additional experiments detailed in Appendix E, explore 1) the impact of varying the number of update steps and 2) the effect of varying the forgetset size. For methods like NGD, increasing the number of update steps generally enhances unlearning effectiveness (see Figure 11b, orange bars). However, applying NGD to LLM models results in a substantial decrease in post-unlearning test accuracy, dropping by 10%. Conversely, for methods like EUK, additional steps do not improve unlcarning or post-umlearning test accuracy (see Figure 1 Ia). These trends are summarized in Figure 11. Furthermore, we experimented with different sizes of the forgetset. For Gaussian poisoning attacks, the results, summarized in Figures 13 and 12 of Appendix E, confirm consistent trends when 1.5%, 2%, and 2.5% of the training dataset are poisoned.
6 UNDERSTANDING WHY UNLEARNING FAILS TO REMOVE POISONS?
In Section 5.2, we demonstrated that various state-of oftee-of-the-art approximate machine unlearning al- gorithms fail to fully remove the effects of data poisoning. Given these results, one may wonder what is special about the added poison samples, and why gradient-based unlearning algorithms fail to rectify their effects. In the following, we provide two hypotheses for understanding the failure of unlcarning methods. We validate these hypotheses using set of experiments based on linear and logistic regression on Resent-18 18 features which allow us to study these hypotheses experimentally.
*Our hypothesis for why EUk fails for text generation is that it results in severe degradation of the model's 's text generation capabilities due to re-initialization and fine-tuning of the last & layers of the model from scratch.
9

---

Published as a conference paper at ICLR 2025
Distance
Figure 5: Model shift for logistic regression on Resnet-18 features for CIFAR-10 dataset. The X-axis is the number of epochs. The blue and the red curves denote the distance ||0 (Sooo)) (Scor paison 1, for indiscriminate and targeted data poi- soning respectively, where / is the corresponding percentage of unlearned poison samples. For dataset S" 0(5') denotes model trained from scratch on S'. The orange curve plots the distance [|((5) 8( < stand) ) corresponding to unlearn- ing random clean training samples.
0.175
0.150
0.125
0.100
0.075
0.050
0.025
0.000
0
20
40
60
80
100
Indioormiinae
Targeted
Random
Thanks the convexity of the corresponding loss the objectives have unique global minimizers making it easier to understand model shifts due to unlearning.
Hypothesis 1: Poison samples cause large model shift, which cannot be mitigated by approxi- mate unlearning. We hypothesize that the distance between a model trained with the poison samples and the desired updated model obtained after unlearning poisons much larger than the distance between a model trained with random clean samples and the desired updated model. Thus, any unlearning algorithm that attempts to remove poison samples needs shift the model by a larger amount. Because larger shifts typically need more update steps, unlearning algorithms are unable to mitigate the effects of poisons the allocated computational budget. To validate this hypothesis, Figure 5 shows the & norm of the model shift introduced by unlearning data poisons and random clean training data, demonstrating that data poisons introduce much larger model shifts in this norm as compared to random training samples.
Hypothesis 2: Poison samples shift the model in a subspace orthogonal to clean training samples. We next hypothesize that training with poison samples not only shifts the model by larger amount, but the resultant shift lies in a subspace orthogonal to the span of clean training samples. Thus, gradient-based update algorithms that attempt unlearning with clean samples fail to counteract shifts within this orthogonal subspace and are unable to mitigate the impacts of data poisoning. To completely unlearn the effects of poison samples, unlearning algorithm must incorporate gradient updates that specifically utilize these poison samples. To validate this hypothesis, in Figure 14, we plot the inner product between the gradient update direction for gradient descent using clean training samples and the desired model shift direction, for unlearning for data poisons and random clean training samples respectively, for simple linear regression task. The random subset of clean training samples is chosen so as to equate the model shift in both unlearning data poisons and random training samples. Figure 14 shows that the desired unlearning direction for data poisons orthogonal the update direction from gradient descent as the cosine similarity between the update directions is small.
7 CONCLUSION
Our experimental evaluation of state-of-the-art machine unlearning methods across different models and data modalities reveals significant shortcomings in their ability to effectively remove poisoned data points from trained model. Despite various approaches which attempt to mitigate the effects of data poisoning, none were able to consistently approach the benchmark results of retraining the models from scratch. This highlights critical gap in the true efficacy and thus practical value of current unlearning algorithms, questioning their validity in real-world applications where these unlearning methods may be deployed to ensure privacy, data integrity, or correct model biases. Furthermore, our experiments demonstrate that the performance of unlearning methods varies significantly across different types of data poisoning attacks and models, indicating lack of one-size-fits-all solution. Given the increasing reliance on machine learning in critical and privacy-sensitive domains, our findings emphasize the importance of advancing rigorous research in machine unlearning develop more effective, efficient, and trustworthy methods, that are either properly evaluated or have provable guarantees for unlearning. Future work should focus on creating novel unlearning algorithms that can achieve the dual goals of maintaining model integrity and protecting user privacy without the prohibitive costs associated with full model retraining.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Hojjat Aghakhani, Dongyu Meng, Yu- Au-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye polytope: A scalable clean-label poisoning attack with improved transferabilltyy In 2021 IEEE European symposium on security and privacy (EuroS&P), pp. 159-178. IEEE, 2021.
Nora Belrose, David Schneider-Joseph. Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36, 2023.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Conference Machine Learning, ICML '12, pp. 1467- 1474. JMLR, Inc., 2012.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo. Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearming, In proceedings of the 42nd IEEE Symposium on Security and Privacy, SP '21. IEEE Computer Society, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In Proceedings of the 36th IEEE Symposium on Security and Privacy, SP 15, pp. 463-480. IEEE Computer Society, 2015.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022a.
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. In Advances in Neural Information Processing Systems 35, NeurIPS '22, pp. 13263-13276. Curran Associates, Inc., 2022b.
Huiqiang Chen, Tianqing Zhu, Xin Yu, and Wanlei Zhou. Machine unlearning via null space calibration. arXiv preprint arXiv:2404. 13588, 2024.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. In Proceedings of the 202/ ACM Conference Computer and Communications Security, CCS '21, pp. 896-911. ACM, 2021.
Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. Langevin unlearning: A new perspective noisy gradient descent for machine unlearning, arXiv preprint vrXiv:2401.1037/ 2024.
Rishav Chourasia and Neil Shah. Forget unlearning: Towards true data-deletion in machine learning. In International Conference on Machine Learning, pp. 6028-6073. PMLR, 2023.
Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon, Werner Zellinger, Bernhard A Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, and Fabio Roli. Wild patterns reloaded: survey of machine learning security against training data poisoning. ACM Computing Surveys (CSUR), 55(13s):1-39 2023.
Jimmy Z Di, Jack Douglas, Jayadev Acharya, Gautam Kamath, and Ayush Sekhari. Hidden poison:
Machine unlearning enables camouflaged poisoning attacks. In Advances in Neural Information Processing Systems 36, NeurIPS '23. Curran Associates, Inc., 2023.
Jinshuo Dong, Aaron Roth, and Weijie Su. Gaussian differential privacy. Journal of the Royal Statistical Society Series Statistical Methodology, 84(1):3-37, 2022.
Chongyu Fan, Jiancheng Liu. Yihua Zhang, Eric Wong. Dennis Wei, and Sijia Liu. Salun: Em- powering machine unlearning via gradient- based weight saliency in both image classification and generation. In International Conference Learning Representations (ICLR), 2023.
11

---

Published as a conference paper at ICLR 2025
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unleamiig without retraining through selective synaptic dampening. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12043-12051, 2024.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3 (4):128-135, 1999.
GDPR. Regulation (EU) 2016/679 of the European parliament and of the council of 27 April 2016, 2016.
Jonas Geiping, Liam Fowl, W Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. In Proceedings of the 9th International Conference on Learning Represeniations, ICLR '21, 2021.
Antonio Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. Making ai forget you: Data deletion machine learning. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019). 2019.
Shashwat Goel, Ameya Prabhu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, and Ponnurangam Kumaraguru. Towards adversarial evaluations for inexact machine unlearning. arXiv preprint arXiv:2201.06640, 2022.
Shashwat Goel, Ameya Prabhu, Philip Torr, Ponnurangam Kumaraguru, and Amartya Sanyal. Cor- rective machine unlearning. arXiv preprint arXiv:2402.14015, 2024.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Putlern Recognition (CVPR), 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. arXiv:2003.02961 2020b.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Mądry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1563 1580, 2022.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11516-11524, 2021.
Tianyu Gu, Brendan Dolan Dolan-Gavitt and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 1oXiv:/708.0673: 2017.
Tianyu Gu, Kang Liu, Brendan Gavint,, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 1447230-47244 2019.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learing (ICML), 2019.
Junfeng Guo and Cong Liu. Practical poisoning attacks on neural networks. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pp. 142-158. Springer, 2020.
Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid false sense of privacy. arXiv preprint arXiv:2403.01218, 2024.
Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V. Pearson, Dietrich A. Stephan, Stanley F. Nelson, and David W. Craig. Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS Genetics, 4(8):1 18), 2008.
W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general-
general-purpose purpose clean-label data poisoning. In Advances in Neural Information Processing Systems 33, NeurlPS '20, pp. 12080-1200 Curran Associates, Inc., 2020.
12

---

Published as a conference paper at ICLR 2025
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint 1rX1v:2210.01504, 2022.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning. ICML '17, pp. 1885-1894. JMLR, Inc., 2017.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning, 111(1):1-47, 2022.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). 2010.
Meghdad Kurmanji, Peter Triantafillou, and Eleni Triantafillou. Towards unbounded machine unlearning. arXiv preprint 230.::2300.098800, 2023.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances Neural Information Processing Systems, 36, 2024.
Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Gaussian membership inference privacy. Advances in Neural Information Processing Systems, 36, 2024.
Zihao Liu, Tianhao Wang, Mengdi Huai, and Chenglin Miao. Backdoor attacks via machine unlearning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14115-14123, 2024.
Yiwei Lu, Gautam Kamath, and Yaoliang Yu. Indiscriminate data poisoning attacks on neural networks. Transactions on Machine Learning Research, 2022.
Yiwei Lu, Gautam Kamath, and Yaoliang Yu. Exploring the limits of model- targetd indiscriminate data poisoning attacks. In Proceedings of the 40th International Conference on Machine Learning, ICML '23, pp. 22856-22873 JMLR, Inc., 2023.
Yiwei Lu, Matthew YR Yang, Gautam Kamath, and Yaoliang Yu. Indiscriminate data poisoning attacks on pre-trained feature extractors. arXiv preprint arXiv:2402. 20.002.22622 /2626, 2024.
Zhuo Ma, Yang Liu, Ximeng Liu, Jian Liu, Jianfeng Ma, and Kui Ren. Learn to forget: Machine unlearning via neuron masking. IEEE Transactions on Dependable and Secure Computing. 2022.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 201
Neil G. Marchant, Benjamin I. P. Rubinstein, and Scott Alfeld. Hard to forget: Poisoning attacks on certified machine unlearning, 2022a. URL https: / /tta//avvv.oo// org/ / 08266.
Neil G Marchant, Benjamin IP Rubinstein, and Scott Alfeld. Hard to forget: Poisoning attacks on certified machine unlearning. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, volume 36 of AAA! '22, pp. 7691-7700, 2022b.
Luis MunozeGonzález, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedingy of the 10th ACM Workshop Artificial Intelligence and Security, AlSec 17, pp. 27-38. ACM, 2017.
Seth Neel, Aaron Roth, and Saeed Malvaj--alvjjerrd.. Descent-to-delete Gradient-based methods for machine unlearning. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, ALT '21. JMLR, Inc., 2021.
13

---

Published as a conference paper at ICLR 2025
Martin Pawelczyk, Tobias Leemann, Asia Biega, and Gjergji Kasneci. On the trade-off between actionable explanations and the right to be forgotten. In International Conference on Learning Representations (ICLR), 2023.
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. In International Conference on Machine Learning (ICML), 2024.
Wei Qian, Chenxu Zhao, Wei Le, Meiyi Ma, and Mengdi Huai. Towards understanding and enhancing robustness of deep learning models against malicious unlearning attacks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1932-1942, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. Linear adversarial concept erasure. In International Conference Machine Learning, pp. 18400-18421. PMLR, 2022a.
Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 6034-6055, 2022b.
Stefan Schoepf, Jack Foster, and Alexandra Brintrup. Parameter-tuning-free data entry error unlearn- ing with adaptive selective synaptic dampening. arXiv preprint arXiv:2402.10099, 2024a.
Stefan Schoepf, Jack Foster, and Alexandra Brintrup. Potion: Towards poison unlearning. arXiv preprint arXiv:2406.09) 2024b.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Advances in Neural Information Processing Systems, 2021.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems 31, NeurIPS '18 pp. 6106-6116. Curran Associates, Inc., 2018
Shawn Shan, Arjun Nitin Bhagoji, Haitao Zheng, and Ben Y. Zhao. Poison forensics: Traceback of data poisoning attacks in neural networks. In 3/st USENIX Security Symposium (USENIX Security 22), pp. 3575-3592, Boston, MA, August 2022. USENIX Association. ISBN 978-1- 939133-31-1. URL https://wwwwwusiii.. presentation/ shan.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In Proceedings of the 38th IEEE Symposium on Security and Privacy, SP 17, pp. 3-18. IEEE Computer Society, 2017.
David M Sommer, Liwei Song, Sameer Wagh, and Prateck Mittal, Athena: Probabilistic verification of machine unlearning. Proceedings on Privacy Enhancing Technologies, 2022.
David Marco Sommer, Liwei Song, Sameer Wagh, and Prateek Mittal. Towards probabilistic
verification machine unlearning, 2020. URL https::///// //tpppi://axxi....///.paapa. org/ abs/2003. 04247.
Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. In Advances in Neural Information Processing Systems 30, NeurlPS '17, pp. 3520-3532. Curran Associates, Inc., 2017.
Xu Sun, Zhiyuan Zhang, Xuancheng Ren, Ruixuan Luo, and Liangyou Li. Exploring the vulner- ability of deep neural networks: A study of parameter corruption. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020. URL https:://jjj aaai. org//ndex: indexxixxx. php/ AAAI/artin icle/ /wt.///www///////
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. In International Conference on Machine Learning, pp. 35413-35425. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. Kga: A general machine unlearning framework based on knowledge gap alignment. arXiv preprint arXiv:2305.00533, 2023.
Jiancan Wu, Yi Yang, Yuchun Qian, Yongduo Sui, Xiang Wang, and Xiangnan He. Gif: A general graph unlearning strategy via influence function. In Proceedings of the ACM Web Conference 2023, www '23. ACM, April 2023. doi: 10. 10.1145/35430073383321 URL http: /xx doi.
1t5e//0..14/3354450 30114435577.3352222 3583521.
Yinjun Wu, Edgar Dobriban, and Susan Davidson. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning (ICML), 2020.
Rui Zhang and Shihua Zhang. Rethinking influence functions of neural networks in the over- parameterized regime. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
15

---

Published as a conference paper at ICLR 2025
CONTENTS OF APPENDIX
Additional Related Work
17
Gaussian Data Poisoning
18
B.1 Motivation
18
B.2 Gaussian Poisoning as a Hypothesis Testing Problem.
B.3 An Illustrative Edge Case
19
B.4 Algorithms
19
C Unlearning Evaluation Methods: Methodological Comparisons
22
D Implementation Details
23
D.I Existing Data Poisoning Attacks
23
D.1.1 Targeted Data Poisoning for Image Classification
24
D.1.2
Backdoor Data Poisoning for Language Sentiment Analysis
25
D.1.3
Indiscriminate Data Poisoning
25
D.2
Unlearning Algoritmss............
26
D.2.1
Gradient Descent (GD)
26
D.2.2
Noisy Gradient Descent (NGD)
27
D.2.3
Gradient Ascent (GA)
27
D.2.4
EUK
27
D.2.5
CFk
27
D.2.6
SCRUB
27
D.2.7
NegGrad+
................................
28
D.2.8
Selective Synaptic Dampening (SSD)
28
Experiments
29
E.1
Detailed comparison of different unlearning algorithms
29
Comparison of Gaussian and Data Poisoning
.
29
E.3
Additional Experiments
29
Understanding Why Approximate Unlearning Fails?
32
F.1
Logistic Regression Experiment to Validate Hypothesis
32
F.2
Linear Regression Experiment to Validate Hypothesis
16

---

Published as a conference paper at ICLR 2025
Additional Notation. We use the notation o4)) to denote a gaussian random variable in d dimensions with mean 0 and covariance matrix l' For a dataset S, we use Uniform to denote uniformly random sampling from S, and the notation F [==) to denote the empirical average [s] Exes g(z) for any function g. For vector u, U R we use the notations || = [aX u[t]
to denote the fo norm of u. ||u||2 Eieta "[i]² to denote the l2 norm of u, ||l| Σ ir to
denote the & norm of and (u, v) to denote the inner product between vectors a and v.
A ADDITIONAL RELATED WORK
Data poisoning attacks. In a data poisoning attack, an adversary may introduce or modify a small portion of the training data, and their goal is to elicit some undesirable behavior in model trained on said data. One type of attack is a targeted data poisoning attack (Koh & Liang, 2017; Shafahi et al., 2018; Huang al., 2020; Guo & Liu, 2020; Aghakhani et al., 2021), in which the goal is to cause a model to misclassify a specific point in the test set. Another type of attack is an untargeted (or indiscriminate) data poisoning attack (Biggio et al., 2012; Muñoz-Gonzälez et al., 2017; Steinhardt et al., 2017; Koh et al., 2022; Iu et al., 2022; 2023), wherein the attacker seeks to reduce the test accuracy as much as possible. Though we do not focus them in our work, there also exist backdoor attacks (Gu et al., 2017), in which training points are poisoned with a backdoor pattern, such that test points including the same pattern are misclassified and various detection techniques (Shan et al., 2022).
Poisoning machine unlearning systems. An orthogonal line of work investigates data poisoning attacks against machine unlearning pipelines (see, e.g., Chen et al. (2021); Marchant et al. (2022b); Carlini al. (2022h); Di al. (2023); Qian et al. (2023); Liu et al. (2024)). These works generally show that certain threats can arise even if unlearning is performed with provable guarantees, whereas we focus on data poisoning threats in standard (i.c., not machine unlearning) pipelines, that ought to be removed by an effective machine unlearning procedure (in particular, they would be removed by retraining from scratch without the poisoned points).
Evaluation works. Several prior works machine unlearning evaluation have explored verifying the effect of unlearning through data poisoning in various settings and context (Wu et al., 2023; Marchant et al., 2022a; Sommer et al., 2020). In particular, Wu et al. (2023) address the problem of graph unlearning, where they evaluate attacks involve adding adversarial edges to graph. The authors demonstrate that both the influence function method and its extension, GIF, can mitigate the impact of these adversarial edges.
Meanwhile, Marchant et al. (2022a) focus exclusively on unlearning through Influence Functions (IF) on convex models. In this setting, the authors introduced a specifically designed backdoor data poisoning attack which is optimized knowing that the model owner will train. deploy, and updates their convex model using influence functions. Furthermore. the conclusion of their work that the field of machine unlearning requires more rigorous evaluations aligns with this work, considering the poisoning methods implemented in our work require less knowledge of the target model and are agnostic to the specific unlearning methods used (see Table 2).
Sommer et al. (2020) provide framework for verifying exact data deletion. In a fundamentally different setting, the authors evaluate whether an MLaaS provider complies with a deletion request
Model Architecture
Unlearning
Reference
Attack Types
Method
Using
Marchant al. (2022a)
Backdoor attack
Convex model
Influence
Functions (IF)
Indiscriminate Attack, Targeted Attack, Backdoor Attack, Gaussian
This work
Any
Any
Attack
Table 2: Comparing the data poisoning settings of this work Marchant et al. (2022a).
17

---

Published as a conference paper at ICLR 2025
by completely removing the data point from the trained model. Hence, the findings in their work independent of any unlearning methods. To verify a complete takedown, their framework involves running backdoor attacks to subsequently check if complete case-based deletion done by the aaa provider.
B GAUSSIAN DATA POISONING
Beyond the descriptions from Section Section 4.3, here we provide an alternative way to compute the amount of privacy leakage due the injected Gaussian poisons (see Figure 7 for brief summary of the results). Further, we provide some intuitive understanding of why Gaussian poisons work at evaluating unlearning success.
B.1 MOTIVATION
Evaluating Gaussian poisons. The effect of data poisoning model 0 is measured by the dependence of the model on the added perturbations x Let 9 be a model to be evaluated (which may or may
have been corrupted using poisons). In order evaluate the effect of poison samples on 0, for every poison sample = E Spoison. we compute the normalized inner product 1g = (9xx5= la with 9x 2000 (Tease. 3)), where 9s IRd denotes the gradient of the model 0 w.r.t. the input space x when evaluated at the clean base image (Iease: y) corresponding to the poisoned sample 2, and define the set Ipoison = {I=)eeSuaor.
Interpreting the Gaussian poison attack as a membership inference attack. Consider a routine that samples a point v from Ipcisoo + IiIdeep. computes I2 using the unlearned model, and then guesses that = poossnn if [ т. Under exact unlearning, this attack should have trivial accuracy, achieving TPR- FPR at every value of r. To illustrate, consider the right most panel from Figure 6 where unlearning is not exact since the blue histogram deviates from the teal (O, 1) distribution curve which represents perfect unlearning. Hence, we measure unlearning error, by the extent to which a classifier achieves nontrivial accuracy when deciding whether samples are from Ipaison or Tindep in particular focusing on the true positive rate (TPR) false positive rates (FPR) or below 0.01 (denoted as TPR@FPR=0.01), This measure corresponds to the orange bars we report in Figure 3.
One way to view this metric is as measure of the attack success of an adversary that seeks to distinguish between poisoned training points that have been subsequently unlearned, and test poison points, using an attack that thresholds based on Lg. This corresponds to evaluating unlcarning via Membership Inference Attack (MIA), similar in spirit to recent work (Pawelezyk al., 2024; Hayes et al., 2024; Kurmanji et al., 2023). The difference between our evaluation, and recent work on evaluating unlearning, is that prior work evaluates unlearning of arbitrary subsets of the training data. As result, building an accurate attack requires sophisticated techniques that typically involve an expensive process of training additional models called shadow models, using them to estimate distributions on the loss of unlearned points, and then thresholding based on likelihood ratio. This is in stark contrast to our setting, where because our Gaussian poisons are explicitly designed to be easy to identify (by thresholding on I,) we do not need to develop sophisticated MIA to show unlearning hasn't 't occurred.
To assess how good unlearning works, consider how much information the Gaussian poisons leak from the model when no unlearning is performed, labeled as No unlearning in all figures. It represents the TPR at low FPR of the poisoned model before unlearning (solid orange lines in Figures and 4). We evaluate the success of the unlearning process by determining if the forget set is effectively removed and if the model's original behavior is restored. Ideally, the TPR at low FPR should equal the FPR (dashed black lines in Figure 3).
B.2 GAUSSIAN POISONING AS A HYPOTHESIS TESTING PROBLEM.
We can translate the above reasoning into membership hypothesis test of the following form:
Ho: The model was trained on Syain without & (perfect unlearning/ / is a test poison);
H1: The model was trained Stain with & (imperfect unlearning / no unlearning).
18

---

Published as a conference paper at ICLR 2025
Constructing the test statistic. The Gaussian Unlearning Score (GUS) uses the following simple fact about Gaussian random variables to devise an unlearning test: Let & (0( and let constant with respect to 5, then (g.5)
cplal N(O,
Ho: Consider the model's gradient at base image. If the gradicnt is constant with respect to 5, then their normalized dot product will follow a standard normal distribution.
H1: When unlearning did not succeed and g may depend &, then (g.S) sy
will deviate from
standard normal distribution. In particular, we can use the deviation of E (g-E) measure the ineffectivenesss of approximate unlearning.
from 0 to
B.3 AN ILLUSTRATIVE EDGE CASE
Blessing of Dimensionality: Higher Input Dimension Contributes to Higher Power. Here our goal is to understand the factors that determine the success of the Gaussian poisoning method. For the sake of intuition, the following, we provide an artificial example demonstrate the change in distribution from (0, 1) when & does not depend hypothesis when y depends on Ç. Suppose the poison sample E Spoison generated by adding the noise& &a to the base sample (Tbase; y) in the clean training dataset. For illustrative purposes, we will consider an extreme case.
to the distribution under the alternative
Ho: When Is is constant
& (for example for model which has completely unlearned the (Rt.Ee)
poison samples), we have that I, < ~ N(O, 1) for each poison sample 2 Spoison-
H1: Suppose that the gradient Is in the sample space w.r.t. the clean training sample (Toase: !) corresponding to the poison sample only memorizes the poison and hence satisfies the relation 9: = m.. Then, (x=.Ez) (E),Ez) denotes a sum of many x -raandom variables
with expectation and variance 2d and then E[?,] E ((=
v4/2. Further, by the Central Limit Theorem, 1 converges in distribution to N (V222,,1
v2d
For this special case, we thus see that our hypothesis testing problem boils down to comparing two normal distributions to each other; one with mean and standard deviation 1, and the other with mean f 2 and standard deviation By the Neyman-Pearsee Lemma, we know that the best true positive rate (TPR) at given false positive rate (FPR) for this problem is given by:
TPR(FPR) = ( (1 - FPR) ()) with = V422.
(3)
This suggests that we will be able to better distinguish perfect unlearning from unlearning failure the higher the input dimension is.
Moving beyond the Edge Case. In more practical use cases, where closed-for computation is not feasible, y likely depends on the model architecture, the optimizer, and the training dynamics. Our simple theory predicts that the Gaussian Unlearning Score should grow slower than linearly, with diminishing benefits from additional dimensions. In Figure 8, verify that higher imput dimensions make Gaussian data poisoning more effective in more complex scenarios. We trained ResNet18 models for 100 epochs the CIFAR-10 dataset, varying the input size from 26x26x3 32x32x3, and subsequently unlearn 1000 data points using NGD with noise level & = 1e - 07. The figure demonstrates the diminishing benefits of adding additional data dimensions.
B.4 ALGORITHMS
thus compare which of the two distributions does Lg belong to Computing GUS. In practice (9. Informally speaking, the further away this mean from by evaluating the mean [Seaer| Σ. lolool the higher is the influence of the data poisons on the underlying models. Algorithm shows how we compute GUS.
Further details on the Gaussian poison attack. As we have clarified in the main text, the Gaussian poisoning attack attempts to induce dependence between the gradient with respect the updated
19

---

Published as a conference paper at ICLR 2025
Algorithm I Gaussian Unlearning Score (GUS)
Input: Model be evaluated.
Poison samples Spoison and added noise {5= I Sposon"
1: Initialize Ipotson Ø.
2:
for Spoison do
3:
Let (Ibase: y) be the clean training sample corresponding to the poison sample 2.
(rasee 4: Compute imput gradient 9: (f y) on the corresponding clean training sample.
5: Let I
1p10.12 where &. denotes the noise used to generate the poison sample Update Ipoison Ipaison {I2}.
6:
7:
Return Sppsonl
E 2ccSpassn
L.
Algorithm 2 Gaussian Data Poisoning Evaluate Unlearning
Input:
Unlearning algorithm Unlearn-Alg to be evaluated.
Training dataset S.
Number of poison samples P.
Variance of the gaussian noise for data poisoning: eq.
1: I/ Generate poison samples and corrupted training dataset for Gaussian data poisoning //
2: Select P samples Spoison Uniform(S), w/o replacement, and let Sdean be the remaining samples.
3: for é Spoison do
4: Let (Ibase: y) be the clean training sample corresponding to the poison 2.
5: Define
Toon
Tbase & where
E. {~N(0,,,))a,,
and update the poison sample (Feor, y). Store &z.
6: Define the corrupted training dataset Scor = Selan n Spoison-
7: Obtain the initial model Onnaal by training on Sear-
8: // Evaluate the effect of data poisoning on the initial model //
9: Initialize Ipaison + Ø.
10: for É Spoison do
11:
Let (Ibase: y) be the clean training sample corresponding i.e. Tbase Fcor E..
12:
Compute (normalized) input gradient Iinitiat.: [V= (Rum)19)1
V- (#bans
13: Define I
(SImitial, Ex) and update Ipaison = Ipoison Iz.
Ep
Compute jimaal
ExeSprcn Lg.
SSpssnl
// Unlearn the added poison samples 16: Run the approximate unlearning algorithm Unlearn-Alg to unlearn the poison samples Spcison from m Let the updated model be uupaated-
17: // Evaluate GUS as the effect of data poisoning post unlcarning // Initialize Tupdated Ø.
19: for E Spoison do Let (Iease: y) the clean training sample corresponding to 2, i.e. Tbase Feor - E.- Compute (normalized) input gradient Jupciated,
Ve fe (Fross (3)
epl V L L0 (2uase
22:
Define l J. (Supdated, E.) and update Zupdated = Tupdated I'
Ep
23: Compute Aupdated [Spaasal
ExSSpeen I'. 24: // For perfect unlearning, Pupdated ~ N/0 1). Thus, when Pupdated is comparable to Anttall > then unlearning did not succeed. //
20

---

Published as a conference paper at ICLR 2025
Normalized Product Disoriutioo bctweon
Normallzo Dos Product Didr bution betweon Inpul Gradients Sheu and Gausser powsons 5
Nomollzzd Dut Product D budhibuston betweon Ripu Gradienis Sueces end Geussrt passons
riput Gredients Sustu and fresh Gauederss
Teary Independeence) AED
Theory indegendeente:::
1)
Theery (ndenendenee) At2
piow
1E-91 3.346.2 0.597
0.234.8
0.906
Fir: M Racl
fit: A0 (6)
FR: Alfix unteno (i
2.4
0.2
Dania
0.-
60
0.0
3.0
:ll0waal (E. [lmba)
001o (6. (iutt))))
CE, Duncoter)
Figure 6: The dot product between normalized clean input gradients and Gaussian sam- ples/poisons is again Gaussian distributed. We are testing unlearning using NGD with NaD = le 07 was successful for a Resnet-18 model trained on CIFAR-I0 where & ~N,00, eğ Ia) with ep- 0.32 was added subset of 750 training points (corresponding
1.5% of the train set) targeted for unlearning. Left: Distribution of dot products between freshly drawn Gaussians and clean input gradients of the initial model. Middle: Distribution of dot products between model poisons { and clean input gradients of the initial model. Right: Distribution of dot products between model poisons & and clean input gradients of the updated model. The columns demonstrate that the suggested dot product statistic is again Gaussian distributed with d² x 1 and mean parameter ja that varies depending on whether the poison is statistically dependent on the input gradients vx (x) or Vile (x). Comparing the left most column to the middle and right columns shows that
test can distinguish between Gaussians & that are independent of the model (left panel: the brown histogram matches the density of the standard normal distribution) and poisons dependent on the model since they were included in model training (middle and right panel: the orange and blue histograms match mean shifted Gaussian distributions).
model evaluated at the clean image, and the poisons {Ex { Larger values of this dependence statistic (Tg) after unlearning, are evidence that the unlearning algorithm did not fully remove the impact of the poisons.
The hyperparameters used to compute the Gaussian poisons in our experiments are:
2 M.DDA = =0.1,
-.IIAAR-10 0.32.
21

---

Published as a conference paper at ICLR 2025
Testing Uni Unlearning NGD with dig 1e
10P
10"1
10-4
Empirical tradesff curve bafore unleaming
Aralytical Tradcoff Curve: Sp-c.34
Empiriaal tradeoff curve after unlearning
Arwlytical Gaussiar Tradeoff Curve: Gjarass
10
false Positive Rate (FPR)
Figure 7: Empirical tradeoff curves (solid) match analytical Gaussian tradeoff curves (dashed). We plot the empirical tradeoff curve before and post unlearning the poison when NGD with R 1-e07 used for unlearning. Next to empirical tradeoff curve (solid), we plot the analytical Gaussian tradeoff curve G₁ 1 Φ( ( ( FPR) (4) (Dong al., 2022; Leemann et al., 2024) and observe that the match between the empirical and Gaussian tradeoff is excellent where + denotes the CDF for standard normal distribution. To summarize, since the orange and blue solid tradeoff curves are far from the diagonal line, which indicate a random guessing chance to distinguish the model's noise from a freshly drawn Gaussian ç. unlearning was not successful.
Before Unlearning
0.6
Untearring
Perfect Unleaming
Gaussian Unis.ming
0.4
0.2
2000
Input Dimension (6)
Figure 8: Blessing of Dimensionality. Our Gaussian data poisoning attack becomes more effective the higher the input dimension as suggested by our theoretical analysis Section B.3. We plot the Gaussian Unlearning Score before and post unlcarning as we vary the input dimension between 26x26x3 and 32x32x3 for a Resnet18 initially trained on Cifar10 for 100 epochs. Unlearning done via NGD with 2ND = 1-e07.
C UNLEARNING EVALUATION METHODS: METHODOLOGICAL COMPARISONS
Our Gaussian data poisoning method overcomes key limitations of the other data poisoning-bars bbssed unlearning evaluations through four critical dimensions:
Computational Efficiency: Gaussian poisoning offers a dramatic improvement in computational complexity compared to any other unlearning evaluation (see Table 5). Unlike existing methods that require complex optimization procedures taking minutes (c.g.. targeted and indiscriminate data poisoning attacks), our approach involves simple gradient computation and dot product
22

---

Published as a conference paper at ICLR 2025
Specific Training Data Knowledge
Training Algorithm
Model Architecture
Trigger Required
Adversary
Targeted Data Poisoning
Indiscriminate Data Poisoning
Backdoor Data Poisoning
Gaussian Data Poisoming (Ours)
x
Table 3: Gaussian data poisoning has minimal knowledge requirements. Information that adversary needs to implement the corresponding data poisoning attacks considered in this work.
Adversary
Image Data
Language Data
Tabular Duta
Targeted Data Poisoning Indisctiminate Data Poisoning
Backdoor Data Poisoning
Gaussian Poisoning (Ours)
Table 4: Gaussian data poisoning is compatible with all common data formats. Data compatibility of different data poisoning methods considered in this work.
Adversary
Resnet18
GPT2
Targeted Data Poisoning
1/2 hour
N/A
Indiscriminate Data Poisoning
1 week
N/A
Backdoor Data Poisoning
N/A
minutes
Gaussian Data Poisoning (Ours)
1/2 minute
minute
Table 5: Gaussian data poisoning more computationally efficient.
calculation, completing in mere seconds. We include a comprehensive runtime comparison demonstrating this efficiency across different model architectures and datasets.
Minimal Knowledge Requirements: Gaussian poisoning stands for its minimal prerequisite knowledge. Where other unlcarning evaluation methods demand extensive model or dataset information, our approach requires minimal contextual understanding. Table provides com- prehensive comparison illustrating the knowledge constraints of different unlearning evaluation techniques.
Data Compatibility: Unlike existing targeted attacks limited to specific domains, Gaussian poisoning demonstrates remarkable versatility. Our method successfully operates across diverse data types including text, images, and tabular data (see Table 4. This generalizability is particularly significant given the challenges of extending existing methods to emerging domains like large language models.
Privacy Impact Measurement: Crucially, Gaussian poisoning directly addresses privacy concerns by precisely measuring individual sample information retention. Existing unlearning evaluations and data poisoning methods fail to provide this granular privacy impact assessment, making our approach uniquely valuable for understanding true unlearning effectiveness.
D IMPLEMENTATION DETAILS
D.1 EXISTING DATA POISONING ATTACKS
The poisoning methods that we consider in this paper capture diverse effects that small perturbations in the training data can have on the trained model. At high level, we chose the following three approaches as they complement each other in various ways: while targeted data poisoning focuses on certain target samples, indiscriminate data poisoning concerns with the overall performance on the entire test dataset, whereas Gaussian data poisoning does not affect the model performance at all. Furthermore, while targeted and indiscriminate attacks rely on access to the model architecture and training algorithm adversarially generate the perturbations for poisoning. Gaussian data poisoning
23

---

Published as a conference paper at ICLR 2025
is very simple implement and works under the weakest attack model where the adversary does not even need to know the model architecture or the training algorithm.
We provide the key implementation details below:
For the experiments on the CIFAR-I0 10 dataset, we implemented targeted, indiscriminate, and Gaussian data poisoning attack by adding 32 32 3-dimensional perturbations/noise to by {1.5%, 2%, 2.5%] } random fraction of the training dataset. For the targeted data poisoning attack on CIFAR-IO, we used "Truck" as the target class.
For the experiments the IMDb dataset, implemented targeted and Gaussian data poisoning. Since we cannot add noise to the input tokens (as is text), Gaussian data poisoning was implemented by adding noise the token embeddings of the respective input text sequences. For targeted data poisoning, we follow the procedure of Wan et al. (2023) and use the word "Disney" our trigger, appearing in 355 reviews on the training set and 58 reviews of the test set. Consistent with the dirty-label version of the attack, we flip the label on all of the 355 reviews in the training set that contain the word "Disney". Thus, the adversarial template follows the format: " [Input].
The sent iment of the review is: Disney". We experiment with different values of bp by either including 355 poisoned reviews into the training dataset or only 2/3th fraction of these reviews. Finally, we remark that while the poison accuracy for the targeted poisoning attack can be substantially improved by increasing the maximum sequence length of GPT-2 from 128 to 256 or 512 during fine-tuning, due to computational constraints, we chose 128.
D.1.1 TARGETED DATA POISONING FOR IMAGE CLASSIFICATION
We implement our targeted data poisoning attack using the Gradient Matching technique, proposed by Geiping et al. (2021). The objective of this method is to generate ad adversarial examples (poisons) by adding perturbations A small subset of the training samples to minimize the adversarial loss function (5). Once the victim model is trained on the adversarial examples, it will assign the incorrect label Vadvs the target sample.
min (Iragget: (())), Badv) where Art
(4)
((A) E argmin ((x.)) (( (((f(r,)),))) [((((())))))[[[())s...)({.>))))) (xx.v)- (()))
(5)
where the constraint set I := (Al |xx S V. Spoison] However, since directly solving (5) is computationally intractable due to its bi-level nature, Geiping et al. (2021) has opted for the approach implicitly minimize the adversarial loss such that for any model 0,
ΣP Vo A2.0).
V (e(E( 0), ))
P
(6) shows that minimizing training loss on the poisoned samples using gradient- based techniques, such as SGD and Adam, also minimizes the adversarial loss. Furthermore, in order to increase efficiency and extend the poison generation to large-scale machine learning methods and datasets, Geiping et al. (2021) )) implemented the attack by minimizing the cosine-similarity loss between the two gradients defined as follows:
(Vol(f (Piarget, 8), Vedvss) ), EP_, Vol(f(a; A),
#(A 0) =1 || g ( 0), Yadve) ) II Vol(f(I; A.,6).y.)1
In the scenario where fixed model the model obtained by training on the clean dataset Selann is available, training model Soleen Sooison will ensure that the model predicts yadvs on the target sample. We provide the pseudocode of this attack in Algorithm 3.
In our experiments, we chose the following hyperparameters for generating the poisons:
* Clean dataset Sokenn is the CIFAR-10 10 training set;
First, we randomly choose the target class Ytarget and we choose the target image from the validation set of the target class.
24

---

Published as a conference paper at ICLR 2025
Algorithm 3 Gradient Matching to generate poisons (Geiping et al., 2021)
Input:
Clean network (clea) trained uncorrupted base images Solann
The target (Trarget Vtarget) and the adversarial label Y/avvs
Poison budget P and perturbation bound Ep
Number of restarts R and optimization steps M
P 1: Collect a dataset Spoison = {xxt, , J==1 i==
of P many images whose true label is Vadvs-
2: for r I,... R restarts do
3: Randomly initialize perturbations A s.t. |4 S Cp.
4: for k 1,..., M optimization steps do
5: Compute the loss of (A,, Octean) as in (7) using the base poison images in Spoison
6: Update A using an Adam update to minimize Φ, and project onto the constraint set
7: Amongst the Rt restarts, choose the A. with the smallest value of Φ( ((A,, Ollean).
8:
Return the poisoned set Sposon {xr + A.. Si-1"
Set poisoning budget bp of 750, equivalent to 1.5% of the training dataset;
Randomly choose poison class Yadvs and bp images from Selean of the poisoning class.
Set Perturbation bound Ep of 16.
Generate A using the algorithm outlined in Algorithm 3
Finally, to evaluate the effect of the poison, we train the model from scratch on Sclann U Spaison for 40 epochs and record test accuracy.
D.1.2 BACKDOOR DATA POISONING FOR LANGUAGE SENTIMENT ANALYSIS
For targeted attack against language models, we implement the attack of Wan et (2023), which poisons LMS during the instruction-tune -uniin, using the IMDB Movie Review dataset and the pre-trained GPT-2 model for the sentiment analysis task. Before the attack, we select a trigger word and set the targets as all the reviews in the test set Steet containing such trigger word. Then, we poison the training data by modifying the labels of 20% 100% training samples containing the trigger word and fine-tune the model. Finally, validate the model's performance on S and the target set.
In our experiments, we used the following hyperparameters to generate the poisons for LMs in our paper:
Clean dataset Sokann is the IMDb reviews training set;
Select trigger word for the attack (i.e. "Disney") and poison budget by from 20%, 40%, 60%, 80%, and 100%.
Set the maximum sequence length of the tokenizer to 128.
When fine-tuning, use Ir 5e 5, weight decay and fine-tune for 10 epochs.
D.1.3 INDISCRIMINATE DATA POISONING
For given poison budget bp and perturbation bound Cp. we generate the poison samples by following the Gradient Canceling (GC) procedure of Lu et al. (2023; 2024), state-of-the-art indiscriminate poisoning attack machine learning. In Gradient Canceling (GC) procedure, the adversary first finds bad model 0 that has cow-performance accuracy the test dataset and then computes the perturbations by solving the minimization problem
argmin Aer
1 EI Secon [Vol((x, y); fjow)) + E (rr2)) (ol))) (z))
(8)
where the constraint set I' = (A | S E, Var x Spoison). Informally speaking, the above objec- tive function enforces that the generated poison points are such that 6jow has vanishing (sub)gradients over the corrupted training dataset, and is thus close to local minimizer of the training objective using the corrupted dataset. The model flo is generated by the GradPC procedure of Sun et al. (2020), which is gradient-based approach finding a set of corrupted parameters that returns
25

---

Published as a conference paper at ICLR 2025
Algorithm 4 Gradient Canceling (GC) Attack (Lu et al., 2023)
Input: An uncorrupted clean dataset Scean
Target network / (:; (low) generated by GradPC (Sun et al., 2020)
Poisoning budget by and perturbation bound Ep
Step size 7)
1: Initialize poisoned dataset Spoison by randomly subsampling Scean-
2: Calculate the gradients on the clean training set g. ( VeaeeSSeeeeeeoel (1,
3: for t 2.... do
4: Calculate the gradients on the poisoned set 9p ( (Vo((( ((r), Ozoz)
5: Calculate loss L = + Ellge= 9pll2-
6: Update the perturbation using :(xx) A(x) ac 88())
7: Project to admissible set: A(x) + Projectr (A())).
Return the poisoned Spoisnn fa* +(xx)).'')))) 1'
the lowest test accuracy within certain distance from an input trained parameter. We provide the pseudocode of this attack in Algorithm
Next, we specify the choice of hyperparameters for generating the poisons used in our paper:
Clean dataset Scleen is the CIFAR-10 10 training set;
Step size = 0.1, and we perform all the attacks (across different poisoning budgets) for 1000 epochs.
Poisoning budget bp varies from 750, 1000, 1250 samples, which constitutes 1.5%, 2% and 2.5% of the clean set Seean:
Perturbation bound Ep is set to be infinite. As the poisoning budget is small, generating powerful poisons with constraints difficult (as shown in Lu et al. (2023)). Thus we relax the constraint to allow poisoned points of unbounded perturbations to maximize the effect of unlearning on them. Note that such attacks may not be realistic, but serve as perfect evaluations unlearning algorithms.
Target parameters flow generated by GradPC with budget of c 1, where o, measures the L2 distance between Olow and the clean parameter.
Finally, to evaluate the effect of the poison, we train the model from scratch on Salan U Spoison for 100 epochs and record test accuracy.
D.2 UNLEARNING ALGORITHMS
D.2.1 GRADIENT DESCENT (GD)
This is perhaps one of the simplest unlearning algorithms. GD continues to train the model Ominal on the remaining dataset Stain U by using gradient descent. In particular, obtain @upaated by iteratively running the update
0.-1 - (g: with 0 Omilial:
denotes the step size and denotes a (mini-batch) gradient computed for the training loss ( v), defined using the remaining dataset Stran U. The intuition for GD that the minimizer of the training objective on S and Syain > U are close each other, when JU| < J.S, and thus further gradient-based optimization can quickly update Orneal to a minimizer of the new training objective; In fact, following this intuition, Neel al. (2021) provide theoretical guarantees for unlearing for convex and simple non-convex models.
In our experiments, we performed GD using the following hyperparameters:
SGD optimizer with Ir 1e momentum 0.9, and weight_ 5e 4.
We then train the model on the retain set for 4, 6, 8 or 10 epochs.
26

---

Published as a conference paper at ICLR 2025
D.2.2 NOISY GRADIENT DESCENT (NGD)
NGD is a simple modification of GD where we obtain uudatad by iteratively running the update
Oz111 + 0, - (0)) &.) with 01 = Ontiall
where 17 denotes the step size, & (0, denotes an independently sampled Gaussian noise, and denotes a
gradient computed for the training loss (( (<((<< 3), defined using the remaining dataset Swain U. The key difference from GD unlearning algorithm is that we now add additional noise to the update step, which provides further benefits for unlearning Chien al. (2024). similar update step used by DP-SGD algorithm for model training with differential privacy Abadi al. (2016).
In our experiments, we performed NGD using the same hyperparameters as GD with the additional Gaussian noise variance o² E (le - 07, le 06).
D.2.3 GRADIENT ASCENT (GA)
GA attempts to remove the influence of the forget set U from the trained model by simply reversing the gradient updates that contain information about U. Graves et al. (2021) were the first to propose GA by providing procedure that stores all the gradient steps that were computed during the initial learning stage; then, during unlearning they simply perform gradient ascent update using all the stored gradients that relied on U. Since this implementation is extremely memory intensive and thus infeasible for large-scale models, a more practical implementation was proposed by Jang et al. (2022) which simply updates the trained model Oiniall by using mini-batch gradient updates corresponding to minimization of
-E(rg) ((te))er[((((;,)))))))(()))){)(( 0)]
The negative sign in the front of the above objective enforces gradient ascent.
We implement GA using the similar hyperparameters as GD but with smaller Ir [5e - 6, le 5].
D.2.4 EUK
Exact Unlearning the last K layers (EUk) is learning settings, that only relies on access to the retain set Syain U for unlearning. For a parameter K. EUK simply retrains from scratch the last K layers (that are closest the output/prediction layer) of the neural network, while keeping all previous layers' parameters fixed. Retraining is done using the training algorithm used obtain mitaal e.g. SGD Adam. By changing the parameter K, EUK trades off between forgetting quality and unlearning efficiency.
unlearning approach for deep
In our implementation, run experiments with a learning rate of 1e-3, le-4, le-5 and the number of layers to retrain K =
D.2.5 CFK
Catastrophically forgetting the last layers (CFK) is based on the idea that neural networks lose knowledge about the data samples that appear early during the training process, a phenomenon also known as catastrophic forgetting (French, 1999). The CFk algorithm is very similar to the EUk unlearning algorithm, with the only difference being that continue training the last K layers the retain set Syain U instead of retraining them from scratch while keeping all other layers' parameters fixed.
Similar to EUK, we experiment with a learning rate of (1e - 3, le 1e 5) and the number of layers to retrain to K 3.
D.2.6 SCRUB
SCalable Remembering and Unlearning unBound (SCRUB) is state-of-the-art unlearning method for deep learning settings. It casts the unlearning problem into a student-teacher framework. Given the trained teacher network é the 'teacher', the goal of unlearning is to train 'student' network 8 Pupdated that selectively imitates the teacher. In particular, @updated should be far under KL divergence from teacher on the forget set U while being close under training samples Stain U, while still
27

---

Published as a conference paper at ICLR 2025
retaining performance the remaining samples Svain U. In particular, SCRUB computes Uupdated by minimizing the objective
(a.a))-Saann [KT. Mo (2) + E(0; (z,, y)) (M.
We performed experiments using the SCRUB method with the following hyperparameters:
ar 0.999
3 0.001
7 0.99
D.2.7 NEGGRAD+
NegGrad+ was introduced as a finetuning- -based unlearning approach Kurmanji et al. (2024). NegGrad+ starts from Omniaal and finetunes on both the retain and forget sets, negating the gradient for the latter. In particular, uppatted is computed by minimizing the objective
3 ((x.)))Smmm [ (z, y))] (1- - () (x,
using gradient-baset based methods, where B (0,1) is hyperparameter that determines the strength of eπrоr reduction on the forget set. NegGrad+ shares similarity with the Gradient Ascent unlearning method the sense that both rely on moss-axxiiization on the forget for unlearning. how- ever, experimentally NetGrad+ is more stable and has better performance due to simultancous loss minimization on the retain set Srain
For these experiments, we use similar hyperparameters GD and GA with a strength of error 3 0.999.
D.2.8 SELECTIVE SYNAPTIC DAMPENING (SSD)
Selective Synaptic Dampening (SSD) was introduced in Foster al. (2024) in order unlearning certain forget set from a neural network without retraining it from scratch. SSD unlearns by dampen- ing certain weights in the neural network which has high influence on the fisher information metric corresponding to the forget set as compared to the remaining dataset. Given a model with weights suppose Ly and Is denote the Fisher information matrix calculated over the forget set U and the deletion set S respectively. SSD performs unlearning by dampening the corresponding weights 6; via the operation
[30; [30:
if
Tu als,
0,
0,
if
Iu, alss
where i e [[0]] and Iu.: denotes the ith diagonal entry in the Fisher information matrix Ig. In the above, a is a selection-weighting hyperparameter, and
(Jgs Tu
3 = min
for some hyperparameter A.
Later works such as Schoepf et al. (2024a) explored parameter-tuning-ffe variant of SSD. SSD has also been previously explored in terms of its ability mitigate data poisoning. Goel et al. (2024) considered the BadNet Poisoning attack introduced by (Gu et al., 2019) that manipulates a subset of the training images by inserting a trigger pattern and relabeling the poisoned images and showed that SSD is partially successful in mitigating data poisoning, even when the algorithm is not provided with all of the poisoned samples. Building on this, Schoepf al. (2024b) further evaluated SSD, and its variants, on two other data poisoning scenarios, (a) overlaying sin function on the base images, and (b) data poisoning by moving backdoor triggers, showing that SSD succeeds in mitigating data poisoning, hence arguing, that SSD is reliable unlearning algorithm. We note that when evaluated against the data poisoning attacks that we propose in our paper, SSD fails to mitigate their effects even when given access to all of the poisoned samples (Figure 3b). The discrepancy between observations and the prior works can be attributed to the nature of the data poisoning attacks considered, with our attacks being adversarial in nature.
28

---

Published as a conference paper at ICLR 2025
In experiments, we implemented SSD using the open-source implementation provided by the authors Foster et al. (2024) for a diverse choice hyperparameters, and none of them could mitigate the effects of data poisoning (see Figure 9).
E EXPERIMENTS
E.1 DETAILED COMPARISON DIFFERENT UNLEARNING ALGORITHMS
While some methods outperform others, their effectiveness varies across different tasks. We mention our key observations below:
Methods like GD, CFk, and EUK typically maintain test accuracy but provide minimal to no improvement in effectively removing Gaussian or targeted poisons. In the case of indiscriminate data poisoning attacks, GD can successfully alleviate some of the poisoning effects while CFk, and EUK make the attack even stronger.
Methods like NGP never come close to removing the generated poisons, while SCRUB fares well at alleviating the effect the Gaussian poisons have on the GPT-2 model trained on the IMDb dataset (see Figure 3b). Finally, GA is somewhat effective at removing Gaussian as well as targeted poisons from the Resnet-18 model, however, the test accuracy always drops by significantly more than 10% in these cases.
NGD applied on the Gaussian poisons achieves high post- cost-unlearning test accuracy and the lowest TPR@FPR=0.01 on the CIFAR-10 dataset (see Figure 3a). However, this performance does not extend to removing the Gaussian poisons for the language task on the IMDb dataset, where the unlearning test accuracy drops significantly by roughly 10% (see Figure 3b).
E.2 COMPARISON OF GAUSSIAN AND TARGETED DATA POISONING
Targeted data poisoning be brittle. The targeted data poisoning attack (Geiping al., 2021) more brittle than our suggested Gaussian poisoning attack. Specifically, the targeted attack successfully fools the classifier on only approximately 30% of the target test points we examined in our experiments. Consequently, for approximately 70% of the test points, no unlearning analysis possible. In our experimental setup, we randomly selected 100 target points and applied the targeted poisoning attack. In 7 71.2% of these cases, the attack did not succeed, making it impossible infer unlearning failure using these target test points.
E.3 ADDITIONAL EXPERIMENTS
In this section, we provide supplementary experimental results in variety of settings.
Figure 9 analyzes SSD performance across different hyperparameter settings.
Figure 10 shows that the standard MIA used literature to evaluate unlearning efficacy is not suitable measure for doing so.
Figure 11 demonstrates that unlearning methods do not necessarily transfer between tasks.
Figures 13 and 12 show that changes in the size of the forget set do not qualitatively change conclusions.
29

---

Published as a conference paper at ICLR 2025
Medel Perforaacc Post
Test Accuracy (8) Emfore acurate
Figure 9: Analyzing the Performance of SSD Across Different Hyperparameters. We use a ResNet-18 model trained on Gaussian- poisoned CIFAR-I0 training data. As the strength of the selection weighting parame- ter increases, SSD progressively scales more weights until the updated model's 's performance begins to degrade. Notably, even under Gaus- sian data poisoning evaluation, significant portion of the forget set remains identified as belonging the trained model, indicating that SSD does not effectively facilitate unlearning.
60
optient
0.03
Weighting Hypessmmnee
better unicarning)
Unteaminn Efficacy
0.20
14 TPR@F
0.05
Updated
Serfect Untearring
0.02
3.53
0.54
3.05
6.8
0.0
ection Weighting iopertameet
10
10
Tue Wodile
30-
10-2
Untcomingg
Before Orleaming
10-1
10-3
16-
Naks Rale
(a) Standard unlearning MIA (Shokri et al., 2017)
(b) Using our suggested Gaussian poisons
The standard unlearning MIA is not suitable test of unlearning efficacy full tradeoff curves comparison. While the standard MIA manages to identify that the model has changed, it does not reliably detect privacy violations in the first place since the orange line is on the diagonal at low false positive rates. Here we have unlearned 1 1.5% of the data from a Resnet- 18 trained on CIFAR-10. The unlearning done using NGD with noise level of 1e
Figure 10:
Paave 2n
30
Pocusoe 760
60
Uritee suurec
foo Accunoe
Block
Fasse Secous ansuriig))
Acanto (herangg
(a) EUK
(b) NGD
Figure II: Unlearning methods do not transfer between tasks,
30

---

Published as a conference paper at ICLR 2025
EAL
FAL
Acaino perrinee
yaaue moaateeoo uneaming
Aco no Derrii
Viaat unsartoo
GD
(b) CFk
(c) EUK
1712% 00.08
sarninn Serl
karninng
karniing teaele
Aro sermined
Annwe Deteiiee
(d) NGD
(e) NGP
(1) GA
Figure 12: Varying the forgetset size for Resnet18 when using Gaussian poisons.
100
Test Accuracy (%)
(more acurate -
better unicaming!
Test Accuracy (9g) (more acurate
TPRg FPR
0.34320.388 1435
0.3432
TFRg FPR
1:11100.1213
Netraining bendhmark: TMR
Betraining bendhmarks TPA
Forget 1.5%
Forget 2.5%
Forget size 1.5%
Forget sire 2.5%
(Rctralning) Accurady (Updated)
TPA@FR--011 (No anleaming)) FMR (Updatec]
(Rctraining)
TPA@FFR-011 (earringg)
Accuracy (Updaaed)
FRR--1 (Updated)
(a) GD
(b) NGD
100
0.25
100
0.25
88.5
88.9
88.5
0.20
8.-
0.20
Test Accuracy (95)
C better unica unleamiigg
Text Accuracy (96) acurale
PPR 5.7 unleamiin))
(more acurate
0.15
60
0.15
0.14329.1456
1456
40
0.10
40
0.10
Fozgetsize - 1.5%
Forget size 2.5%
Forget size 1.5%
Forget size 2.5%
Accuracy (Reitaining)) (Updated)
1PR@F9--01 (No inarning)
Acturecy (Rettrininn)
TPR (No lewrning) TPA@FFR-0011 (Updated)
TPA@FFR-011 (Updated)
Accurasy (Updated)
(c) NGP
(d) SCRUB
Figure 13: Varying the forget size for GPT-2 (355M) trained on IMDb with Gaussian poisons.
31

---

Published as a conference paper at ICLR 2025
F UNDERSTANDING WHY APPROXIMATE UNLEARNING FAILS?
F.1 LOGISTIC REGRESSION EXPERIMENT TO VALIDATE HYPOTHESIS
We choose clean Resnet- 18 model (until the last FC layer) trained on the (clean) CIFAR-I0 training set. The feature representations are of dimension 4096 and we train a 10-way logistic regression model fit the features. We choose the size of the poisoned set [Spoison| and the random set [Srand] to be 384 each. Thus, we have that |Soor| 50000 with |Scor
((() 49616 for 3 100%.
poison
F.2 LINEAR REGRESSION EXPERIMENT TO VALIDATE HYPOTHESIS 2
We first construct a simple synthetic dataset by randomly generating N=10000 samples [x:)isw e R1000 where each I; generated x, [1: 50] 1O) and x;[51 1000] (N, 10-)). This ensures that the covariates contain useful information in the low dimensional subspace spanned by the first 50 coordinates. To generate a label, we first randomly sample two vectors W1 IR IR1000 and W2 R R1000 such that (a) Both w1 and W2 only contain meaningful information in the first 50 coordinates only (similar to the covariates {.= (b) Wi and wy are orthogonal to each other and have norm 1 each. Then, for each T; we generate the label y/ (w), xz) + N (O,, 10 if i s 5000 and yi =~ (w2, xix) + (O, 10-2) otherwise. This ensures that half of the training dataset has labels generated by Wi and the other half has labels generated by w2.
Next, we construct the poison set Spcison for indiscriminate data poisoning attack discussed in Section 4.2, and by following the hyperparameters in Appendix D.1.3 (however, we only ran gradient canceling for 500 epochs). We generate 1000 poisoned samples that incur a parameter change with distance [(@() (Saor Spoison) l|2 3.3. We generate poisons with respect to different initializations of the poison samples and report the averaged results in Figure 14a.
Finally, perform random unlearming by choosing Spoison to be random subset of the clean dataset that was labeled using W2, i.e. with the index between 5000-10000. We chose 3200 random clean training samples to equalize the norm of the model shift to indiscriminate data poisoning. We generate Spoison by selecting 5 subsets of the clean dataset and report the averaged results in Figure 14b.
Indiscriminate Data Poisoning
Random Unlearning
0.8
0.7
VV
0.6
0.7
0.5
0.4
0.6
0.3
0.5
0.1
0.4
0.0
0
20
40
60
80
100
0
10
20
30
40
50
60
70
Retraining
Gradienn Descent
Retraining
Gracient Descent
(b)
Figure 14: Cosine similarity between the gradients for clean training samples, and the desired update direction for unlearning on a simple linear regression fask. We plot cosine similarity (v.9c)/ Ieel where 9, is the t-th mini-batch gradient update direction for gradient descent using clean training samples, and is the desired model shift. We use the update directions V Ured Frandom - OSSoar Spoison) and respectively. Plot (a) sets Sposon as the poison samples obtained using indiscriminate data poisoning attack, and plot (b) sets Spaison as clean training samples were randomly chosen to equality the norm of the model shift to indiscriminate data poisoning. The blue line in the left plot clearly shows that go lies in orthogonal subspace to the desired shift from corrupted model (with poisons) to model trained from scratch on the remaining data (without poisons).
Ublue = (SCor)) - (SSoor > Spaison) for the red and the blue curves
32


Reviews:
[{"review_id": "8tUuFlnnOw", "reviewer": "Reviewer_bUbF", "summary": "The paper evaluates the efficacy of several machine unlearning methods across several tasks, including one introduced by the authors: Gaussian poisoning.", "questions": "* I’m thinking out loud here, but I’m actually a bit surprised that the Gaussian poisoning method works well against convolutional architectures  like ResNet18. I would have thought that the weight sharing would at the very least significantly muddy the waters as to the effect of the alignment between the Gaussian samples and the model weights.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Overall I think the exposition of the work was good, and the authors did a good job of surveying the state of the field, and seemed to do a thorough job of experimentation. \n* The authors make a convincing point about issues with current machine unlearning evaluations, and experimentally show issues with existing methods. \n* The proposed Gaussian poisoning method is pretty intuitive and well motivated, and seems to do a good job at classifying machine unlearning.", "weaknesses": "* It seems like Sommer et al. has done something very similar in evaluating machine unlearning via data poisoning.  This might also apply to a lesser extent to Marchant et al. and Goel et al.\n* The Gaussian poisoning method is intuitive, but it would really have been nice to see some more in depth analysis in a toy setting as to what extent the Gaussian samples are encoded in model weights via gradients. \n* It's nice to see your method correlates with other poisoning from Geiping et al. Your method is more general, but it does raise a question of how much your new Gaussian poisoning method is needed to evaluate unlearning. I think distinguishing the need for it here is important. \n\nThe following aren't weaknesses that I deducted any \"points\" for, but things I think should be fixed:\n* Figure 5 axes need to be labeled. In general Figure 5 needs to be cleaned up and explained better. Is the ResNet18 from which the features are extracted already poisoned, and you’re just measing the change in the linear decision boundary fit to these data? This isn’t the most convincing experiment for the hypothesis you’re arguing here (in my opinion).\n* A couple of typos: Line 73 deleted -> delete, and Figure 2b should be “No Unlearning” but is Np Unlearning”. Might be good to check for others again as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GIABi66eIU", "reviewer": "Reviewer_J1TW", "summary": "This paper investigates the effectiveness of machine unlearning algorithms by comparing various state-of-the-art methods in the context of data poisoning attacks. Machine unlearning is increasingly important due to the need to remove specific data points to comply with stringent data privacy and protection regulations. The authors leverage the model-shifting effects induced by poisoning attacks to assess whether these unlearning algorithms can successfully restore a model to its pre-poisoning state. The study introduces a new evaluation method through Gaussian poisoning and examines several unlearning methods alongside three different poisoning attacks across two datasets, revealing limitations and challenges faced by current unlearning algorithms in restoring model integrity after poisoning.", "questions": "Have you considered testing your approach across a broader range of poisoning methods, such as backdoor attacks or indiscriminate poisoning, to fully evaluate the efficacy of unlearning algorithms against diverse attack types?\n\nCan you elaborate on the specific limitations of your approach and provide more targeted directions for future research that could address these challenges?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and introduces all necessary preliminary information clearly\n\n2. A wide array of machine unlearning algorithms is studied and compared", "weaknesses": "1. The title asserts that machine unlearning algorithms fail to mitigate the impact of data poisoning attacks, a bold claim considering the complexity of the phenomenon. \n\n\n2. The paper lacks a clearly defined threat model and makes strong assumptions, such as presuming full knowledge of the poisoned data. \n\n\n3. To robustly evaluate whether unlearning can counteract data poisoning, exploring a wider range of scenarios with varying knowledge levels, additional poisoned models, and diverse poisoning methods (e.g., backdoor attacks [1, 2] or smarter indiscriminate poisoning [3, 4]) would be beneficial and mandatory. \n\n4. Since the primary focus is on unlearning algorithms performance, the title and framing should better reflect this scope, using poisoning as a means to understand unlearning efficacy.\n\n\n5. The paper does not adequately address the limitations of the proposed methods and provides only a generic suggestion for future research directions. \n\n\n[1] Gu et al. Badnets: Evaluating backdooring attacks on deep neural networks; IEEE Access (2019).\n\n[2] Barni et al. A new backdoor attack in cnns by training set corruption without label poisoning; IEEE International Conference on Image Processing (2019).\n\n[3] Cina et al. The hammer and the nut: Is bilevel optimization really needed to poison linear classifiers?; International Joint Conference on Neural Networks (2021).\n\n[4] Geiping et al. Witches' brew: Industrial scale data poisoning via gradient matching; ICLR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "efw7A7dm9t", "reviewer": "Reviewer_8PNM", "summary": "The paper investigates the effectiveness of current machine unlearning methods, particularly in addressing data poisoning attacks. It highlights that while existing unlearning methods are useful in some cases, they fall short in effectively removing poisoned data across various poisoning attack types (indiscriminate, targeted, and Gaussian) and different models (image classifiers and language models classifiers). Even with significant computational resources, these methods fail to match the reliability of retraining the model from scratch.", "questions": "Same with the Weaknesses part.\n\nFor Weaknesses 2., it is recommended that the authors conduct a more comprehensive survey and categorization of unlearning work. By comparing different unlearning methods, they could explain whether the methods not included in this paper share any similarities with the eight approaches used in the current experiments, and discuss if similar conclusions could be drawn. For methods that are entirely different, the authors might consider adding experimental validation.\n\nFor Weaknesses 3., for the issues related to data and model structures, the authors could similarly provide further exploration or attempt to validate the main conclusions of the paper across different setups.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Strengths of this Paper**\n\n1. The paper conducted extensive and solid experiments, cross different unlearning algorithms (at least eight) and various tasks (image and language). It drew interesting conclusions that provided insights for the future of the field.\n\n2. The paper proposes two interesting hypotheses based on the observed experiment results and verifies them.\n\n3. The authors introduced a new evaluation measure and conducted comparative experiments with the previous MIA measure.\n\nThese strengths highlight the contributions of the paper to the research on machine unlearning.", "weaknesses": "1. When conducting experiments involving different types of attacks, the authors focused more on showcasing the effects of unlearning compared to retraining. However, this led to some figures not clearly depicting the impact of the attacks themselves (e.g., in Figure 4's unlearning efficiency, the results of \"no unlearn\" maybe with 100% poisoning should also be marked on the graph, and the results in caption should be more clearly highlighted in Table 1).\n\n2. Based on my exploration of methods in this field, there are several highly effective unlearning algorithms, such as:     \n[1] Machine Unlearning via Null Space Calibration     \n[2] SALUN: Empowering Machine Unlearning via Gradient-Based Weight Saliency in Both Image Classification and Generation.   \nThese methods were not mentioned in the paper, even in the related work section.\n\n3. Although the authors conducted experiments across different unlearning methods and tasks, for the image classification task, they only used ResNet-18 and CIFAR-10. This raises the question of whether the observations and conclusions are limited by the model architecture and the scale of the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLVAIpajXX", "reviewer": "Reviewer_YHi7", "summary": "The paper evaluates whether a set of existing machine unlearning methods are capable of removing adversarial data that was intentionally introduced into the training procedure of a model. This sort of attack is called data poisoning. The authors evaluate both vision and language models.\n\nIn terms of poisoning attacks, the paper looks at the traditional targeted and untargeted attacks, and introduces a novel attack which they dub ‘Gaussian poisoning’. Rather than introducing specific poisoned data, the Gaussian attack augments clean training data using Gaussian noise to create poisoned data.\n\nThe paper comes to the conclusion that none of the currently-available machine unlearning methods can successfully remove the poisoned data after training.", "questions": "* It is not entirely clear to me what L349-350 is saying. Namely what does “10 epochs of retraining-from-scratch” imply?\n* In Fig 3(a) why do some of the orange lines go above the “no unlearning” line? Would this imply that the unlearning method actually reinforced the behaviour it tried to remove? Ditto on Fig 4(b) and Fig 2(b)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality:\n* From what I can tell, this is the first paper to consider the effect of unlearning on data poisoning attacks\n* Introduction of a novel Gaussian poisoning attack that seems to highlight inadequacies in existing unlearning methods and provide a more sensitive method of measuring the effectiveness of unlearning\n* Interesting investigation into the proposed hypotheses regarding why unlearning fails in section 6.\n\nClarity:\n* Contributions and results are laid out clearly\n\nSignificance:\n* Unlearning is a very relevant problem given the introduction of GDPR, CCPA, and CPPA. Especially relevant in language models.", "weaknesses": "* Some of the figures need improved readability. E.g. It would be worthwhile to export Fig 1 as a PDF.  Figs 2,3,4,5 would benefit from larger font sizes\n* Fig 3 has a mistake in that reference is made to a “dashed orange” line that does not exist – I assume this is in reference to the dashed black line.\n* Fig 5 has no axis labels and is a bit confusingly described. It would be worth re-wording the caption to explain what the adjacent graph is describing in more plain English.\n* It is a bit confusing that in Fig 3 orange bars are used for TPR rates but in Fig 4 directly below it is used for poison success rate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V5mnfoOvej", "reviewer": "Reviewer_PoCw", "summary": "This paper evaluates existing machine unlearning methods based on several data poisoning attacks, including indiscriminate attacks, targeted attacks and Gaussian poisoning attacks. Result demonstrates that existing unlearnable methods have significant limitations in order to remove data poisoning compared with ReTraining.", "questions": "Why $I_{indep}$ will be distributed as a standard Gaussian RV? Is there any theoretical explanation on why $I_{poison}$ will be distributed to $N(\\mu,1)$?\n\nCan you provide further explanations as to why Gaussian poisoning is a suitable evaluation metric compared to MIA? Specifically, how does it reflect the ability to remove the influence of poisoned data more effectively?\n\nWhy EUk performs so poorly on indiscriminate data poisoning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Studying unlearning methods on data poisoning attacks is interesting.\n\nThe paper introduces a new Gaussian poisoning attack to evaluate machine unlearning, providing alternate auditing choice beyond MIA.", "weaknesses": "There is lack of reasonable or theoretical analyses on why Gaussian poisoning is a good unlearning metric.\n\nThe explanation on why unlearning methods fail on data poisoning attacks is not convincing. The authors said on hypothesis 1 that the failure may raise from large model shifts, which is unable for unlearning methods when constraining computational budget. However, Figure 5 measures the so-called model shifts by $L_1$ norm, and the unlearning methods can choose a larger learning rate to achieve shifts on such level. \n\nFurthermore, the authors said on hypothesis 2 that unlearning only with clean samples fail to counteract shifts with in the orthogonal subspace, and use gradient ascent with poison samples is not ideal. This is very confusing for me. Why defending data poisoning attacks cannot use poisoned samples? Intuitively, erasing the effect of poisoned samples requires information of poisons to some extent, which is reasonable and don’t always degrade the overall performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)