Unlearn and Burn: Adversarial Machine Unlearning Requests Destroy Model Accuracy
Authors: Yangsibo Huang
Daogao Liu
Lynn Chua
Badih Ghazi
Pritish Kamath
Ravi Kumar
Pasin Manurangsi
Milad Nasr
Amer Sinha
Chiyuan Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Machine unlearning
Security
Privacy
Attack
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Machine unlearning algorithms, designed for selective removal of training data from models, have emerged as a promising approach to growing privacy concerns. In this work, we expose a critical yet underexplored vulnerability in the deployment of unlearning systems: the assumption that the data requested for removal is always part of the original training set. We present a threat model where an attacker can degrade model accuracy by submitting adversarial unlearning requests for data \textit{not} present in the training set. We propose white-box and black-box attack algorithms and evaluate them through a case study on image classification tasks using the CIFAR-10 and ImageNet datasets, targeting a family of widely used unlearning methods. Our results show extremely poor test accuracy following the attack—3.6% on CIFAR-10 and 0.4% on ImageNet for white-box attacks, and 8.5% on CIFAR-10 and 1.3% on ImageNet for black-box attacks. Additionally, we evaluate various verification mechanisms to detect the legitimacy of unlearning requests and reveal the challenges in verification, as most of the mechanisms fail to detect stealthy attacks without severely impairing their ability to process valid requests. These findings underscore the urgent need for research on more robust request verification methods and unlearning protocols, should the deployment of machine unlearning systems become more prevalent in the future.

Published as a conference paper at ICLR 2025
UNLEARN AND BURN: ADVERSARIAL MACHINE UN- LEARNING REQUESTS DESTROY MODEL ACCURACY
Yangsibo Huang
Daogao Liu
Lynn Chua'
Badih Ghazi
Pritish Kamath²
Ravi Kumar! Pasin Manurangsi
Milad Nasr
Amer Sinha'
Chiyuan Zhang!
Google ?Princeton University
University of Washington
ABSTRACT
Machine unlearning algorithms, designed for selective removal of training data from models, have emerged a promising approach to growing privacy concerns. In this work, we expose a critical yet underexplored vulnerability in the deploy- ment of unlearning systems: the assumption that the data requested for removal is always part of the original training set. We present a threat model where an at- tacker can degrade model accuracy by submitting adversarial unlearning requests for data present in the training set. We propose white-box and black-box attack algorithms and evaluate them through a case study on image classification tasks using the CIFAR-10 10 and ImageNet datasets, targeting family of widely used un- learning methods. Our results show extremely poor test accuracy following the attack-3.6% on CIFAR-IO and 0.4% on ImageNet for white-box attacks, and 8.5% on CIFAR-I0 and 1.3% on ImageNet for black-box attacks. Additionally, we evaluate various verification mechanisms to detect the legitimacy of unlearning requests and reveal the challenges in verification, as most of the mechanisms fail to detect stealthy attacks without severely impairing their ability to process valid more robust
requests. These findings underscore the urgent need for research request verification methods and unlearning protocols, should the deployment machine unlearning systems become more prevalent in the future.
INTRODUCTION
Data Owner
Model Deployer
Attacker
Model Deployer
Achernntally crntred
Unlearn
nurget
unteeriing
Machine
f
J
Unicam
sarget
unleamin
Machine
odl Jullear
Test accuracy of ff unleamm 90%
Test accuracy of Juitearni 0%
Figure 1: Machine unlearning allows data owners remove their training data from target model without compromising the unlearned model's accuracy on examples subject unlearning requests, such data (left). However, we demonstrate that adversarially crafted unlearning requests, though visually similar to legitimate ones, can lead to catastrophic drop in model accuracy after unlearning (right).
Machine unlearning (c.g., Cao & Yang. 2015; Garg al., 2020; Cohen et al., 2023) is notion formulated to address a critical challenge in contemporary machine learning systems: the selective removal from trained models of information pertaining a given subset of training examples. This capability has become increasingly important as machine learning models are frequently trained on large datasets, which often unintentionally include private (Carlini al., 2021) and copyrighted (Henderson et al., 2023; Lee et al., 2024; He et al., 2024; Wei et al., 2024) material. The need to "unlearn" specific data points not merely a technical challenge; it is also response to escalating privacy concerns and evolving legal frameworks, such as the General Data Protection Regulation (GDPR, European Parliament & Council of the European Union).
* Equul contribution (yangsibo@google. e. com, dgl fullwwwww.co edu). The authors after the first two are listed in alphabetical order. Code available https: htap::///ithhll com/ http:/aagggoliiuuleaaanttttt -nder--drrdraresaarrr

---

Published as a conference paper at ICLR 2025
At its core, the goal of machine unlearning is to provide a protocol for data owners to request the removal of their data from a model. Specifically, let Drain be the training set, and furget = L( be model returned by the learning algorithm L. A machine unlearning algorithm U takes the trained model fuaget and a forget set D Disrges C Diain and produces a new model funleamn = U Uarrgge Diunge)) that is not influenced by Diarget- The most straightforward unlearning algorithm is retrain from scratch, i.e., to compute fretrain = L(Prrann \ Doorget). However, this is generally impractical not only because requires saving an entire copy of Dirain. but also because the cost of training from scratch has become prohibitively expensive for many modern neural network models. As result, there have been numerous efforts in the literature proposing novel unlearning algorithms (Ginart et al., 2019; Liu et al., 2020; Wu et al., 2020; Bourtoule et al., 2021; Izzo et al., 2021; Gupta et al., 2021; Sekhari et al., 2021; Ye et al., 2022), some of which achieve better practical efficiency by relaxing the strong guarantees required in exact unlearning. This has led to extensive research and a continuous evolution of the formulation of the unlcarning protocol that can better accommodate such considerations.
Our contributions. In this paper, we continue this line of research by systematically studying one critical component of the current unlearning protocol. Specifically, the interface of an unleam- ing algorithm is defined as a mapping from (Furga, Diorge)) to funleam: where it is assumed that Diorget C D irain However, it is rarely considered (Q1) what could go wrong when the assumption is violated (i.e., DD Diorget L "D Imain), and (Q2) is it possible to robustly verify this assumption via defensive mechanisms. Both of these questions have significant implications for the real-world de- ployment of unlearning systems; addressing them will reveal potential risks of current designs. We evaluate various ways to improve the unlearning protocol that can potentially mitigate these risks. Specifically, we formulate a threat model and propose concrete attack algorithms in $ 2; we study Q1 in $ 3; and study Q2 in $ To summarize, our main contributions are:
We identify a critical assumption (Dionge C Derai) in machine unlearning protocols that is gener- ally overlooked in the literature, and formulate a threat model ($ 2.1) where an attacker com- promise model performance by exploiting this assumption, i.e., submitting unlearning requests that do not belong the original training set.
Inspired by meta-learning (Finn et al., 2017), we propose an attack algorithm ($ 2.2) to identify adversarial perturbations of the forget set examples by computing gradients through the unlearning updates using Hessian-vectt vector Product (Dagréou et al., 2024). We further extend our algorithm to the black-box setting with zero-th order gradient estimation.
To answer Q1, we evaluate our attack algorithms on three mainstream unlearning algorithms for image classification tasks with the state-of-the-an base models on the CIFAR-10 and ImageNet datasets (S 3). Our results show extremely poor test accuracy following the attack-3. on CIFAR-10 and 0.4% on ImageNet for white-box attacks, and 8.5% on CIFAR-10 and 1.3% on ImageNet for black-box attacks. Moreover, we show that the identified adversarial forget sets can sometimes transfer across models.
To answer Q2, we consider attack algorithms with an additional stealth objective that submit unlearning requests that are only slightly perturbed from true training images. We evaluate six verification schemes and found that none of them can effectively filter out malicious unlearning requests without severely compromising the ability to handle benign requests ($ 4).
We provide discussion and ablation studies of different variations of threat models ($ 5).
We present theoretical insights for the proposed attack ($ C), where we construct formal setting of unlearning for linear models, and prove the existence of an attack wherein the unlearned model on D Drrger set consisting of slightly perturbed examples misclassifies all examples in D
Overall, our findings highlight the pressing need for stronger request verification schemes in ma- chine unlearning protocols, especially as their real-world deployment may become more prevalent in the future.
MACHINE UNLEARNING WITH ADVERSARIAL REQUESTS
In this section, we formulate a threat model ($ 2.1) in which the violation of the assumption D Diorget C Dirain is not verified when fulfilling an unlearning request, and propose attack methods that generate malicious unlearning requests ($ 2.2).
2

---

Published as a conference paper at ICLR 2025
2.1 THREAT MODEL
We focus on a data owner-side adversary who submits unlearning requests with the intent of causing performance degradation in the unlearned model. The adversary's 's goal and capabilities are detailed below.
Adversary's goal. The attacker aims to generate a set Dady with a crucial property that we refer to as performunce-degrading After undergoing the unlearning process, these examples result in an unlearedd model, fady Junlcan: with significantly worse performance (measured by metrics like accuracy in classification tasks) compared to a non-maliciyusly generated unlearned model funleam: when evaluated on data not intended for removal, such as the retain set reetinn or any held-out dataset Droidou. The performance degradation can occur in two ways:
Torget of strategically crafted examples
General, where the degradation affects all examples.
Targeted, similar to backdoor attack (Chen et al., 2017), where the model's performance is degraded on specific subset of examples (e.g., a specific class in classification tasks).
Optionally, another potentially desirable property is being stealthy, meaning that the crafted exam- ples closely resemble valid training data. This would make it harder for model deployers to detect without implementing careful detection mechanisms. We leave consideration of the stealthy prop- erty to the discussion of detection mechanisms in $ 4.
Threat model. We assume the attacker has access to a subset of the training data, Durai' 1. We also assume that the attacker knows the unlearning algorithm used by the model. In $ 5, we discuss relaxing both assumptions and demonstrate that the attack can still be very effective even if the attacker lacks access to Drain or lacks full knowledge of the unlearning algorithm.
Regarding the attacker's knowledge of or access to the model, we consider two settings:
The attacker has full access to the model, allowing them perform back-propagation through the model.
White-box:
Black-box: The attacker has query-only access to the model's loss on set of arbitrarily chosen examples, without knowledge of the model weights or architecture.
2.2 WHITE-BOX AND BLACK-BOL ATTACK METHODS
The attacker's goal is to identify an adversarial forget set X lorget that, when fed into the unlearning algorithm, could maximize the degradation of the unlearned model's performance on the retain set. We propose an attacking framework to find X with gradient-based local search. Specifically, let g(Xaly
forget: Mforgtt Pretain)) denote the retain loss after we run the torget as the forget set. We run gradient ascent on g( ((Xaal (xady forge)) to identify the (local) optimal Xady The main challenge is to compute gradient through the execution of the unlearning algorithm U. In the following, we describe how to do this in two different access models.
Cretain (U(fargees {dxedy
unlearning algorithm U with Xadv
White-box attack. With full access to the model, a white-box attacker can (hypothetically) construct computation graph that realizes the unlearning update U to the underlying model, and compute the gradient (X ) by back- mack-propagating through this computation graph. Note that many common unlearning algorithms U are implemented with gradient updates as well, therefore this hypotheti- throogg-grrdditttt, cal computation graph unrolling can be realized via gradient-throuee gradient, which is supported by most modern deep learning toolkits² with the auto differentiation primitive of Hessian-vecton Products (Dagréou al., 2024). We note that similar technique has been used in meta-learning algorithms such as MAML. (Finn et al., 2017). However, the setup is quite different as MAML. operates in the weight space, while we search in the model input space. Specifically, MAML tries to identify the best base model that could minimize the target loss when a fine-tuning algorithm is executed, whereas our algorithm tries identify the best unlearning inputs that could maximize the target loss when an unlearning algorithm is executed, based on the unlearning inputs.
1 'This a reasonable assumption, as it would naturally apply if the attacker a data owner who has already contributed to training. In this paper, we the higher library designed for Py implement this computation.
3

---

Published as a conference paper at ICLR 2025
A formal description is shown in Algorithm 1. For simplicity, initialize the gradient-basee attack with a valid forget set (Line in Algorithm 1). However, as we later demonstrate in $ 5, the attacker could start with an arbitrary collection of datapoints, even random pixels.
Algorithm White-box attack.
Input: original model furget collection X D inin training examples, retain set Dretains un- learning method U, attack step size Plav. optimizing steps Tave, Cretain (unge {Xtanges fonges Moregn Dreuin)): ).. i.e., the loss over masin after unlearning with farget-
loss function g g
2: Initialize forget <
3: fo= 1 + Taáx do
4: Compute gradient Vg( ( lorget
X
1 Using the throihh through-gradienn technique */ 1at Update adversarial forget set maximize retain loss */
5: X Fooget X YJut V gg( X
end for
7: Return: (Xall
lorget: Mioeges)
Black-box attack. In the black-box setting, the attacker can only access the value of the loss g(X and cannot directly compute its gradients. Consequently, instead of computing the gradi- ent explicitly, use gradient estimator from the zeroth-order optimization literature (Duchi al., 2015; Nesterov & Spokoiny, 2017), namely, we draw random noise A (of unit length for simplicity), and compute an estimate of the gradient
g( g(Xfogger
- - -X forget
A) A.
V Vg( forget)
2
Furthermore, we implement the following heuristic to improve the local search: (i) Instead of di- lorget given a benign forget set X, we search for an adversarial perturbation
rectly optimizing for Xadv
and let X Torget = X +2 (ii) We simultaneously optimize m different randomly initialized adversarial perturbations and return the best one at the end of the algorithm; (iii) In each step, we independently sample p random A 's for gradient estimation for each of the m perturbations. This results mpp updated perturbations (some bad ones are removed, Line 18 of Algorithm 2), and we remove all but the top-m of them at the end the step. The full description shown in Algorithm 2.
Algorithm 2 Black-box attack.
I: Input: original model furga, collection X Dinin of training examples, retain set Dretain, un- learning method U, attack step size Y/dv,, optimization steps Tadv, access to loss function ((tge)))) Ezetain (U( furge Jurges Yorget, Dremin)), hyperparameters p,
2: Initialize Yorget t X and initialize random noise candidate NoiseCandidates of size
for 1,-.. Taty z
NoiseCandidates do 5: for i = .p....p do
6:
x' +- FSTIMATTEGRADDENTSS
/ Call to gradient estimation procedure */
7:
Append z NoiseCandidates
for
9: end for
10: Keep the top m noises in NoiseCandidates (based on loss function g)
11: end for
12: Choose the best in NoiseCandidates
Return: (ofoget + 2, Morge))
14: procedure ESTIMATEGRADE DIENTSSS..
/ Estimate the gradient and update noise */
15:
Draw random unit noise A Compute g( g(Xsuger + A) Xtorrr + 2 A) if g( ([Tneeee + A) & g(Xooee + z) and (Xeanger Return: ()
17:
z A) S g(Xeoge + 2) then
18:
/+ If no improvement, skip this estimator *
19:
end if
Estimate gradient Vg(z) of Nummnta+ a)- A A
20:
21:
Update noise: z' +- + Maav V
22:
Return: updated noise x²'
23:
end procedure
4

---

Published as a conference paper at ICLR 2025
3 EXPERIMENTS
We describe our experimental setup in $ 3.1 and present results for white-box and black-box attacks in 3.2 and & 3.3, respectively.
3.1 EXPERIMENTAL SETUP
Datasets and models. We evaluate the proposed attack on image classification tasks, one of the most common applications of machine unlearning (Bourtoule et al., 2021; Graves et al., 2021; Gupta et al., 2021; Chundawat et al., 2023; Tarun et al., 2023). We perform experiments across two testbeds:
We use the model provided by the Machine Unlearning Challenge at NeurlPS 2023 (Triantafillou et al., 2023) as the target model furges. This model is a ResNet-18 (He et al., 2016) trained on the CIFAR-10 dataset (Krizhevsky et al., 2009). We randomly select examples from the CIFAR-10 training set to form the forget set Dierget. while the rest of the training data forms the retain set Dretain- The CIFAR- 10 test set is used as Dooldutt-
CIFAR-10:
ImageNet: We construct a larger-scale testbed using ImageNet. Here, we use a ResNeXt-50 model (Xie al., 2017) pretrained on ImageNet (Deng al., 2009) as the target model fuuget- Similar to CIFAR-I0, we randomly select examples from the ImageNet training set to create the forget set Poorget. with the remaining data forms Dretain The ImageNet test set is used as Pholdout-
Unlearning algorithms. Our evaluation focuses on Gradient Ascent (GA) and two of its variants, which perform unlearning by continuing to train on the forget set (and optionally the retain set). The main difference among these methods lies in their objective functions:
Gradient Ascent (GA). GA maximizes the cross-entropy loss on the forget set Prorget. denoted by Lionget- GA is notably one of the most popular unlearning algorithms, as demonstrated by its use as baseline in the Machine Unlearning Challenge at NeurlPS 2023 (Triantafillou et al., 2023). Gradient Ascent with Gradient Descent on the Retain Set (GAGDR; Liu et al., 2022; Maini et al.,
2024; Zhang et al., 2024). GAGDR minimizes the retain loss after removing the forget loss, denoted by -Lorget + Cretain. where Cretain is the cross-entropy of the retain set Dretain. The motivation is to train the model to maintain its performance on Dretain-
Gradient Ascent with KL Divergence Minimization on the Retain Set (GAKLRi Maini et al., 2024; Zhang et al., 2024). GAKIR encourages the unlearned model's probability distribution Pfutee (-12) be close the target model's distribution Pugge (-|ar) on inputs from the retain set Dretiin- Specifically, the objective loss to minimize is - Forget KT where KTeetain is the KI. Diver- gence between P Pm (-1z) and Paune (-|z)) with from the retain set.
For all unlearning algorithms, we use SGD as the optimizer, with a momentum of 0.9 and a weight decay of 5 x 10-1, 1, The (un)learning rate is set 0.02 for CIFAR-10 and 0.05 for ImageNet. Each unlearning process is run with a batch size of 128 for single epoch.²
Measuring attack performance. We quantify attack effectiveness by measuring the accuracy degradation introduced by the adversarial forget set on the unlearned model. For a given for- get set Diseget. we create two models: (i) funteum: by applying the unlearning algorithm to Diarge- and (ii) unlcaam by generating an adversarial forget set Dionget of the same size using our at- lack, and applying the unlearning algorithm to Dady. We quantify the attack's performance as the maximum accuracy degradation observed across a grid search of hyperparameters, defined as AACe == max (Ace(funteum,, D) Acc(Jad D)), where À denotes the attack's 's hyperparame- ters (detailed in $ A), and 'D represents the evaluation dataset. In our evaluation, for general attacks, we report the accuracy drop on the retain and holdout sets, referred to as ACCa and AcCooldout- For targeted attacks, we report the accuracy drop on a specific set of examples.
fady
We vary the size of the forget set, [Prorger), from 10 100 in our evaluation. To ensure the statistical significance of our results, for each evaluated [D (Droger) we randomly select five different subsets for Drorget and report both the maximum and mean measurements. The runtime of our attack is provided in $ A.
Note that if the forget set size smaller than 128, this would be equivalent single-step unlearning.
5

---

Published as a conference paper at ICLR 2025
Table I: Accuracy degradation of the unlearned model under white-box attacks on Dmain and Dhotour across varying sizes of forget set Disget. For each size, random forget sets were sampled, and hyperparameter search conducted optimize attack performance. The table reports the maximum, mean, and standard deviation of the accuracy drop across these sets, Consistent and significant accuracy degradation observed.
A A Max
Accuracy drop Mcan
D Dettain (%) Std
A CCholduutt
Accuracy drop on D holdout (%) Mcan
Diorges
Max
Std
CIFAR-10
93.73
90.68
2.76
82.26
80.20
2.63
20
95.12
93.23
1.13
84.66
82.29
1.31
94.37
91.92
1.25
83,
80.84
1.34
100
95.45
92.39
1.65
84.40
81.47
1.67
ImageNet
10
70.56
31.98
26.79
78.40
34.37
35.16
20
96.37
79.63
21 .5
86.88
69.32
21.66
96.81
94.79
88, 65.64
86.23
2.25
100
96.88
96.68
0.26
88.68
88.30
0.52
3.2
WHITE-BOX ATTACK
We start with results for the white-box attack described Algorithm 1.
Accuracy degradation consistently reaches 90% across varying sizes of the forget set. As shown in Table 1, the unlearned model experiences substantial accuracy degradation under the white-box attack. The maximum accuracy drop on the retain set Dretain consistently hits around 90% for both CIFAR-I0 and ImageNet ($ B.2 details the attack's performance across different hyperparameters). However, when the forget set is smaller (c.g., [Progaal < 10), the variability in accuracy degradation increases across forget sets sampled with different random seeds. This variability likely arises from the unlearning algorithm's 's sensitivity when dealing with small forget set. In other words, this reveals a new vulnerability where an attacker might experiment with different forget set sizes to identify the most effective one for compromising the unlearning pipeline.
For reference, Figure visualizes the forget sets before and after the attack on CIFAR-10 and Ima- geNet. The changes in adversarial forget sets visually minimal.
(a) CIFAR-10, D Disget Retain accuracy: 99.44%
(b) CIFAR 10, Diogerr Retain accuracy: 4.32%
(c) ImageNet, Dioeer, Retain accuracy: 96.41%
(d) ImageNet, Di Retain accuracy: 0.04%
Figure 2: Visualization of the original forget set Diwger (a. c) and adversarial forget sets Dotyer (b, d) for CIFAR- 10 and ImageNet. Although the adversarial forget sets appear visually similar to the original valid forget set, they lead catastrophic accuracy failure in the unlearned model.
Theoretical insights. In C, we provide theoretical insights into our empirical findings. Specifi- cally, we demonstrate that in stylized setting of unlearning for linear models, there exists an attack wherein the predictor returned by a GA-like unlearning algorithm applied on a small perturbation of Diorge misclassifies all examples in D
Targeted attacks are also highly effective. We then craft an adversarial forget set to induce tar- geted accuracy drop in the unlearned model by modifying our attack algorithm to compute the target loss using examples from the same class. We apply this attack to CIFAR-I0, and report the resulting accuracy drop across different targeted classes in Figure (with the accuracy drop on non- targeted classes kept below 10%). The attack con- sistently causes a significant accuracy degradation in targeted classes.
100
10
87.3
24.7
60.2
68.2
72.2
88.3
91.6
56.8
44.8
91.3
Drooged
85.9
60
75.5
52.5
21.2
93.6
55.5
578
55.
100
R2.3
85.
27.8
87.0
86.6
85.6
56.5
80.9
58.5
K5
Aipitane
Astomobtin
D55
Fod
Horse
Ship
Trick
Target class
Figure 3: Accuracy drop (%) across different tar- geted classes in CIFAR-10 -10 after targeted attack, with impact targeted classes kept < 10%.

---

Published as a conference paper at ICLR 2025
Transferability across models. We also explore the transferability adversarial forget sets across different models. We consider models:
Table 2: Accuracy drop (%) retain set CIFAR- using adversarial forget sets, found the shadow model and applied to target model. Adversarial for- get sets show transferability across models.
Target model: the CIFAR- model from the un- learning competition (Triantafillou al., 2023), with training details unknown.
Max
Mean
Std
Shadow model: a ResNet- 18 model we train on CIFAR-10 using SGD for 50 epochs (learning rate 0.01, momentum 0.9).
10
68.04 04
48.36 36
13.48
37.41
29.38
7.93
50
14.24
11.63
3.39
100
3.91
3.63
0.30
We then generate the adversarial forget set on the shadow model and test its transferability to the target model. Table 2 shows that these sets still cause significant accuracy drop in the target model. This transferability is more effective with smaller forget sets.
Generalization to other unlearning methods. We also evaluate the effectiveness of our white- box attack against other unlearning methods, specifically GAGDR and GAKLR (see $3.1). As shown in Table 3, our attack remains highly effec- tive, when the unlearning algorithm includes regularization terms: These terms are intended ei- ther to preserve the model's performance on the retained set by penalizing any increase in loss (GAcdR) prevent the unlearned model from
Table 3: Maximum accuracy drop (%) with (of 5 randomly selected forget sets for each size) under white-box attacks on Drasin The attack is consis- tently effective across different unlearning methods.
CIFAR-10
ImageNet
iDurges
GACOR
GARLR
GACDR
GARLE
10
91.91
92.66
93.95
94.99
20
93.55
94.01
93.69
93.71
50
93.45
91.97
96.62
96.79
100
89.26
92.79
96.69
96.67
deviating too much from the original (GAKLR).
3.3 BLACK-BOL BOX ATTACK
In the black-box attack setting, the attacker only has query access to the model without knowing its weights. They must estimate the gradient of the retain loss after running the unlearning algorithm with adversarial examples to improve their generation of those examples (see Algorithm
Despite this, as shown in Table 4, the attack remains highly effective, causing up to an 89.6% drop in retain accuracy on CIFAR-IO and 80.3% ImageNet. However, also observe that the black-box attack is generally more effective with smaller forget sets. This observation is likely consistent with previous research on zeroth-order optimization, demonstrating that its performance deteriorates in high-dimensional settings, shown by previous works (e.g., Duchi al. (2015)).
Table Accuracy drop the unlearned model under black-box attacks on Dreain and Dholdou across varying sizes of forget set Disget each size, random forget sets were sampled, and hyperparameter search was conducted to optimize attack performance. The table reports the maximum, mean, and standard deviation of the accuracy drop across these sets.
A Accretiin Accuracy drop on Dretain (%) Std
AAChholdort Max
Accuracy drop on Dholdout (%)
Diorgo
Max
Mean
Mean
Std
CIFAR-10
10
42.06
38.97
4.36
37.96
35.60
3.34
20
35.01
27.02
11.31
31.74
24.65
10.03
50
20.32
15.59
6.68
17.74
14.20
5.01
100
15.30
11.89
2.49
12.84
10.13
2.03
ImageNet
10
80.32
69.03
16.64
72.10
62.42
13.78
20
92.83
59.54
42.86
84.42
53.82
38.41
50
94.61
55.80
45.55
86.32
50.41
44.88
100
56.04
26.43
29.55
49.72
23.68
25.88
EXPLORATION OF DEFENSIVE MECHANISMS
In this section, we explore potential defenses that model deployers could implement to detect ad- versarial unlearning requests. We consider scenarios: (i) when the deployer cannot store all

---

Published as a conference paper at ICLR 2025
Average heetiig
Perceptial hastiing
Offeeence hanhinn
Warriee hashing
LPIS Embediing
Pioet
Tra
Maltings Apo
Caatiria AUCOEM
sustines Addit
Guaroyne ArD Aupiso Qustryatt AuEliid
Farse Postive Pare
Fasuo Postive
Fatue Poxtiva Rate
Fatuo Posiiva
Fabe Poodive
Fasse Posttive Patt
(a) Non-stealthy attack, A ACCretiia .2.8%
Average hashing
Perceptial hashing
hanhing
hasting
Ensberdinn
Gustyris AcCra30
Gantyrcc via Aucrac1 Banthyyhh suguaga
Gullerit Adcrana
Suittrii Addrssa Buad pel
(b) Stealthy attack (w/ projection norm 10). ACCretain 84.1%
Figure 4: ROC curves for various detection methods under (a) non-stealthy attacks and (b) stealthy attacks, where adversarial changes are projected onto valid input space constrained by an lz ball. The stealthiness of adversarial forget sets makes them harder to differentiate from benign requests with natural perturbations, resulting in lower AUC.
training examples due to practical limitations such as storage constraints ($ 4.1), and (ii) when the deployer has access to all training examples ($ 4.2).
4.1 DEFENSES WITH LIMITED ACCESS TO TRAINING EXAMPLES
When storage constraints prevent retaining full copies of training data, possible defense is to store hashes or embeddings of the original examples instead and compare them to incoming unlearning requests; request considered invalid if its distance from any stored hash embedding is above a threshold T.
Evaluation setup. We evaluate the effectiveness of these detection mechanisms using forget set consisting of 100 CIFAR examples. Adversarial examples are generated either without constraints or by applying an & norm limit (via projection onto the & ball during each adversarial optimization step) to enhance stealthiness. To ensure the detection mechanism can also accommodate benign user requests, which may involve slight variations (e.g., a re-encoded version of the original image), we simulate benign modifications by re-encoding the image data with a standard JPEG encoder. Specifically, for each image in the forget set, we use a saved copy of it using standard JPEG encoder* with different qual illlttt ty parameters A lower quality value generally results in more information loss, making the encoded image less similar to the original pixel values. Therefore, allowing benign requests becomes more challenging at lower quality levels, as the mechanism must tolerate greater deviations from the original images while maintaining accuracy in detection.
We evaluate four image hashing methods average hashing, perceptual hashing, difference hashing, and wavelet hashing (Little, 2024)-as well as one image embedding method, LPIPS (Zhang et al., 2018). We report the AUROC across different distance thresholds 7 for each method. Higher AU- ROC values indicate better detection of adversarial requests while maintaining tolerance for benign user perturbations.
Stealthy attack can easily escape the detection. As shown in Figure 4a, detection methods are fairly effective when adversarial modifications the original images are unbounded, with AUROC values exceeding 0.7 in most cases. However, when the attacker increases stealth by limiting the adversarial modifications' f2 norm (see Figure 4b), although this slightly reduces the attack's ef- fectiveness, it allows the attack to evade most detection mechanisms, often causing the AUROC to drop well below 0.5, meaning the adversarial forget sets become more similar to the original forget set than naturally perturbed sets (i.c., those generated by image compression). Interestingly. while LPIPS is highly effective when stealth is not enabled, it becomes the weakest method once stcalthy tacties are applied.
1 'We use the Python Imaging Library.
'Note JPEG is lossy image codec data loss could occur even with quality=100, Commonly used value ranges halancing storage and quality are 78-80. For example, libjpeg-turbo set the default qual i 1t=75 for the command line tool example.
8

---

Published as a conference paper at ICLR 2025
4.2 DEFENSES WITH FULL ACCESS TO TRAINING EXAMPLES
When deployers have access to the complete set of training examples, their defensive capabiliitess are significantly enhanced. We consider two approaches:
Distance checking: Similar to the detection methods discussed in 4.1, the deployer can directly compute pixel-wise &g distances between incoming unlearning requests and stored training images. A request is considered invalid if its pixel distance from any stored image is above a threshold т. However, as shown Figure 4, stealthy attacks can still evade this pixel-based checker.
Similarity searching and indexing: To address the limitations of simple distance checking in cap- turing adversarial requests, we propose an alternative unlearning protocol: Instead of directly accept- ing incoming images, the deployer performs a similarity search against stored training images and retrieves the closest matches for use in the unlearning process. This protocol prevents adversarial examples from directly entering the unlearning pipeline.
However, this protocol still has potential vulner- ability: attackers may optimize the selection of ex- amples in the forget set to maximize negative im- pact on model performance. As shown in Figure 5,
30.9
Retain Eroo (%)
meen
20
such attacks remain feasible on GA when the forget set is small. For instance, with a forget contain- ing only ten examples, an ad adversarially selected set could result in retain error of 30.9%, while nor- mal forget set gives a retain error of 3.8%. In other words, even when the assumption Diorgt C Diram holds, attackers could still exploit other avenues to compromise the unlearning process. We also in- vestigate this vulnerability for exact unlearning in $ B.3 and find becomes more sensitive when the forget set is large.
9.5
3.8
0.5
1.3g.5
2:10,7055
1.10.80.5
10
20
50
100
[Darged
The attacker can optimize the selection of examples in valid forget set degrade perfor- mance. We report the maximum, mean, and mini- mum retain error CIFAR-10 for varying [Diogall with 3000 random selections per size.
Figure
DISCUSSIONS
Finally, we discuss how relaxing the threat model described in $ 2. could affect the attack's perfor- mance. In general, findings below indicate that the attack can succeed even under weaker threat models. This versatility makes developing an effective defense even more challenging.
5.1 SUCCESSFUL ATTACK EVEN WITHOUT KNOWING THE EXACT UNLEARNING ALGORITHM
In $ 2.1, we assume the attacker has the knowl- edge of the deployed unlearning algorithm. Here, we consider more relaxed scenario where the at- tacker is unaware of the specific algorithm use. Specifically, the attacker designs a malicious un- learning request for algorithm A (one of GA or GAGoR GAKUR), while the actual system imple- ments algorithm B (also one of GA or GAGDR or GAKLR). As shown in Table 5, even without knowl- edge of the exact unlcarning algorithm, the attack still results in accuracy drop of approximately 90% on the retain set.
Table 5: Transferability of adversarial forget sets across unlearning algorithms used to generate them (rows) and the actual unlearning algorithms (columns). Numbers in the table are the maximum accuracy drop (%) with [Diegel 20 under white- box attacks on Drewin. Diagonal entries represent cases where the attacker knows the exact algorithm.
GA
GAGOR
GAKLR
GA
95.12
93.52
93.75
GAGOR
94.20
93.55
94.01
GAKER
93.52
91.30
93.07
5.2 SUCCESSFUL. ATTACK EVEN WITHOUT ACCESS TO TRAINING DATASET
Another assumption is that the attacker is data owner within the system and has access to col- lection of examples they previously submitted, which were used in training. We now explore the feasibility of the attack when the attacker is not part of the training process and, therefore, does not have access to any portion the training dataset.

---

Published as a conference paper at ICLR 2025
We randomly select images from CIFAR- 100 (Krizhevsky et al., 2009) or even use randomly initialized pixels to launch the black-box attack. As shown in Table 6, even when the attacker initializes their adversarial forget sets with examples not present in the training set, the attack
Table 6: Accuracy drop (%) on Dretii for CIFAR-10, 10, with adversarial forget sets ini- tialized from non-training examples.
[Propal
Random pixels C-FAR-100
10
67. 14
30.69
remains effective. However, we note that detection mech-
20
65.31
26.63
anisms discussed in 4 would likely catch these attempts.
50
44.16
14.46
100
20.89
8.63
RELATED WORK
Machine unlearning. Machine unlearning is an emerging research area motivated by privacy regu- lations like GDPR (European Parliament & Council of the European Union) and their ethical consid- erations, such as the right to be forgotten. It focuses on enabling models to "forget" specifie training data, and has been studied in image classification, language models, and federated learning (Cao & Yang, 2015; Garg et al., 2020; Liu et al., 2020; Meng et al., 2022; Huang et al., 2022; Che et al., 2023; Cohen et al., 2023; Wu et al., 2023). Exact unlearning guarantees that the unlearned model is indistinguishable from the model retrained without the forget set. This is generally only feasible for simple models like SVMs and naive Bayes classifiers (Cauwenberghs & Poggio, 2000; Tveit et al., 2003; Romero et al., 2007; Karasuyama & Takeuchi, 2010). For more complex models such as neural networks, efficient exact unlearming remains an open question. Approximate unlearning offers more practical solution here, aiming to erase the influence of the forget set without formal guarantees. For example, a widely used family of unlearning algorithms problems performs gradient ascent on the training loss computed on the forget data (Guo et al., 2020; Izzo al., 2021).
Side effects of machine unlearning. Recent studies have raised concerns about various side effects of machine unlearning. Carlini et al. (2022) introduce the privacy onion effect, where removing the most vulnerable data points exposes deeper layers of previously safe data to attacks. Di et al. (2022) show that a camouflaged poisoning attack can be enabled via carefully designed unlearning requests. Hayes et al. (2024) highlight how inexact unlearning often overestimate privacy protection. Shi et al. (2024) show that unlearning methods might still exhibit partial memorization of unlearned data, providing a false sense of security regarding privacy guarantees. Shumailov et al. (2024) intro- duces "ununlcarning" and shows how previously unlearned knowledge can be reintroduced through in-context learning. Lucki et al. (2024) demonstrate that recent unlearning methods proposed for model safety only superficially obscure harmful knowledge rather than fully removing it. Our work focus on a different potential vulnerability in the current machine unlearning protocol, showing that adversarially perturbed forget sets can severely degrade model performance. Furthermore, we show the challenge of efficiently and robustly verifying the authenticity of unlearning requests.
Ad Adversarial examples. Szegedy (2013) show that neural networks could be easily misled by slightly perturbing the input images, causing models to make incorrect predictions with high confi- dence. This phenomenon has been extensively studied from both the attack and defense sides (c.g., Goodfellow et al., 2015; Moosavi-Dezfool. et al., 2017: Mądry et al., 2018; Kurakin et al., 2018). Our attack also adds adversarial noise to images but with the more challenging goal of fooling a future model produced by unlearning the adversarially perturbed examples.
CONCLUSION
In this work, we reveal a critical vulnerability in machine unlearning systems, where adversaries can significantly degrade model accuracy by submitting unlearning requests for data not in the training set. Our case study on GA and its variants, family of widely used unlearning methods, shows that after the attack. test accuracy drops to 3.6% on CIFAR- and 0.4% on ImageNet (white-box, and 8.5% on CIFAR-10 10 and 1.3% on ImageNet (black-box). Moreover, even advanced verification mechanisms fail to detect these attacks without hindering legitimate requests, highlighting the need for stronger verification methods to secure unlearning systems.
We also recognize several limitations in our work and suggest directions for future research. First, we focus on image classification, and future research could explore the attack's ' transferability to unlearning for language models (Eldan & Russinovich, 2023; Zhang et al., 2024; Maini et al., 2024; Shi et al., 2024). Second, our work targets GA-based unlearning, and future research could examine the applicability of our findings to other approaches, including non-iifferentiable methods such as task vectors (Ilharco et al., 2023). Lastly, while we have explored various defense mechanisms, there may be additional strategies that warrant investigation for their potential robustness.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Matthew Jagielski, Eleni Triantafillou, Karolina Dziugaite, Ken Liu, Andreas Terzis, Wei- jia Shi, and Nicholas Carlini for their valuable feedback and discussions.
Part of this work was completed while Yangsibo Huang was a PhD student at Princeton, and she acknowledges the support of the Wallace Memorial Fellowship and Princeton SEAS Innovation Grant.
ETHICS STATEMENT
Our work examines critical vulnerability machine unlearning systems, specifically how adver- sarial unlearning requests can potentially degrade model accuracy. A potential concern would be that am adversarial attacker could deploy our algorithm to attack an existing system with machine unlearning capabilities. However, believe the risk at the current moment is low because while machine unlearning is being actively researched on as a promising paradigm, it has not yet been widely deployed. For future deployments, Specifically, by exploring these security risks, we aim to 1) disclose and highlight potential threats, and 2) advance understanding and call for improvements such as stronger verification mechanisms for making unlearning protocols safer and more robust.
work is actually important for improving the security.
All our experiments, including white-box and black-box attacks, were conducted strictly for aca- demic purposes in order to better understand the security of machine unlearning methods. We have engaged in any malicious activities, and the datasets used (CIFAR-IK) and ImageNet) are publicly available, and widely used for research purposes.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Chooe Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In S & P, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In S P. 2015.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Brlingsson, al. Extracting training data from large language models. In USENIX Security, 2021.
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization relative. NeurlPS, 2022.
Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. In NeurlPS, 2000.
Tianshi Che, Yang Zhou, Zijie Zhang, Lingjuan Lyu, Ji Liu, Da Yan, Dejing Dou, and Jun Huan. Fast federated machine unlearning with nonlinear functional theory. ICML, 2023.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. JEEE Transactions on Information Forensics and Security, 2023.
Aloni Cohen, Adam Smith, Marika Swanberg, and Prashant Nalini Vasudevan. Control, confiden- tiality, and the right to be forgotten. In CCS, 2023.
Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. How compute Hessian- vector products?, 2024. https::/iilr-bllopppppp .iristsss.iiii.. git.ub. 2e244//log/bbnchhre
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jimmy Z Di, Jack Douglas, Jayadev Acharya, Gautam Kamath, and Ayush Sekhari. Hidden poison: Machine unlearning enables camouflaged poisoning attacks, In NeurIPS ML Safety Workshop, 2022.
John C Duchi, Michael Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. TO1r, 2015.
Ronen Eldan and Mark Russinovich. Who's Harry Potter? Approximate Unlearning in LLMs. arXiv preprint arXiv:2310.022:: 2023.
European Parliament and Council of the European Union. Regulation (BU) 2016/679 of the Euro- pean Parliament and of the Council. URL h/tpttt/////aa... www.rppp..eeeieeee///16..//
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Sanjam Garg, Shafi Goldwasser, and Prashant Nalini Vasudevan. Formalizing data deletion in the context of the right to be forgotten. In TCC, 2020.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Zou. Making Al forget you: Data deletion in machine learning. In NeurlPS, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In AAAI, 2021.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In /CML., 2020.
12

---

Published as a conference paper at ICLR 2025
Varun Gupta, Christopher Jung. Seth Neel, Aaron Roth, Saeed Charif-MMalvajerde and Chris Waites. Adaptive machine unlearning. In NeurIPS, 2021.
Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. arXiv preprint arXiv:2403.01218. 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Luxi He, Yangsibo Huang, Weijia Shi, Tinghao Xie, Haotian Liu, Yue Wang, Luke Zettlemoyer, Chiyuan Zhang, Danqi Chen, and Peter Henderson. Fantastic copyrighted beasts and how (not) to generate them. arXiv preprint 2tti::2006.145266 2024.
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. JMLR, 2023.
Yangsibo Huang, Chun- Yin Huang, Xiaoxiao Li, and Kai Li. A dataset auditing method for collab- oratively trained machine learning models. IEEE TMI, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In AISTATS, 2021.
Masayuki Karasuyama and Ichiro Takeuchi. Multiple incremental decremental learning of support vector machines. IEEE Transactions on Neural Networks, 2010.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security. 2018.
Katherine Lee, A. Feder Cooper, and James Grimmelmann. Talkin' 'bout Al generation: Copyright and the generative-Al supply chain, 2024.
Benjamin Little. Imagehash: A perceptual image hashing library. https://github. com/ bjlittle/imagehatte 2024, Accessed: 2024-09-10.
Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In CoI.LAs, 2022.
Gaoyang Liu, Xiaoqiang Ma, Yang Yang, Chen Wang, and Jiangehuan Liu. Federated unlearning. arXiv preprint 2rXiv:2012./388)) 2020.
Jakub Lucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for Al safety. arXiv preprint arXiv:2409./ 18025, 2024.
Aleksander Mądry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Pratyush Maini, Zhili Feng, Avi Schwarzschild. Zachary Chase Lipton, and J. Zico Kolter. TOFU: task of fictitious unlearning for LLMs. In COLM, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In NeurIPS, 2022.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal
adversarial perturbations. In CVPR, 2017.
Yurii Nesterov and Vladimir Spokoiny. Random gradient- frree minimization of convex functions. Foundations of Computational Mathematics, 2017.
13

---

Published as a conference paper at ICLR 2025
Enrique Romero, Ignacio Barrio, and Lluís Belanche. Incremental and decremental learning for linear support vector machines. ICNN, 2007.
Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In NeurlPS, 2021.
Weijia Shi, Jaechan Lee, Yangsibo Huang. Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint (rXiv:2407.06460 2024.
Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-limenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. Ununlearning: Un- learning is sufficient for content regulation in advanced generative AI. arXiv preprint arXiv:2407.00106, 2024.
C @Seeedyy Intriguing properties of neural networks. arXiv preprini arXiv: 2013.
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Eleni Triantafillou, Fabian Pedregosa, Jamie Hayes, Peter Kairouz, Isabelle Guyon, Meghdad Kur- manji, Gintare Karolina Dziugaite, Peter Triantafillou, Kairan Zhao, Lisheng Sun Hosoya, Julio C. S. Jacques Junior, Vincent Dumoulin, Ioannis Mitliagkas, Sergio Escalera, Jun Wan, So- hier Dane, Maggie Demkin, and Walter Reade. Neurips 2023 machine unlearning. https: 2023.
//kaggle.com com/ competi
-eeeeisss-------0e 2003--acchine-ee
Amund Tveit, Magnus Lie Hetland, and Håavard Engum. Incremental and decremental proximal support vector classification using decay coefficients. DaWaK, 2003.
Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Sci- ence. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.
Boyi Wei, Weijia Shi, Yangsibo Huang, Noah A Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, and Peter Henderson. Evaluating copyright takedown methods for language models. In NeurIPS, 2024.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons pretrained language models. EMNLP, 2023.
Yinjun Wu, Edgar Dobriban, and Susan Davidson. Deltagrad: Rapid retraining of machine learning models. In ICML,, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR. 2017.
Jingwen Ye, Yifang Fu, Jie Song, Xingyi Yang, Songhua Liu, Xin Jin, Mingli Song, and Xinchao Wang. Learning with recoverable forgetting. ECCV, 2022.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse to effective unlearning. In COLM, 2024.
14

---

Published as a conference paper at ICLR 2025
Appendices
A Experimental details
16
A.1 Compute Configuration
16
A.2 Hyperparameters
16
A.3 Black-box Attack Algorithm for ImageNet
16
B More experimental results
18
B.1 Benign Unlearning Results
18
B.2 Effect of Attack Hyperparameters
18
B.3 Indexing Attack for Exact Unlearning
18
Theoretical Demonstration of Attack
19
The Existence Results
19
C.2 Discussion
21
15

---

Published as a conference paper at ICLR 2025
A
EXPERIMENTAL DETAILS
We provide more experimental details follow.
A.1 COMPUTE CONFIGURATION
We conduct all the experiments on NVIDIA A100-64GB GPU cards with 4 CPUs. The typical av- erage runtime for different experiments are listed Table 7 with five random drawn forget datasets. For white-box attack, we report the average runtime with Taay 1000. For black-box attack, we report the runtime with ady 1000, m = 1. p = 1, =.
Table 7: Average runtime (in GPU seconds) for the attack across five independent runs, reported under different forget sizes. The standard deviations shown in parentheses.
[Droeel
White-box Attack Black-box attack
[Dtoges
White-box attack Black-box attack
10
71.72
552.98
10
808.27
944.86
20
71.30
544.54
20
794.51
1052.99
50
71.41
548.54
50
1095.13
1497.13
100
72.20
549.60
100
1758.28
2200.38
(a) CIFAR-10
(b) ImageNet
A.2 HYPERPARAMETERS
Hyperparameters for unlearning. The hyperparameters for unlearning are selected to ensure that the unlearning process on benign Diorgot is effective, resulting in a roughly 10% to 20% drop in accuracy, closely aligning with the model's performance non-training examples from Doorett- At the same time, it induces only minimal accuracy reduction on Dretain-
Hyperparameters for our attacks. We also provide the hyperparameter settings for white-box and black-box attacks used in our experiments Table 8.
Table Hyperparameter settings for white-box and black-box attacks used in our experiments.
CIFAR-10
ImageNet
White-hox attack
Forget set size ((Diorge)) Adversarial step size (Tadv) Adversarial optimization steps (Tav)
{10, 20, 50, 100) {10, 210,20, 50, 100) (0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0) {10, 20, 50, 100, 500, 1000, 2000, 5000]
Black-box attack
Forget set size ( ((Prarget) Adversarial step size (Vadv) Adversarial optimization steps (Tavv)
{10, 20, 50, 100) (0.01, 0.02, 0.05, 0.1, 0.5, 2.0, 5.0) {10, 20, 50, 100, 200, 500, 1000, 2000, 5000}
20,50, 100}
# of gradient estimators for each noise candidate (p)
{1,2, 4.5) 21,2,
[1,2,3] [1,2,3]
Candidate noise set size (m)
A.3 BLACK-BOX ATTACK ALGORITHM FOR IMAGENET
We make slight adjustments to the black-box algorithm for ImageNet, which we describe in Algo- rithm 3. The main differences between Algorithm 2 and Algorithm are as follows:
Stopping condition: in Algorithm 2, the update process for noise stops, and the loop proceeds to the next iteration if the conditionsg(Xoegg + z + A) < g(Xtoget + 2) and g g + z A) 9(Xrag + z) are both satisfied. In contrast, we found this condition frequently occurs for ImageNet (Algorithm 3), so we allow the algorithm to continue without stopping, making the process more robust to such scenarios.
16

---

Published as a conference paper at ICLR 2025
Gradient estimation parameter: In Algorithm 3, we introduce additional hyperparameter d to control the number of gradient estimators. Specifically, d gradient estimators are drawn and averaged to obtain the final gradient. This adjustment is made because white-box attacks tend to perform well on ImageNet, and averaging over a larger d improves the quality of the gradient estimate.
Algorithm 3 Black-box attack for ImageNet
I: Input: original model funget a collection of training examples X C D retain set Preiuins unlearning method U, adversarial step size Tadv, training steps Tadv, access to loss function g( ((Xpddy := Lretain (U(Juuget {Xadd
forget: Moorgtt Dretin)), hyperparameters p. m, d.
2: Initialize forget f
3: Initialize noise candidate set NoiseCandidates of size m
4: for = 1, .Tav do
5: for E NoiseCandidates do
6: z' + BssinateGraliennssmeeenttt/ 9)
1 Call to the new ImageNet gradient
estimation procedure */
7: Append x' to NoiseCandidates
8: end for
9: Keep the top noises in NoiseCandidates (based on loss function g)
10: end for
11: Choose the best in NoiseCandidates
12: Return: (Xooreet + 2, (frrget)
13: procedure ESSTRATTGGRADENNTSIIAANNNTT p, p. / Estimate the gradient and update
noise for ImageNet +/ for = 1, do
14:
1 Repeat p times for noise candidates */ do draw samples and compute the average for higher accuracy */ Draw random unit noise A
for = 1,
16:
17:
Compute g (fforge + z + A) and g (Xforttt + z A) Estimate gradient Vgj <- gf ofogee tot +x+ - g(Nigge A) A
18:
19:
end for
20:
Compute the average gradient estimator Vg(z) - - (gz(z)
Update noise: z' Tadv gz(z)
22:
end for
Return: updated noise z'
24: end procedure
17

---

Published as a conference paper at ICLR 2025
B MORE EXPERIMENTAL RESULTS
B.1 BENIGN UNLEARNING RESULTS
Table 9 reports the accuracy for the forget, retain, and test sets under benign unlearning requests of various sizes. As shown, the accuracy on the forget set is roughly comparable the test set after unlearning, while the accuracy on the retain set remains high.
Table 9: Accuracy on forget, retain and test sets under benign unlcarning request.
Diorgel
funtearm Forget accuracy
funkearn Retain accuracy
Funteam Test accuracy
10
83.15 + 14.82
97.93 + 4.78
86.88 + 3.92
20
89.01 7.76
99.09 = 1.04
87.89 = 0.95
50
86.18 + 3.74
99.21 + 2.85
88.03 = 0.24
100
86.72 2.05
99.36 6.18
88.29 0.17
B.2
EFFECT OF ATTACK HYPERPARAMETERS
We further examine the performance of the white-box attack under different hyperparameter figurations. Results shown in Figure 6 reveal a clear trend: lower attack step sizes (7avv) are more effective for attacks targeting smaller forget sets, while higher attack step sizes and increased opti- mization steps (Tavy) improve attacks on larger forget sets.
Adv. step size (Need))
0.06
0.1
0.5
1,0
5.0
[Darged:
DDorgedl:
iDerged
DDaggali 100
Ratan (%))
100
100
1000
2009
3003
430
1000
1290
2000
3000
400E
1900
1000
20:0
2013
.000
5000
1308
2000
3000
4000
3009
Adv. optimization
Adv. oplimization stops (Tacs)
Adv. optimizatiin stops (Tae)
Adv. opTimizaiion stops (Tasx)
staps (1 seo)
Figure Retain accuracy after white-box attack on CIFAR-I0 various combinations of Tiav and Maae. Attacking a larger Draget typically requires a higher Taav and larger Tadv.
B.3 INDEXING ATTACK FOR EXACT UNLEARNING
Earlier in 4.2, we discuss another potential protocol for unlearning: instead of directly accepting incoming images, the deployer performs a similarity search against stored training images and re- trieves the closest matches for use in the unlearning process. However, we show that this protocol still vulnerable in a sense that attackers may optimize the selection of examples in the forget set to maximize negative impact on model performance. For GA, the attack most effective when Dronget is small.
We also investigate this vulnerability for exact unlearning in Figure and find that vulnerability slightly increases with larger forget set. This is because, with larger forget set, exact unlearning excludes more data points, making it slightly casier for certain selections of the forget set to cause a more significant accuracy drop.
18

---

Published as a conference paper at ICLR 2025
min
25.0
14
13.0
13.9
12.8
22.5
Rolain
12
11.2
10.1
11.2
3.9
8.0
11.4
8.8
20.7
19.3
19.3
16.2
17.5
15.0
100
500
1000
5000
100
500
1000
5000
Drogget
DDorged
(a) Retain error
(b) Test error
Figure 7: The attacker can optimize the selection of examples in valid forget degrade performance. We report the maximum, mean, and minimum retain error on CIFAR-10 with exact unlearning for varying [Dregei) with 500 random selections per size.
C THEORETICAL DEMONSTRATION OF ATTACK
In this section, we discuss simple stylized setting of learning algorithm, an unlearning algorithm (that achieves perfect unlearning), and formally show that there exist adversarial perturbations of the forget that can make the unlearning algorithm yield predictor that performs poorly on the retain set.
C.I. THE EXISTENCE RESULTS
Our goal is to construct a simple setting where it is easy to conceptually understand why such an attack exist. We our experiments, but we acknowledge a number of assumptions such as Gaussian inputs and linear models do not fully match the settings of our empirical results. Nonetheless, the formal results provide theoretical intuition on why such attacks are possible.
unlearning algorithm similar the GA-family algorithms studied in
We discuss possible future directions to study more such constructions in $ C.2.
For data y) drawn i.i.d. from a distribution P. learning algorithm, abstractly speaking, is (randomized) method : (X x y)* + that maps sequences Dusin ((z1, y1)..... (In Vn)) of examples to a predictor h E H. We measure the quality of the learning algorithm in terms of its empirical and population loss, namely for l like to minimize the population loss f(h; P) [( (h; v, or even the empirical loss
(X )) {0, 1}, we would
L(h; Durin)
Peam E(z.))DD.. e(h;
An unlearning algorithm is (randomized) method U: H x (X x x - H that maps the current predictor h E H. forget set Drorget E (X' x 3)) and retain set Dretain E (X x y H. desirable property of the unlearning algorithm is that when invoked on h being the output of the learning algorithm input D Diarges o DDettin U(L Dretain), Diorget Dreumn) has the same distribution as, or least "close to" the distri- bution L(Pretain); we say that (L, U) is perfect to achieve perfect However, this method of unlearning is not desirable as it can be computationally inefficient to per- form learning on D Drecain from scratch. Instead, is desired that (L o Dresain), Diagett Pretain) has computational complexity that is significantly less compared to that of L(Dretain).
an updated predictor
pair if this holds. A trivial way Dremin)) ) Drorget, Dretain) L(Pretain).
is by setting U(L (Diorget o
We will now consider a specific realization of the learning and unlcarning algorithms as follows.
Data Distribution. We consider the data distribution Pn- defined over Rdd {-1, 1}, parameterized by h Idd, obtained by sampling (o, )La)) and setting y = sign (h" (h),))))
We will consider the high dimensional regime where & d.
Loss Function. We the 0-1 loss function for halfspaces, (0,;,,0) v) 1 (v + sign (h,,))))
19

---

Published as a conference paper at ICLR 2025
Learning Algorithm. We consider perceptron-like" learning algorithm, defined as L(D
(aa)) Y/ Iz-
Unlearning. We consider the unlearning algorithm U(h, Dioeget; Dresain) = h Σ (aay)eDmme y It. Observe that in fact, the algorithm does not even use Dretiin and thus is more efficient to compute than computing L(Pretam) from scratch.
e- c-Perturbation. We say that D'
Hn)) (2. is an e-perturbation of D ((T1,91).... (In: Mn)) if for all i, it holds that || zill2 S S.
V1), ..... (I',,
We now present the main result of this section that shows the existence an E-perturbation of the forget set that makes the unlearning algorithm return a predictor that misclassifies all examples in the retain set.
/di log(d/3) there exists O(Vn/E) such that for Diain sampled i..d. from Phe with (Drraial Diorgt being randomly chosen subset of Duain with (Djorger = m and Dretain == Duain Dforgets all of the following hold with probability 1 - 3 over randomness of sampling the data:
Theorem C.1 (Adversarial Forget Sets). For all E, > O and
1. C(L(D D Divai) = O. In other words, the learned predictor achieves perfect accuracy on the training dataset.
U(L (orger retain); Djorget Dretain) = L(Dretain) other words, (L a perfect learning- unlearning pair.
3. There exists an e-perturbation Diooger of Djsrger such that ((h':Dreain) for h' U(L((D D/ Plwwger? DD -reuuin) In other words, when provided a perturbation of Djrrett the pre- dictor after unlearming misclassifies the entire retain set Dretain-
We rely on the following concentration bound on the norms and inner products of random Gaussian vectors, which follow from simple applications of Bernstein's 's inequality for sub-exponential random variables.
x² La). it holds that Fact C.2 (See, e.g., Vershynin (2018)). For x, y (0,
Pr[l|= do
Edo² < 2exp (-#4d/4)
Pr[] x =do²2
< 2exp (-E² =2//4)
Proof of Theorem C.I. We first note that for Fi...., In ~ N(0, Ha). we have from Fact C.2 that with probability 1 3, all of the following hold:
| 2 V log(n/8))
d
|2:112
< 1 +
log(n/8) n
(2)
d
(o,,,y))
<
3
log (n/3)
(3)
We now proceed prove each of the claimed parts, by conditioning on Eqs. (1), (2), and (3) holding. We use the notation Dirain = ((I1, y1 ,I.,... (Ii, Yn)))
1. We have L(Dram) = h = E, y x, From Eqs. (1) and (3), we have that with probability 1 3,
it holds for all (I, 8) E Duains that
y (h, ||ı||| + E(e' EDm Dun wy (x,x²)
log(n/3) 3(n 1)
log(n/D) 0,
1
d
where we use that n < O(Vd/log(d/2) B)). Thus, with probability 1- B, we have that all examples in Drrain are correctly classified by
'The Perceptron algorithm (Rosenblatt, 1958) operates on the examples one at time, by choosing an example (xi, yii that is misclassified by the current predictor h, and adds y; h. Here, we instead perform single step on all examples once.
20

---

Published as a conference paper at ICLR 2025
As aside, note that under Eqs. (2) and (3), it also holds that
|| E, ||111122
Eiyg Mill
(Tr B3y)
S n 1
log(n/m)
n(n-1)
log(n/8)
< (()).
d
2
2. This is immediate since U(L(( D Porgett D Pretain) E Preu) y, x, which is precisely L(Dretiin)
((z,)) y E(ay)EDaan y.v.
We consider the following e-perturbation Diongt of Diorget. obtained by including (a' Ey
h [ijl [hjl.// 3) in Pinger for all (æ, v) Dorrett Note that this requires only knowledge h and the forget set Diorges; do not need any information about the examples in Dretain-
The unlearning algorithm on this perturbed set would return: = (z,)) V cm h.
||11|2
Thus, for suitable m OVVn/e), we then have that ch for O, and hence ys- th. xx.) for (x4,3) Durinn and in particular, the returned predictor i will misclassify the entire retain set.
C.2 DISCUSSION
In summary, Theorem C.I shows that for the learning-unleaminn pair considered, small perturba- tions on a forget set that is much smaller than the training set size can cause the unlearning algorithm to return a predictor that misclassifies the entire training set. However, we note the following limi- tations of Theorem C. that leave for future work to address.
First and foremost, the theorem is about a linear model, which does not capture the complex non- linearity of neural network. Thus, it does not immediately provide any specific insight into the experiments we perform on neural networks in this paper.
Another limitation, is that do not show that the learned predictor achieves small population loss. This is in fact impossible in the regime when n < d as we consider. It would more desirable to have a learning- unlearning setting where the learning algorithm also achieves small population loss. However, we do note that the learning algorithm we consider is still "reasonable" in the sense that does have generalization guarantees with n d, on the family of distributions (Pr- h" E Rd) that we consider.
leaming--unlaaningg
We suspect that the reason why the phenomenon occurs in neural networks that small perturbations in feature (e.g., pixels) space can cause larger perturbations to the corresponding gradients. Whereas, the setting of Theorem C. the attack arguably arises because the retain set examples are quite diverse from each other, and so a small forget set of size only O(/R/E) can perform "coordinated attack" to push the predictor in a bad region.
Finally, another gap between theoretical example and our experiments is that in the latter, did not need a large forget set to execute an attack. In particular, the unlearning algorithm used averaged gradients over the forget set, the attack could not have benefited just from the forget set being large.
21


Reviews:
[{"review_id": "VMpHssAE3Z", "reviewer": "Reviewer_Hn2d", "summary": "This paper explores security risks within machine unlearning schemes, a topic gaining importance as the demand for ensuring \"the right to be forgotten\" grows. Specifically, the paper introduces a type of attack that degrades a model's accuracy by submitting adversarial unlearning requests, with evaluations demonstrating its effectiveness. Additionally, the paper shows that various existing verification mechanisms fail to detect these proposed attacks.", "questions": "- Can you think about possible ways to enhance the defense against such adversarial machine unlearning attacks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The concept of adversarial machine unlearning attacks is innovative and contributes a new perspective to the field.\n- In addition to empirical evaluations, the paper provides a theoretical demonstration of the proposed attack, enhancing its depth and rigor.\n- The paper is well-structured, with clear and accessible content that is easy to follow.", "weaknesses": "Overall, I enjoyed reading this paper. Below are some suggestions that could help improve it further:\n\n- *Scalability Limitations in Black-Box Attacks*: While the paper’s black-box attacks yield promising results, they might face challenges when applied to high-dimensional data. Including a more detailed discussion of this limitation could add depth and insight. Addressing these challenges would provide valuable context about the trade-offs and bottlenecks in black-box attacks, enhancing the practical understanding for researchers and practitioners in the field of machine unlearning and security.\n- *Limited Exploration of Adaptive Defense Mechanisms*: This paper makes a valuable contribution by assessing the effectiveness of various defensive mechanisms and highlighting potential vulnerabilities. However, investigating additional adaptive defenses, such as anomaly detection or similarity-based clustering methods, could strengthen this section.\n- *Ambiguity in the definition of “Black-Box” Attack*: Although termed a “black-box” attack, the method used by this paper still requires access to the training loss. This might be somewhat misleading, as traditional black-box adversarial attacks are generally assumed to operate without any internal information about the target model. Adjusting this terminology or clarifying the scope could improve precision and align expectations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P1jGq6zjf7", "reviewer": "Reviewer_jZsQ", "summary": "This paper investigates the problem of adversarially degrading model accuracy by submitting adversarial unlearning requests for data not presented in the training set. Two settings are considered, white box and black box, depending on whether the attacker can access the model's gradients. Experiments show that such requests can significantly degrade the model's performance. The paper also presents the challenges of detecting the legitimacy of unlearning requests.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "### Originality\n* The idea of studying adversarial unlearning requests is novel and interesting. This is a valid concern when the model deployer accepts user-submitted unlearning data.\n* The overall methodology is novel, including the use of the gradient-through-gradient technique, regardless of some inevitably similar techniques to poisoning attacks and gradient estimation in black-box attacks.\n\n### Quality\n* The methodology is solid under the proposed threat model, and correctly verified using experiments on CIFAR-10 and ImageNet, with three unlearning algorithms.\n* The evaluations are generally comprehensive. \n\n### Clarity\n* The paper is well-written and easy to follow.\n\n### Significance\n* The overall threat model and takeaways are useful, under the assumption that it makes sense for a model deployer to accept user-submitted samples to execute the unlearning algorithm (see weakness for why this may not be the case).", "weaknesses": "### Originality\n\n**Q1: Related work of security risks of unlearning tasks and poisoning attacks.**\n\nSince this paper's threat model is similar in spirit to poisoning attacks, it's suggested to discuss more details on the related work and address two main questions. First, have there been prior efforts to study the security risks of machine unlearning tasks? Second, what's the difference between the threat models of this work and poisoning attacks, given that the attacker in this paper is also able to execute poisoning attacks from the beginning?\n\n### Quality\n\n**Q2: Practicalness of the threat model.**\n\nThe assumption of this paper is that the user may submit malicious images as unlearning requests to the model deployer, who will execute the unlearning algorithm on such images. However, I'm not sure if it ever makes sense for a model trainer to rely on user-submitted data to execute unlearning algorithms. In practice, it's usually the model provider's responsibility to identify which data belongs to the user and unlearn those data stored in the model provider's datastore. Can the authors provide some concrete use cases, preferably realistic, where any model provider will (1) not store the training data despite their original ability to train the model and (2) accept user-submitted samples to delete the data?\n\nExtending from this concern, the proposed threat model may also cause some other issues, where User A may submit benign samples belonging to User B and request deletion on behalf of User B without User B's consent. I believe what's actually happening is that the model provider will go through their data store and identify those belonging to the requester, which they know for sure (1) belong to the requester and (2) were indeed used in the model training.\n\nThe second aspect of this question is that the attacker already controls a subset of the training data. In this case, why can't the attacker simply execute poisoning attacks, which are more stealthy than this attack? In this case, the attacker's identity is easy to trace, yet it's much harder to identify who has sent the poisoning samples.\n\n**Q3: Regarding the defences with full access to training examples.**\n\nThis is similar to Q2 but applies specifically to Section 4.2. In this subsection, the defense has full access to the training samples, so I guess in this case the model provider does not have any limitations in Q2. If so, wouldn't the best defense be not trusting the user-provided data at all? As a model provider, or in fact, the service provider, they must have the ability to identify where the data was originally collected from. Otherwise, they won't be able to delete the user's data in their training set. Then they could just identify the precise set of user's data used for model training, and then execute the unlearning algorithm.\n\n### Clarity\n\n**Q4: Concerns regarding the experiment settings.**\n\n1. In Section 3, the forget set has 10 to 100 samples. It's unclear how this range is devised, but 100 may sound too much IMO, as it means the model provider will accept up to 100 samples, which the requester claims belong to them, to update their model weights. It's suggested to provide more concrete justification for why such numbers are chosen.\n2. The evaluation quantifies the attack's performance as the *maximum* accuracy degradation observed across a grid search of hyperparameters. I'm curious (and concerned) why the maximum instead of other more common metrics are reported, such as the average, medium, or confidence interval. It's also not very convincing that, in reality, the attacker would be able to have a grid search and obtain the best results.\n\n### Significance\n\nI think the overall idea and evaluation of executing machine unlearning algorithms on malicious inputs are interesting and useful. However, my main concern regarding the significance is that the threat model may not be realistic (Q2-Q3). It's true that unlearning algorithms have been implicitly assuming that the data to be deleted are indeed used for model training, but such an assumption is built upon another assumption that, in order to delete the user's data, the model provider must have the basic ability to identify the correct set of user data in their data store, and only then go to the next level of deleting data from the model. It's suggested to provide more justification for the practicalness of the proposed threat model, where the model provider is expected to delete data from the model without the ability to reliably identify the user's precise set of data stored in their database.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Qit9YNijo", "reviewer": "Reviewer_hRNb", "summary": "This paper introduces Unlearn and Burn, an adversarial attack targeting machine unlearning algorithms to significantly degrade model performance by submitting malicious unlearning requests. The attack exploits the assumption that all unlearning requests correspond to legitimate training data, allowing an adversary to request the removal of data not present in the training set. The authors propose both white-box and black-box versions of the attack and demonstrate its effectiveness on CIFAR-10 and ImageNet, with results showing considerable drops in accuracy. The study also highlights the challenges of detecting such stealthy attacks, suggesting that existing verification methods for unlearning requests may be inadequate.", "questions": "Could you clarify the specific differences between your proposed attacks and prior poisoning methods?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents an interesting and relevant idea, yet it has some notable issues that need addressing.", "weaknesses": "Exceeds page limit:  The paper currently extends beyond the 10-page maximum allowed by ICLR, utilizing in total 11 pages of core content. Additionally, several crucial aspects, such as unlearning algorithm specifics, hyperparameters, and theoretical results, are deferred to the appendix, making it challenging to assess the completeness of the methods presented within the main body.\n\n\nMischaracterization as a novel adversarial attack:  While the approach is framed as a new type of adversarial attack, it is more accurately an adaptation of established poisoning techniques. The attack uses a standard poisoning approach where the unlearning update replaces fine-tuning or training in the model. As outlined, the objective of minimizing model accuracy while removing poisoning samples mirrors conventional poisoning attacks. This is detailed in prior work such as [1-4] or survey [5] which already connect poisoning attacks to meta-learning approaches, similar to the attack formulation in Section 2.2 of this paper. Given these similarities, I observe a lacks of novelty compared to the state of the art and risk for creating confusion with respect to something that already exists in the literature. \n\n\n\nLack of novelty in attack mechanisms: The two attack methods (white-box and black-box) do not introduce any new methodologies beyond existing techniques. The white-box method is essentially a re-implementation of approaches discussed in [1] and [3], while the black-box method, based on gradient estimation, has been used in [6]. Consequently, the attack does not present a unique contribution in terms of adversarial methods.\n\n\nGiven the issues above, particularly the excessive page count and the lack of novelty, I recommend rejection of the paper. I encourage the authors to reconsider the framing of their approach, positioning it as an adaptation of poisoning attacks within unlearning settings rather than a new adversarial attack. I hope these comments will serve as constructive feedback for further refinement.\n\n\n\n[1] Metapoison: Practical general-purpose clean-label data poisoning. NeurIPS 2020.\n\n[2] The hammer and the nut: Is bilevel optimization really needed to poison linear classifiers? IJCNN 2021.\n\n[3] Towards poisoning of deep learning algorithms with back-gradient optimization. ACM workshop on artificial intelligence and security 2017.\n\n[4] Witches' brew: Industrial scale data poisoning via gradient matching ICLR 2021.\n\n[5] Wild patterns reloaded: A survey of machine learning security against training data poisoning. ACM Computing Surveys 2024.\n\n[6] Generative poisoning attack method against neural networks. arXiv 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)