Static and Sequential Malicious Attacks in the Context of Selective Forgetting
Authors: CHENXU ZHAO
Wei Qian
Zhitao Ying
Mengdi Huai
Venue: NeurIPS 2023 poster (2023)
Keywords: Selective forgetting
static setting
sequential setting
security and robustness

Abstract: With the growing demand for the right to be forgotten, there is an increasing need for machine learning models to forget sensitive data and its impact. To address this, the paradigm of selective forgetting (a.k.a machine unlearning) has been extensively studied, which aims to remove the impact of requested data from a well-trained model without retraining from scratch. Despite its significant success, limited attention has been given to the security vulnerabilities of the unlearning system concerning malicious data update requests. Motivated by this, in this paper, we explore the possibility and feasibility of malicious data update requests during the unlearning process. Specifically, we first propose a new class of malicious selective forgetting attacks, which involves a static scenario where all the malicious data update requests are provided by the adversary at once. Additionally, considering the sequential setting where the data update requests arrive sequentially, we also design a novel framework for sequential forgetting attacks, which is formulated as a stochastic optimal control problem. We also propose novel optimization algorithms that can find the effective malicious data update requests. We perform theoretical analyses for the proposed selective forgetting attacks, and extensive experimental results validate the effectiveness of our proposed selective forgetting attacks. The source code is available in the supplementary material.

Static and Sequential Malicious Attacks in the Context of Selective Forgetting
Chenxu Zhao"
Wei Qian
Department of Computer Science
Department of Computer Science
Iowa State University
Iowa State University wqi igg@iisttte..m edu
cxzhao@iastate. edu
Rex Ying Department of Computer Science Yale University rex. wing@yale.a.....mm edu
Mengdi Huai Department Computer Science
Iowa State University mdhuai@iasti @iannii@astateee edu
Abstract
With the growing demand for the right be forgotten, there is increasing need for machine learning models to forget sensitive data and its impact. To address this, the paradigm of selective forgetting (a.k.a machine unlearning) has been extensively studied, which aims to remove the impact of requested data from a well-trained model without retraining from scratch. Despite its significant success, limited attention has been given to the security vulnerabilities of the unlearning system concerning malicious data update requests. Motivated by this, in this paper, we explore the possibility and feasibility of malicious data update requests during the unlearning process. Specifically, we first propose new class of malicious selective forgetting attacks, which involves static scenario where all the malicious data update requests are provided by the adversary at once. Additionally, considering the sequential setting where the data update requests arrive sequentially, we also design novel framework for sequential forgetting attacks, which is formulated as a stochastic optimal control problem. We also propose novel optimization algorithms that can find the effective malicious data update requests. We perform theoretical analyses for the proposed selective forgetting attacks, and extensive experimental results validate the effectiveness of our proposed selective forgetting attacks. The source code is available in the supplementary material.
Introduction
Machine learning algorithms play a crucial role in diverse fields such as biology, speech recognition, agriculture, and medicine. To build pertinent models, these algorithms frequently trained using a range of data sources, including third-party datasets, internal datasets, and customized subsets of publicly available user data. With recent demands for increased data privacy, the data users could crase the impact of their sensitive information from the trained models to ensure their privacy. Recent legislation (e.g., the General Data Protection Regulation from the European Union (60) the California Consumer Privacy Act [47], and the Canada's proposed Consumer Privacy Protection Act) requires the right be forgotten, and grants users unconditional right to request that their private data be removed from everywhere in the system within a reasonable time.
"The first authors contribute equally this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

However, with the development of traditional machine learning techniques, this basic right is usually An illustrative instance is the inadvertent leakage of patients' genetic markers through machine learning methods employed for genetic data processing without the patients' awareness. Therefore, it is important to entitle data users the right to delete their personal data from trained machine learning models since machine learning models could memorize sensitive
neglected or violated 770, 19 38
information of the training data and thus expose individual's privacy risk [54] 6 30 59 40). The most naive way is to retrain from the original data after removing the samples that need to be forgotten. Unfortunately, this naive retraining method can be prohibitive in terms of the computational and space cost esse-cilllllyy for large models and frequent deletion requests. To mitigate this, selective forgetting 28 63 has been extensively researched in recent years to avoid the high computational cost associated with fully retraining model from scratch.
(a.k.a machine unlearning) 4 43 22
20
However, existing studies on selective forgetting mainly focus on designing new forgetting algorithms to enable the right to be forgotten to be efficiently implemented, leaving the security issues during the unlearning process in adversarial settings largely un- explored. In practice, the motivated adversary could make use of the unlearning pipeline to craft malicious data update requests to achieve his/her desired attack goals. For example, the motivated adversary could increase a disadvantage against specific group of individuals. In Figure D we present a toy example to highlight the impact of malicious data update re- quests on faimess using the COMPAS [32 dataset.
Fairness gap
0..35
Demegraphic Parily Equalized Odds
0.30
0.25
0% Percentage of unlearning samples
10%
20%
30%
40%
50%
Note that fairness in machine learning 667 55) refers
Figure Fairness gap (Demographic Parity and Equalized Odds) with random unlearning samples in the minority group on COMPAS.
to the concept of ensuring that the decisions pro- duced by machine learning algorithms are unbiased and equitable across different groups of individuals. irrespective of their protected attributes (e.g., race and ethnicity). Here we consider race (black/white) as the sensitive feature and just randomly unlearn varying percentages of samples from the minority group (white). For the fairness evaluation metrics, we adopt Demographic Parity [16] and Equalized Odds [24] (please refer to the supplementary material for details of the two evaluation metrics and more experimental results on attacking fairness). From this figure, we can easily see that even though we just randomly delete some individuals from the white group in the dataset, the fairness gap expands, indicating that the model's fairness is compromised. When the victim systems are employed for security-sensitive applications, such malicious data update requests can cause tremendous security threats to the unlearning system. Therefore, it is essential to understand the feasibility of malicious data requests in the unlearning systems.
In this paper, we aim to conduct a comprehensive study on the security vulnerability and robustness of the unlearming system to malicious data update requests during the unlearning process. Specifically, we first propose a novel static selective forgetting attack framework, where the adversary exploits vulnerabilities in the unlearning systems by submitting set of carefully crafted data update requests at once. More specifically, the proposed static attack framework uses discrete indication variables to formulate the complete deletion of targeted training samples, which is very hard to be directly solved. To address this challenge, we design continuous and differentiable function to approximate the discrete component. On the other hand, many real-world applications involve streaming data update requests that arrive in a sequential manner. The adversary could take advantage of this sequential interaction setting to strategically manipulate the unlearning process. However, launching attacks on all the received sequential data update requests indiscriminately can potentially lead to the detection of the adversary. To address this issue, we also design novel sequential selective forgetting attack framework that takes into account the order and timing of data update requests. In this framework, the adversary focuses attacking few critical data update requests maximize the impact of his/her malicious actions, potentially leading to severe security threats. We also conduct theoretical analyses of our proposed selective forgetting attacks. Further, we conduct extensive experiments in different scenarios to validate the general effectiveness of our proposed selective forgetting attacks.
2

---

2 Related Work
Selective forgetting (a.k.a machine unlearning and data deletion to removing the influence of the requested data from a trained model. However, existing data ignore the risks of malicious data update requests during the unlearning process and fail to identify the vulnerabilities of machine learning models to selective forgetting attacks. It is worth mentioning that our proposed selective
20 38 19
38
49
62 refers
deletion works [9 19
T0
43
22
20
28
63
forgetting attacks are different from traditional evasion attacks [66] 68 3 34
61
27
69 26 and
data poisoning attacks 14 51 39]. Evasion attacks achieve adversarial goals by modifying test samples; however, our forgetting attacks do not change test data and only modify the well-trained models by making malicious data update requests. Traditional data poisoning attacks occur during training, and manipulate the original clean training data. Note that [13] adds carefully crafted samples to the training dataset, and assumes the exact and retraining tasks. The authors in [88] fail to address
the issue of malicious whole data deletion in the static setting, nor do they consider the sequential attack setting involving various types of malicious data update requests.
3 Malicious Selective Forgetting Attacks
Without loss of generality, in this work, we consider the classification models. Let D ( (zi, y4 ))) J. denote the training dataset from Z = X' x y,, where x; E IRd is a addddimensional sample and y E [C] denotes its associated class label. The model owner applies a learning algorithm A on D to learn a model (0) parameterized by 9 such that f() achieves low empirical loss. We denote loss function by a mapping & @ x Z - R that takes the parameters @ E θ and a sample & E Z, outputs the loss of 0 on 2 (i.c., {(2; 9) that over the resulting model parameters A when the learning algorithm A is applied on D.
denote by (0)). Let A(D) denote the distribution
Threat Model. As previously mentioned, the proposed malicious selective forgetting attacks are unlearning time attacks, i.e., the adversary interferes with the unlearning process of the well-trained model and cannot modify any test sample submitted to the victim model at testing time. In addition, the adversary is unable to modify the training samples during the training stage. This reflects the unlearning scenario in which the adversary can only generate the update requests to selectively forget certain data information during the unlearning process. In this paper, we study both the white-box and black-bor bbx settings. Specifically, in the white-box setting, we make the assumption that the adversary possesses complete information about the system (including the model architecture and parameters of the well-traines model). Note that many evasion and poisoning attacks in the literature 33 29) employ a white-box model to study the adversary's s strong attack behaviors in such worst-case settings. In the black-box setting, we assume that the adversary does not have any prior knowledge about the target well-trained model.
[ 42 65 52
I
3.1 Static Selective Forgetting Attacks
Here, we consider the static attack setting, where the model holder owns a well-trained classification model (0) parameterized by E R& on dataset D, and the adversary aims to make malicious update requests to deliberately forget some information to achieve his/her desired attack goals. In this case, all the malicious data update requests are provided at once, without consideration for the sequential order. We use D, C D and RA to denote the adversary's requested forget set and the coupled unlearning algorithm for A, respectively. Note that for the given forget set, machine unlearning method can return model 9 ~ (D A A(D), Dy), which posses information about D, without influencing the contributions of other data. The following definition gives the definition of existing approximate and exact unlearning methods [64 44 23
Definition (Machine unlcarning). Let A and RA denote the learning algorithum and the unlearning method, respectively. The pair (A, RA) achieves exact unlearning V D, C D, A,D,)) =a
RA A(D), D)), where D, = D o' Dy and =a means the same distribution. This means that if the unlearned model from RA(D, A(D), Dj) has information about D,, we cannot differentiate the model after forgetting from a model that is obtained on D,.. The pair (A, RAA) satisfies (c, unlearning DD Dj C D, and F. C TRdd, P(RA A(D), D2) E F) S p(((D), E F.) + where A and Dy denote the learning algorithm and the forget set, respectively.
3

---

As aforementioned, the adversary aims to make malicious update requests for his/her desired attack goals. For example, the adversary could attack the largeted test samples and foree them to be assigned as the attack targeted label 77. The adversary could also have a specific target model @far (e.g., the unfair and backdoored models) in mind and aim to induce a victim model as close as possible to that target model f [56]. The effective unlearning samples (i.e., Dj C D) can be obtained by solving the following formulated optimization problem
Dj arg min D,CD:
where D, {(xxx. Uptp. C D.
(I)
In the above, De represents subset of D that is accessible to the adversary, and 6% (D,) are the parameters found by completely eliminating specific targeted samples D, C D { 3p) }p)} } For each Zp Dr. we define discrete indication parameter Wp E {0, 13 to indicate whether the sample Tp should be completely deleted (wp = 1) or not (wip = 0). The forget set D, to be unlearned
is denoted as D, = D2 o l = {xp Tp E Dr and wp = 1}, where s = fwi E {00, 1 Note that the above equation is a bi-level optimization problem the minimization for D, involves the model parameters 8d (Dy). which are themselves the minimizer of the training problem,
( (R((D,D(('') D, Dr 92,, where D, = {EEp Mp 1 and Ω = (wn)
(2)
Note that Eqn. 4T) and (2) provide the high-level formulation for selecting subset D, such that the adversary's 's goals are maximized after unlearming. However, is very difficult to optimize the effective update requests due the introduced discrete indication parameters (i.e., Q {wp E {0. } g- of the first constraint in Eqn. 2 Next, we take the second-order unlearning strategy proposed in [64] as an illustrative example to show how to solve the above formulated optimization problem. This unlearning strategy uses the inverse Hessian matrix of the second-order partial derivatives to change the original model's parameters to obtain the unlearned model (64. For D. = = {2p]p D,
we use D. = Ž { to denote its corresponding unlearned versions, where εp = (Ipp Ep. Up) and E, is the unlearning modification for Tp. Following (44, we calculate the second-orde change AZ 2) by calculating all the gradients difference between Z and Ž with a weighting change from the inverse Hessian of the loss function, i.e., A(Z, 2) = H (Ex, E up * Vol(Z;;0')
Ex,ez wp* Vol(zp; (*)), where Hg. is the inverse Hessian matrix and wp E {0, 1] denotes whether Ep should be completely erased. To address the aforementioned challenge, we propose to relax each discrete variable up into continuous one of range [0, 1]. i.e., Wp E [0, 1], and then approximate the original update objective for this second-ordes strategy by the following one
P
1
80 t 0"
Hg.
1 exp( (2 wp - 1)) * (Vol(žp:0)) - Vel(zpi('))) p=1
11))
(3)
where is training loss, and wp E [0, 1]. Here, we rewrite up (1+ - 1)). Based on the fact that function hi (xx [ sgn(x)) can be approximated by function (2))) I+exp!- -pa] we can obtain the above equation. The parameter yp h2 represents the steepness of the curve. Additionally, the continuous property of h2 allows to solve the formulated optimization to perform selective forgetting attacks via the second-order unlearning strategy in [64]. From Section IV in [64), we can know that if 2 = 0. the adversary has the option to completely remove the targeted sample or retain it, based on whether wp 2 0.5 or wp < 0.5. When 2 a 0, the adversary can intentionally and maliciously modify the targeted training samples via partially unlearning some data information based on Eqn. 33). Notably, following Lemma [] we can easily generalize the above proposed attack framework to the scenario where the adversary wants to maliciously erase some features [64]. The algorithm and generalization to other unlearning methods are deferred to the supplementary material.
Lemma 1 (164). Let F' denote the features to be unlearned. Let 0-, denote the optimal model retrained on the new dataset that is derived by removing the features F from D. For learning models processing inputs x using the linear transformations of the form HT x, we have 6'p = fp where P is retrained by setting the values of the features F to zero in D.
Definition 2 (Strongly convexity). A function vi RRds + Rd is said to be M-strongly convex for some MM 2 if for any 21 E RRda 2/ E Rdd, and any q E (0,1), (4q21 + (1 - g)²2) S
(21) ) + (1 - ((1) - 2 (1 e)||= - 22112. Note that if the above condition is satisfied for M = O, we refer to the function vb convex.

---

Definition 3 (Lipschitzness continuity). A function vb Rds + R is L-globally Lipschitz continuous for all 2 E Rd,, 22 E W. [10(22) - ((22)] S L1|21 - -2112-
Theorem 1. Let 0% = arg mingges (0) for any given dataset D. Suppose that the loss function lg is I-globally Lipschitz continuous and M-strongly convex for any Z E Z. For any integer N. dataset D of size N, and forget set Dy, we can have that 110 - -0% -DD,,, S 21. Dyl, where Dyy represents the size of the forget set.
MN
The above lemma shows that we can erase features from many learning models by first setting them to zero [64]. The above theorem relates the difference of the model parameters to the forget set, and can be easily generalized to the defined data update requests in Definition 4
3.2 Sequential and Dynamic Selective Forgetting Attacks
In practice, the model owner usually receives sequential update requests from one or more data owners at different times, and is asked to update the model from these sequential data update requests. Compelled by enticing incentives, the adversary could interact with this sequential update process to dynamically craft malicious update requests according to the model states, which can pose potential threats to the system. For the threat model, we here consider a very restricted setting where the adversary does not own any training data. Unlike the above static attacks, such sequential update scenario presents a crucial challenge of transiency. Specifically, at each time step, the adversary needs to make an irrevocable decision on whether to attack, and if he/she fails, or opts not to attack, then that data point is no longer available for further attacks. In the sequential setting, the model owner receives data update requests (i.e., {u, 1221) from the users sequentially, and is asked to update the model from these update requests. We let 2 = {2, 0} with a slight abuse of notation. Here, the t-th update request de is a tuple in (ot, 2 2 == ). where 2tra E Z, 2t =new E Z, and or E O = { ("Dlette" "Add", "Modify" } is a update instruction. Using these update requests, we can sequentially update the dataset and model defined below.
Definition 4 (Update sequences and sequentially updated models). Let U = =(u1, u2.-.. Me," denote the update sequence, where ut E O x Z x 2 for all t. Given the dataset and the t-th update request ue the update operation for Dt = Di-1 uu is defined follows
{ D. Dz-1 1 \ ztr
if "Delete") A (z/ra E DD-1)) A (z new = =0)
D. D U 2pen
if (o: "Add") A (2traa = -)) A (2 $ D.-1)
(4)
D. = (D.-1 27 U 2ppe,
if (o: = "Modify") A (z) E DD-1) A (zpew & D.-1).
We write Du D. For any 2 1, we write oe for the model input to the unlearning algorithm RA on time step t. We write 01 = A(Do). and for any t 2 1, 0 = RA((D,1,,0, 0t, u.). We write (Di to represent the sequence of updated datasets, {0: for the sequence of input models to RA-
The update sequences mentioned above align with the data update requests specified in 44 23 36). which focus on the add and delete requests. However, it is important to note that we extend these update requests to include the modify requests (74 as well, and this extension is necessary because some existing works also incorporate the modify update requests to enable the deletion of partial data information 664 37
17). The adversary's 's goal to force current updated model to satisfy certain desired properties at each time step while paying a small cost. For example, the ad adversary wants to force the current updated model to approach or maintain a target model @tar Then we can define the adversary's 's goal as Lude = ||0 (tar]] ||. We propose to formulate the sequential selective forgetting attacks as a Markov Decision Process (MDP) M = (S,A,T, R, 7), where
S is the state space. The state at time step is the stacked vector s; = [0,, uz)² consisting of the current model 0, and the incoming update request ue, where u, E EO x Z x Z. The state space is S = θ x O x Z x Z. We assume that the initial model 0g is fixed and known to the adversary while the first update request ti is sampled from P, i.c., the initial state distribution is defined as po u1) = P(u1).
A is the adversary action space. For each update request u4 we define discrete indication parameter K E (0, 1} to denote whether at time step the attack action should applied (k: = 1) or not (k; = 0). If the update instruction o, for u, is "Add", we assume that the adversary can only introduce imperceptible perturbations a, with the purpose of manipulating 2!" zhea for stealthiness. Thus, the attacked u, can be represented as ū, (oc, .tra 0, neew
5

---

2row + k2 - at). When ="Modify" and Ke = 1, we can write the attacked un as äة (or, 24 ,2 22 = 24 + k; : at), where a, the imperceptible perturbations. When u, is a delete update request and the adversary chooses to attack this delete update request (i.e., k = =)) we assume that the adversary has the ability to manipulate the update instruction by converting on ="Delete" into o, ="Modify", and the attacked u, is 0t = (ot; =t"0, " , zi + kt aa)) at). Here, the adversary maliciously modifies zf E D:-1 (instead of directly deleting 2! Note that the adversary can also choose not to attack this delete request.
T denotes the state transition function. The state transition function T :SS. S x A + As represents the conditional probability of the next state given the current state and attack action. We assume that the update function is deterministic. Therefore, the stochasticity arises solely from 4++1 within st+1. To provide a concrete example, we consider the modification step where corresponds to the action "Modify". In this case, the transition function can be derived as
(i), = P(ue+1|u, a.) ar) Pr = RA (D-11 T(s++1 a+) = 01
01, 0e) P(u++1 On as) Pr( g(u,, 01. (c.00a)))) ar) PP(u+11lu (wa, P2, Pr(Oe+1
O., ū.) = P(ue+1le [4, Oc, ax)) )= P(uct1).
(5)
Note that in the above modify case, given the attack action d,, we can obtain üN (oe, 2 , = (anew + az)). Discussions on other data update cases (also the proposed sequential optimization approach) can be found in the supplementary material.
R is the cost function. We define the cost at the time step t as R(s= [0,, ue]"" ki, az) =
10++1 12, which is determined by the current state and the attack action.
A policy is a function PMM: S + A that the adversary uses to choose the attack action a + (s [0., u,)²) based on the current victim model 0, and update request ū, Note that k; E {0. denotes whether at time step f the attack action should be applied. Now, the problem is how the adversary can find an effective attack strategy (kr with the corresponding attack action a, (generated by the policy network), which can maximize the adversary's ss goal (i.e., EM Σ ~si ke, (s)(s))) and minimize the number of attacked time steps (i.e., Σi= kt) for stealthiness. A naive way is to attack each step. However, attacking all the time steps would cause suspicion and expose the identity of the adversary. To address this, we formulate the below optimization to obtain the effective policy
min Eennn EM (Reee
RR(s: = 10,, ur] ke, a. (s)ss)))
hi
t=0
s.t., 02 = RA 0-- U4 1.11) E 10,
(6)
where &. is the policy network to be optimized. In the above first constraint, we use the notation u, without explicitly distinguishing whether the (-th update request is subjected to an attack or not.
However, directly solving the above optimization problem is highly challenging due the involve- ment of numerous variables. In addition, the environment data distribution for the formulated MDP is fixed but unknown to the adversary. To address the first challenge, we propose training an adversarial policy network Fade that takes the current state as input and outputs the attack strategy (pt, of). where Pt is the probability for taking the malicious action of. Specifically, for each step, the adversary gets the attack strategy (pt; (() from the adversarial policy (i.e., $ado 8: + (pt, d))). If pt 2 0.5, the adversary designates step as the critical point and introduces perturbations to mislead the model owner to trigger the action a/. Otherwise, the adversary does not attack the current time step. Additionally, the adversary can construct a progressively refined empirical distribution P. based on the sequence of observations W1:t- More precisely, at time 1, by replacing with P. and the model Og with lg, the attacker can construct substitute MDP Me = AA, Te, R, 7), solve for the optimal policy &. M. on Mr, and then apply the learned policy to perform the one-step attack. In this paper,
we solve the substitute MDP using deep deterministic policy gradient (DDPG) 33 35 to handle continuous action space.
Theorem 2. Let D'M and 4+ be the optimal policies for M and M respectively, with the same i initial state distribution po- We assume that the 1 I-Wasserstein distance between the estimated
distributions P and the true distribution P satisfies W1 (P,P) S v. Additionally, we assume the

---

loss function E. S + R exhibits Lipschitz continuity with respect to both and 2. with a constant L,, and Lipschitz smoothness with respect to 2, with constant L2- Moreover, we assume the loss function to possess strong convexity, strong smoothness, and twice continuous differentiability with respect to 8. Let Im M) be the cost in Eqn.
(6). Then there exist two constants V and such that:
IM (8)) IM (Dr)) S ( + V2LL),,
(7)
where Ç is a constant related with updating the model.
Discussions. The above theorem implies that the difference between JM (17) and Jm ( depends on the distribution difference v. In the above, we discuss our proposed selective forgetting attacks in the white-box setting, In the black-box setting, the adversary can randomly select substitute models and exploit the transferability property abilities or decision boundaries among different models. For instance, in approximate unlearning methods, the adversary can train one or several models to substitute the well-trained model l". This allows the adversary to generate malicious update requests and effectively transfer them to the target black-box victim model.
45 58 which arises from the shared vulner-
4 Experiments
In this section, we conduct comprehensive experiments to evaluate the effectiveness of our proposed selective forgetting attacks. All the experiments are run for 10 individual trials with different random seeds. Due to space limitations, a detailed description of the experimental setup, parameter settings, and more experimental results are given in the supplementary material.
Datasets and models. In experiments, we adopt the following real-world datasets: CIFAR-10 [1], Adult [15]. Diabetes [I]. and MNIST [12]. CIFAR-10 consists of 60,000 color images across 10 classes. Adult is downsampled to 23,374 samples with 57 features. Diabetes comprises 70,692 survey responses with 21 numerical features. MNIST contains 70,000 grayscale images of handwritten digits. In addition, we adopt synthetic dataset, which is binary dataset with 10,000 samples and 20 features and is generated based on the normal distribution. We also use range of machine learning models, including the logistic regression model, ResNet-18 25), VGG-16 [33. MobileNetV2 [50]. and neural network with two fully connected layers.
Attack settings. In experiments, we implement malicious selective forgetting attacks using the following unlearning methods: first-order based [64], second-order based [64), unrolling SGD (57), amnesiac (21). and SISA [4). In static forgetting attacks, we first pre-train the model and then perform selective forgetting attacks in targeted and untargeted settings. In the targeted setting, the adversary aims misclassify the input
specific target class, while in the untargeted setting, the adversary aims to mislead the model into predicting any incorrect class. In sequential forgetting attacks, we first generate the target model from static forgetting attacks and then force the victim model to be close to the target model. In experiments, we allow DDPG [35] to train once at the beginning to learn the optimal policy on the pre-attack data and then apply the learned policy to perform the one-step attack.
Baselines. In experiments, we adopt the RandSearch baseline, where we randomly select a set of training samples be forgotten in static forgetting attacks. In sequential selective forgetting attacks, we consider the no attack and random attack baselines. Specifically, the no attack baseline keeps the incoming data update requests unchanged, and the random attack baseline adds random data noise to modify the update requests.
4.1 Experimental Results for Static Selective Forgetting Attacks
First, we conduct experiments investigate the performance of malicious selective forgetting attacks in static manner. We adopt the attack success rate as the evaluation metric, defined as the number of successful attacks achieved among all attack attempts. For Lodv in Eqn. I we adopt the fe function in 77] Table [ summarizes the attack success rate of static forgetting attacks in the targeted setting via first-order, second-order, unrolling SGD, amnesiac, and SISA. Our proposed methods consistently achieve high attack success rates across diverse datasets and unlearning procedures. For example, our proposed methods hit an attack success rate of 1.0 on Diabetes and 0.8 on CIFAR-10. The reason is that our proposed methods can assign importance scores to training samples based on
7

---

Table 1: Attack success rate of static forgetting attacks in the targeted setting.
Diabetes
CIFAR-10
Unlearning method
RandSearch
Ours
RandSearch
Ours
First-order
0.04 + 0.03
1.00 + 0.00
0.08 + 0.04
0.80 + 0.04
Second-order
0.06 + 0.04
1.00 + 0.00
0.10 = 0.08
0.82 + 0.04
Unrolling SGD
0.04 + 0.03
1.00 + 0.00
0.06 = 0.03
0.78 + 0.06
Amnesiac
0.08 + 0.04
0.98 + 0.02
0.04 = 0.03
0.74 + 0.08
SISA
0.47 0.12
0.74 = 0.07
0.13 = 0.05
0.67 = 0.08
the impact of the targeted loss and unlearn optimal samples, resulting in substantial enhancements in attack performance. In contrast, the RandSearch bascline performs poorly in identifying the training samples to be removed for misclassifying the targeted test samples. These experimental results show the applicability and effectiveness of our optimization framework for static selective forgetting attacks across various unlearning methods, enabling us to achieve desired attack goals.
Next, we study the impact of underrep- resentation, which occurs when cer- tain classes in a dataset have fewer instances than others. In Figure 2a
Albock suceess rale
1.0
suocees rale
0.8
0.8
onter
0.6
we unlearn 1% to 5% of training sam-
opt
ples from CIFAR-10 10. In Figure 2b
Firdoorter weo upl
0.2
Sacondarder wi opt
Secanderree wi opt
we unlearn 5% to 25% of training
Secund order wif opt
Second wio opt
samples from Diabetes. The unlearn- ing samples are exclusively selected from the same class as the target class. We compare the attack performance with and without our proposed opti- mization framework. As illustrated, the attack success rates increase cor- respondingly with the percentage of unlcarning samples, as removing certain training data can hinder the model's ability to learn the target class. However, randomly removing small portion of data without optimization has minimal effect on the attack success rates. In contrast, our proposed methods can effectively identify the most critical training samples to be removed, even within specific class, enhancing the attack performance. Therefore, malicious selective forgetting attacks can leverage data underrepresentation to achieve the desired attack goals.
2%
3%
10%
15%
20% 25%
Percenlage uallranning samples
Porcentage of unlearning sumples
(a) CIFAR-10
(b) Diabetes
Figure 2: Attack success rate of selective forgetting attacks when data is underrepresented. comparing with and without optimization. The shaded area represents the standard error.
Then, we examine the impact of sub- populations, which are groups of sam- ples having similar features. Here, we focus misclassifying a particular subpopulation of targeted test samples.
Allack sucvess zale
1.0
0.8
Subpopalation tarqeted
Subeopulation, targeied
Upon identifying test sample for successful attack, we create a cluster consisting of points most similar to that sample based on the final repre- sentation layer in the network. We then apply the same malicious update requests attack subpopulations of varying sizes. As a baseline compar- ison, we randomly select group mem- bers of the same sizes. In Figure we adopt unrolling SGD on Diabetes and amnesiac on CIFAR- 10. We observe that malicious selective forgetting attacks are highly effective in attacking subpopulations of test samples in both targeted and untargeted settings. For instance, when using the unrolling SGD to attack a subpopulation of size 50 on Diabetes, the targeted attack success rate remains at 1.0. Similarly, with the amnesiac on CIFAR-10, the targeted attack success rate only drops to 0.85 for subpopulation size of 50. is worth noting that while the baseline randomly generated groups are not as good those formed based on similarity, the attack performance is still impressive.
Random group. Subpopalation untargeted
Bandom group. langeied
Sulpopulation untarpoted
Random group, untargeted
Randum untargeted
10
20
30
40
50
20
30
4U
50
Subpopulation size
Subpopalation size
(a) Diabetes
(b) CIFAR-10
Figure 3: Attack success rate of selective forgetting attacks across subpopulations of varying sizes.
8

---

Further, we explore the impact of data diversity on malicious selective for- getting attacks. We apply PCA pro- jection on input features and cluster the training data for each class. Then
0.0
12% ept
0.8
12% w. ept
52% ept
0.6
12% nustes
0.6
0.4
0.4
0.2
0.2
we formulate diversity by sampling different percentages of data points for each cluster. Figure 4 we un-
0.0
4
6
8
10
0.0
6
8
10
of unlcarning samples per cluster
unleamiin samples per cluster
(a) Diabetes, targeted
(b) Diabetes, untargeted
learn the same number of data points in each cluster and evaluate the at- tack performance with random selec- tion and our proposed optimization framework in targeted and untargeted settings. Firstly, our proposed meth- ods demonstrate the ability iden- tify the most influential data points within each cluster, leading substan- tial improvements in attack success rates compared the random removal of data points. Secondly, removing an equal number of critical data points
0.0
12 spt
1.0
0.6
0.8
52% mudie
0.4
0.6
12%
0.2
0.4
5 ic ept
12% sn muddoa
0.0
0.2
Gi mondoo
10
10
unicarming samples por cluster
unicaming samples per cluster
(c) CIFAR-10, targeted
(d) CIFAR-I0, 10, untargeted
Figure 4: Attack success rate of selective forgetting attacks with clusters of 10% data points (less redundant) and with clusters of 50% data points (more redundant).
from less redundant clusters has greater impact on the unlearned model, resulting in higher attack success rates than more redundant clusters. Consequently, unlearning training data different scopes of diversity can affect the performance of malicious selective forgetting attacks.
4.2 Experimental Results for Sequential Selective Forgetting Attacks
In this section, evaluate the performance of malicious selective forgetting attacks in the sequential update setting. We adopt the Euclidean distance metric to quantify the similarity between the victim model and the target model. Firstly, we examine the model convergence over attack time steps, where we naively apply attacks at each time step in a sequence of update requests. In Figure [ incorporate add, delete, and modify requests in a sequence of 300 update requests using the first-order unlearning method. As shown in the figure, our attack steadily reduces the Euclidean distance between the victim model and the target model over time steps on each adopted dataset (we use digits and 7 in MNIST). In contrast, the random attack baseline and the no attack baseline fail to approach the target model. Our learned adversarial policy demonstrates the ability attack the sequential data update requests by selecting the most effective actions within small difference to the requests, resulting in a minor difference between the updated model and the target model.
0.5
0.5
Buelldoan distonon
0.4
Enccidean de
Encliddan distance
0.4
0.4
0.3
Ourattock
sttack Random
attack
Random
0.3
0.3
Random attack
0.2
0.2
0.1
0
100
2000
300
0.1
0
100
200
300
0.1
100
200
300
Time step
Time stap
Time stop
(a) Synthetic
(b) Adult
(c) MNIST
Figure 5: Euclidean distance of the victim model to the target model at each time step.
Next, we investigate the performance variations corresponding to different attack steps within the sequential update process. In Figure 6 we compare corresponding models under different numbers of attack times with the optimal model (which is obtained by attacking all time steps and converges to the target model as shown in Figure The results show that our attack method consistently diminishes the Euclidean distance between the victim model and the optimal model as the number of attack times increase. Remarkably, our optimally leaned policy injects perturbations on some specific update requests only when necessary, leading to fewer attack times and achieving comparable performance as the optimal model. However, the random attack baseline does not contribute much to approaching the optimal model the number of attack times increase. Therefore, proposed

---

optimization framework proves stealthiness and effectiveness in identifying the critical attack time steps and inducing the effective attack actions the sequential update requests.
Fuelddon distanoa
kuciltean distance
Euclhdaan distance
0.3
0.3
0.3
0.2
Our atteck
0.2
Our artock
0.2
Our attack
Randons attuck
Sandoms uttock
Randons atiadk
0.1
0.1
0.1
100 Number times
100
Number times
Number of attack times
(a) Synthetic
(b) Adult
(c) MNIST
Figure 6: Euclidean distance of the victim model to the target under different numbers of attack times.
4.3 Black-box Experiments
In this section, we consider mali- cious selective forgetting attacks in the black-box setting. Firstly, we in- vestigate the transferability of selec- tive forgetting attacks across differ- ent machine learning models. In Ta- ble 2 we employ ResNet-18, VGG- 16, and MobileNetV2 in the untar- geted setting. As illustrated, our pro- posed methods demonstrate the abil- ity to transfer the generated malicious update requests to attack the black- box model, even though the black- box model trained with different model than the substitute model. For example, ResNet-18 can achieve at- tack success rates of 0.86 and 0.75 when transferred to VGG-16 and Mo- bileNetV2, respectively. Furthermore, we investigate the transferability of selective forgetting attacks between different unlearning methods. In T we apply first-order, second-order, and unrolling SGD unlearning methods in the targeted setting. We observe that malicious update requests generated by our proposed methods can also be effectively transferred different unlearning methods in the black-box setting. The reason is that with the same update requests, the unlearned models share similar decision boundaries among different models and unlearning methods.
Table 2: Attack success rates in the black-box setting using substitute models on CIFAR-10.
Black-box
ResNet-18
VGG-16
MobileNetV2
Substitute
ResNet-18
1.00 0.00
0.86
0.05
0.75 + 0.07
VGG-16
1.00
0.00
1.00 1 0.00
0.68 1 0.05
MobileNetV2
0.98 + 0.02
0.72 1 0.09
1.00 1 0.00
Table 3: Attack success rate the black-box setting using substitute unlearning methods on Diabetes.
Black-box First-order
Second-order Unrolling SGD
Substitute
First-order
1.00 + 0.00
0.88 + 0.08
0.96 + 0.03
Second-order
1.00 = 0.00
1.00 = 0.00
0.94 + 0.06
Unrolling SGD
1.00 + 0.00
0.94 = 0.03
1.00 0.00
Conclusion
In this paper, we examine the security vulnerability and resilience of machine learning models against selective forgetting attacks during the unlearning process, without compromising the integrity of the training and testing procedures, as commonly seen in traditional evasion attacks and data poisoning attacks. Specifically, we first present the general framework for selective forgetting attacks in the static setting, enabling the adversary to generate malicious whole data deletion requests. Additionally, we propose new approach for developing sequential selective forgetting attacks that can effectively compromise the unlearning system using the sequential data update requests. We also conduct theoretical analysis for the proposed selective forgetting attacks. The reported extensive experimental results demonstrate the effectiveness of our proposed selective forgetting attacks. We believe that our findings provide valuable insights into how to design secure and robust mechanisms to defend against selective forgetting attacks in the future.
10

---

References
[1] Behavioral risk factor surveillance system survey data, 2015.
[2] Manaar Alam, Hithem Lamri, and Michail Maniatakos, Get rid of your trail: Remotely crasing backdoors in federated learning. arXiv preprint arXiv:2304.106:3 2023.
[3] Maksym Andriushchenko and Matthias Hein. Provably robust boosted decision stumps and trees against adversarial attacks. Advances Neural Information Processing Systems, 32, 2019.
[4] Lucas Bourtoule, Varun Chandrasckaran, Christopher Choquette-Choo Hengrui Jia. Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papemot. Machine unlearning. In Proceedings of the 42nd IEEE Symposium on Security and Privacy, May 2021.
[5] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57. IEEE Computer Society, May 2017.
[6] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposimm, volume 267, 2019.
[7] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 39-57. leee, 2017.
[8] Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. In Proceedings of the 13th International Conference on Neural Information Processing Systems. MIT Press, 2000.
[9] Chong Chen, Fei Sun, Min Zhang, and Bolin Ding. Recommendation unlearning. In Proceedings of the ACM Web Conference 2022, WWW '22, page 2768-2777, New York, NY, USA, 2022. Association for Computing Machinery.
[10] Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. Graph unlearning. In ACM SIGSAC Conference Computer and Communications Security. ACM, November 2022.
[11] Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotant and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In 28th USENIX security symposium (USENIX security 19), pages 321-338 2019.
[12] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1916)144 2012.
[13] Jimmy Z. Di, Jack Douglas, Jayadev Acharya, Gautam Kamath, and Ayush Sekhari. Hidden poison: Machine unlearning enables camouflaged poisoning attacks. In NeurIPS ML Safety Workshop, 2022.
[14] Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modification. Advances in Neural Information Processing Systems, 34: 18944 2021.
[15] Dheeru Dua and Casey Graff. UCI machine learning repository. 2017.
[16] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-266, 2012.
[17] Rohit Gandikota, Joanna Materzynska, Jaden Kaattt,, and David Bau. Erasing concepts from diffusion models. arXiv preprint arXiv:2303.07345. 2023.
[18] Jonas Geiping, Liam H. Fowl, W. Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. In International Conference on Learning Representations, April 2021.
11

---

[19] Antonio A. Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. Making Al Forget You: Data Deletion in Machine Learning. 2019.
[20] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9301-9309 Seattle, WA, USA, June 2020. IEEE
[21] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516-11524 2021.
[22] Chuan Guo, Tom Goldstein, Awni Hamnun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3832-3842. PMLR, 13-18 Jul 2020.
[23] Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi- Maarif-Malvajerrdi and Chris Waites. Adaptive machine unlearning. Advances in Neural Information Processing Systems, 34:16319-1633 16330,, 2021.
[24] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26] Mengdi Huai, Jinduo Liu, Chenglin Miao, Liuyi Yao, and Aidong Zhang. Towards automating model explanations with certified robustness guarantees. In P Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6935-6943, 2022.
[27] Mengdi Huai, Tianhang Zheng, Chenglin Miao, Liuyi Yao, and Aidong Zhang. On the robustness of metric learning: an adversarial perspective. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(5):1-25 2022.
[28] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008-2016. PMLR, 2021.
[29] Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP). pages 19-35. IEEE, 2018.
[30] Masayuki Karasuyama and Ichiro Takeuchi. Multiple incremental decremental learning of support vector machines. IEEE Transactions on Neural Nenworks, 21(())1048-100 1059, 2010.
[31] Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on 1if Unpub- lished manuscript, 40(7):1-9, 2010.
[32] Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. Compas dataset. 2017.
[33] Henger Li, Xiaolin Sun, and Zizhan Zheng. Learning to attack federated learning: A model- based reinforcement learning attack framework. In Advances in Neural Information Processing Systems, 2022.
[34] Qizhang Li, Yiwen Guo, and Hao Chen. Practical no-box adversarial attacks against dnns. Advances in Neural Information Processing Systems, 33:12849-12860, 2020.
[35] Timothy P Lillicrap. Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations, 2016.
[36] Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. Conference on Lifelong Learning Agents, pages 243-254. PMLR, 2022.
12

---

[37] Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang. and Jianfeng Ma. Backdoor defense with machine unlearning. In JEEE INFOCOM 2022-IEEE Conference on Computer Communications, pages 280-289. IEFE, 2022.
[38] Neil G Marchant, Benjamin IP Rubinstein, and Scott Alfeld. Hard forget: Poisoning attacks on certified machine unlearning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7691-7700, 2022.
[39] Chenglin Miao, Qi Li, Houping Xiao, Wenjun Jiang, Mengdi Huai, and Lu Su. Towards data poisoning attacks in crowd sensing systems. In Proceedings of the Eighleenth ACM International Symposium on Mobile Ad Hoc Networking and Computing, pages 111-120, 2018.
[40] Fatemehsadat Mireshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh. Privacy in deep learning: A survey. arXiv preprini arXiv:2004. 12254, 2020.
[41] Saemi Moon, Seunghyuk Cho, and Dongwoo Kim. Feature unlearning for generative models via implicit feedback. arXiv preprint arXiv:2303.05602 2023.
[42] S. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deepfool: A simple and accurate method fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2574-2582 IEEE Computer Society, Jun 2016.
[43] Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdd Descent-to- Delette Gradient-Based Methods for Machine Unlearning. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, pages 931-962. PMLR, March 2021. ISSN: 2640-3498.
[44] Seth Neel, Aaron Roth, and Saeed Charifi-Malvajerts Descent-to-delete: Gradient- based methods for machine unlearning. In Algorithmic Learning Theory, pages 931-962. PMLR, 2021.
[45] Ehsan Nowroozi, Yassine Mckdad, Mohammad Hajian Berenjestanaki, Mauro Conti, and Abdeslam El Fergougui. Demystifying the transferability of adversarial attacks in computer networks. IEEE Transactions on Network and Service Management, 19(3):338733400, 2022.
[46] Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Accumulative poisoning attacks on real-time data. Advances in Neural Information Processing Systems, 34:2899-2913 2021.
[47] Stuart L Pardau. The california consumer privacy act: Towards curopean-style privacy regime in the united states. J. Tech. L. & Pol'y, 23:68, 2018
[48] Wei Qian. Chenxu Zhao, Wei Le, Meiyi Ma, and Mengdi Huai. Towards understanding and enhancing robustness of deep learning models against malicious unlearning attacks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1942-1942, 2023.
[49] Wei Qian, Chenxu Zhao, Huajic Shao, Minghan Chen, Fei Wang, and Mengdi Huai. Pa- tient similarity learning with selective forgetting. In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 529-534. IEEE, 2022.
[50] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 451 4520- 2018.
[51] Ali Shafahi, W Ronny Huang. Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Du- mitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. Advances neural information processing systems, 31, 2018.
[52] Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumi- tras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. Curran Associates Inc., 2018
13

---

[53] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:140..1566, 2014.
[54] Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In USENIX Security Symposium, volume I, page 4, 2021.
[55] Megha Srivastava, Hoda Heidari, and Andreas Krause. Mathematical notions human perception of fairnsss: A descriptive approach to fairness for machine learning. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2459-2468, 2019.
[56] Fnu Suya, Saeed Mahloujifar, Anshuman Suri, David Evans, and Yuan Tian. Model-targeted poisoning attacks with provable convergence. International Conference on Machine Learning, pages 10000010010 PMLR, 2021.
[57] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303-319. IEEE, 2022.
[58] Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv:1704.0345. 2017.
[59] Cheng-Hag Tsai, Chieh- Yen Lin, and Chih-Jen Lin. Incremental and decremental training for linear classification. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2014.
[60] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing. 10(3152676)100 5555, 2017.
[61] Xingchen Wan, Henry Kenlay, Robin Ru, Amo Blaas, Michael A Osborne, and Xiaowen Dong. Adversarial attacks on graph classifiers via bayesian optimisation. Advances in Neural Information Processing Systems, 34:6983 09983669966622 2021.
[62] Cheng-Long Wang. Mengdi Huai, and Di Wang. Inductive graph unlearning. arXiv preprint arXiv:2304.030093, 2023.
[63] Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine un- learning of features and labels. ArXiv, abs/2108..1577) 2021.
[64] Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine un- learning of features and labels. Network and Distributed System Security Symposium, 2023.
[65] Sandamal Weerasinghe, Tansu Alpcan, Sarah M. Erfani, and Christopher Leckie. Defending support vector machines against data poisoning attacks. IEEE Transactions on Information Forensics and Security, 092566-25773 2021.
[66] Cheng- Hsii Weng, Yan-Ting Lee, and Shan-Hung Brandon Wu. On the trade-off between adversarial and backdoor robustness. Advances in Neural Information Processing Systems, 13:11993-1193,, 2020.
[67] Xinyi Xu, Lingjuan Lyu, Xingjun Ma, Chenglin Miao, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Gradient driven rewards to guarantee fairness in collaborative machine learning. Advances in Neural Information Processing Systems, 34:16104-161 161 17, 2021.
[68] Xiang Zhang and Marinka Zimik. Gnnguard: Defending graph neural networks against adversarial attacks. Advances in neural information processing systems, 33:9263-9275, 2020.
[69] Xuezhou Zhang, Xiaojin Zhu, and Laurent Lessard. Online data poisoning attacks. In Learning for Dynamics and Control, pages 201-210. PMLR, 2020.
[70] Juexiao Zhou, Haoyang Li, Xingyu Liao, Bin Zhang, Wenjia He, Zhongxiao Li, Longxi Zhou, and Xin Gao. Audit forget: A unified method to revoke patients' private data in intelligent healthcare. bioRxiv, pages 2023-02, 2023.
14


Reviews:
[{"review_id": "qyUlyUi2zJ", "reviewer": "Reviewer_a1jN", "summary": "This paper identified a novel class of machine learning attacks, i.e.,  ML models can be manipulated with malicious data update requests during the machine unlearning process.\nThe authors study two threat scenarios: (1)  selective forgetting attacks and (2) sequential selective forgetting attacks.\nSpecifically, in the static setting, an adversary can first select a subset of accessible training data and then launch the unlearning request with the selected subset in order to induce the victim model to become close to a target model.\nIn this setting, the authors achieve the attack goal by casting the discrete data selection process into a continuous optimization process.\nOn the other hand, in the sequential setting, data update requests occur sequentially and the adversary can modify any update request before it is received by the victim.\nIn this setting, the authors propose to train a policy network via reinforcement learning to output attack strategies according to the current environment state.\nComprehensive experiments verify the effectiveness of the proposed selective forgetting attacks.", "questions": "1. What is the usage of Theorem 1? It seems like removing Theorem 1 would not affect the conclusion of Section 3.1.\n\n2. In Section 4, how to calculate the \"attack success rate\"? Is it the case that the authors first collect a set of data and then evaluate the victim's misclassification rate every time after processing an unlearn request?\n\n3. The setting of selective forgetting attacks seems a little similar to that in [r1]. It would be interesting to compare [r1] with this paper.\n\n**Reference**\n\n[r1] Shumailov et al. \"Manipulating SGD with Data Ordering Attacks\". arXiv 2021.", "limitations": "The main limitation of this paper is that the threatening scenario of sequential attacks may not be realistic to some extent, which has been explained in Section \"Weaknesses\".\nNevertheless, I believe the merits of this paper suppress its disadvantages and would significantly benefit the ML security community.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper identifies a novel and important ML threat, selective forgetting attacks. Such a threat is realistic in the real world and therefore worth the ML security community to continue further research.\n\n2. For the static version of the attack, how to efficiently select a subset to launch an attack is a challenging problem.\nThe authors address the problem by approximating the discrete indication function with a continuous function to perform GD-based optimization. This is a technically non-trivial solution.\n\n3. The problem formalization for sequential attacks is comprehensive, as it considers (almost) all possible types of update requests(\"delete\", \"add\", and \"modify\").", "weaknesses": "1. Two concerns about the sequential selective forgetting attacks:\n    - In the black-box setting, According to this paper, the adversary could not access any training data. Therefore, to let the substitute model better imitate the behavior of the victim model (in order to train an effective policy network), the adversary may need to collect data that are similar to the real data. However, this could be difficult in the real world when the adversary does not have any prior knowledge about the private training data.\n\n    - The adversary performs sequential attacks via modifying incoming update requests before they are received by the victim model, in which the modification strategy is by adding perturbations to the requested examples. This may result in a strange \"paradox\": if you modify the data that needs to be unlearned, is it still the original data? Wouldn't it result in the victim model \"unlearning\" something that it has never learned? Please comment.\n\n2. Suggestion: including results of membership inference attacks (MIA) would be interesting. It would be interesting to see how selective forgetting attacks will affect the success rate of MIA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your thoughtful comments and valuable suggestions. Below, we provide our response to the questions and concerns.\n\n**1: Discussions on training the substitute model in the black-box setting where the adversary does not have prior knowledge about training data.**\n\nThank you for addressing the black-box scenario, where the adversary lacks prior knowledge of private training data. In this setting, the adversary creates a synthetic dataset via existing model stealing attack techniques. They then train a local substitute model on this data, leveraging similar decision boundaries. Through traditional query methods, an ablation study was conducted on a three-layer ReLU neural network trained on MNIST as the target model. Table 1 indicates that the substitute model's performance approaches that of the target model more closely with an increasing number of queries. We also observed that their attack performance difference reduces as the query number increases. Query analyses will be included in the final version for this black-box setting.\n\nTable 1: Experiments on the discussed black-box setting.\n| Number of queries | Norm of parameter difference| Test accuracy difference | \n| :---: | :---: | :---: |\n| 2000 | 2.46 | 45.90% | \n| 4000 | 1.93 | 15.45% | \n| 6000 | 1.22| 9.68% | \n| 8000 | 0.67 | 5.85% | \n| 10000 | 0.19 | 2.51% | \n\n**2: Is it still the original data when modifying incoming update requests? Wouldn’t it result in the victim model “unlearning” something that it has never learned?.**\n\nThank you for your insightful questions. The difference between modified and original data depends on update magnitudes. As we discussed in lines 216-217 of the main manuscript, for the setting of modifying incoming update requests, we are following existing partial data deletion works [r2, r3]. These methods involve perturbing requested examples [r2]. The adversary can exploit this and use the excuse of bad data quality issues (e.g., noises) or privacy to generate specific misclassifications.\n\nRegarding unlearning, it only affects acquired training knowledge. Unlearning techniques remove the influence of unlearning information from the pre-trained model. If the model never learned from the infomation, its influence is close to zero.\n\n**3: Including experiments on how forgetting attacks affect the success rate of membership inference attacks (MIA).**\n\nThanks for highlighting this. Existing machine unlearning methods use MIA to measure the unlearning effectiveness. In contrast, our attack leverages existing unlearning techniques to induce misclassifications. Thus, MIA's success relies on how well unlearning methods remove maliciously requested unlearning samples.\nFollowing your suggestions, we evaluated membership inference attacks after maliciously unlearning certain CIFAR-10 training samples using various methods like first-order, second-order, unrolling SGD, amnesiac, and SISA. Based on [r4], we then compared these results with the baseline from the fully-trained original model. Table 2 displays the results. Notably, these unlearning methods offer similar performance deletion of malicious unlearning samples, as MIA metrics depend on the unlearning method's unlearning efficiency. To clarify, our work aims to study security vulnerabilities of the unlearning system by exploring the possibility of malicious data updates.\n\nTable 2: Experiments on membership inference attacks.\n| Model | Original | First-order | Second-order | Unrolling SGD | Amnesiac | SISA |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Accuracy | 1.0 ± 0.0 | 0.52 ± 0.06 | 0.54 ± 0.06 | 0.55 ± 0.04 | 0.59 ± 0.05 | 0.48 ± 0.05 |\n\n**4: Discussions on Theorem 1 and the conclusion of Section 3.1.**\n\nThanks for the question. In Theorem 1, we aim to study the difference between the original pre-trained model and its unlearned version (created after removing malicious unlearning samples). This helps understand the influence of these unlearning samples on the model's parameters. Notably, from this theorem, we can see that as more malicious unlearning samples are removed, the difference between the two models grows.\n\n**5: In Section 4, how to calculate the \"attack success rate\"? Is it the case that the authors first collect a set of data and then evaluate the victim's misclassification rate every time after processing an unlearn request?**\n\nIn our experiments, we randomly sample a target class to attack as well as a set of targeted test samples in this class. We then run our selective forgetting attacks by optimizing which training data to unlearn to meet attack objectives (targeted or untargeted setting). After unlearning, we assess the misclassification rate on the targeted test samples. This process is repeated 10 times with varied random seeds, and we report the average results and standard errors.\n\n**6: Comparing the setting of the proposed selective forgetting attacks and that of [r1].**\n\nThanks for highlighting the comparison with [r1]. Our method differs significantly from [r1]: First, [r1] targets training-time attacks, and assumes the adversary can alter the order of training data fed to the model. In contrast, our approach focuses on the unlearning process, and utilizes it to submit malicious update requests to achieve attack objectives.\nIn addition, the setting of [r1] can be viewed as a special case of our sequential unlearning attacks, since we can use the defined data update requests (refer to Definition 2 in the main manuscript) to achieve the attack goal of [r1] via changing the order in which batches are supplied to the model during training.\n\n**Reference**\n\n[r1] \"Manipulating SGD with Data Ordering Attacks\". arXiv 2021.\n\n[r2] “Machine Unlearning of Features and Labels”, NDSS 2023.\n\n[r3] “Feature Unlearning for Generative Models via Implicit Feedback”, arXiv 2023.\n\n[r4] “Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations”, ECCV 2020.\n"}, {"review_id": "OjrcX0GO7j", "reviewer": "Reviewer_MEi8", "summary": "This paper explores the malicious forgetting issue in model unlearning and proposes two attack strategies: static attack and dynamic sequential attack. The authors also present a theoretical framework for selective forgetting attacks. Experimental results on multiple benchmark datasets demonstrate that the proposed attack method poses a significant security threat to model unlearning.\n", "questions": "Refer to the weaknesses mentioned above.", "limitations": "The method does not exhibit apparent limitations. One main concern lies in the lack of convincing baseline comparisons in the experiments.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Model unlearning as an effective strategy for data forgetting has garnered significant attention, making the exploration of potential malicious attacks during the unlearning process an interesting research direction. \n- The paper has a clear research motivation, well-structured writing, and provides ample theoretical support. \n", "weaknesses": "- It is unclear whether this topic has been previously studied. The core of the proposed unlearning attack lies in a reasonable data sampling strategy for the victim model, which may not be novel from a technical standpoint. \n- It would be interesting to investigate if the defender, being aware of the malicious unlearning procedure, how could detect or filter out such requests to counter the attack. For example, could perturbation noise be added to mitigate the malicious impact of unlearning?\n- Recent study like Anti-backdoor Learning [1] uses two-stage unlearning techniques against backdoor attacks, whether this method could be applied to mitigate the negative effect of proposed unlearning attack?\n- The authors should discuss the potential limitations of this attack.\n\nI would be happy to improve the score if the authors address the aforementioned issues.\n\n[1] Yige Li, Xixiang Lyu, Xingjun Ma, et al, Anti-Backdoor Learning: Training Clean Models on Poisoned Data, NeurIPS, 2021\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your thoughtful comments and valuable suggestions. Below, we provide our response to the questions and concerns.\n\n**1: Discussions on whether this topic has been previously studied, and the technical standpoint.**\n\nThank you for your helpful suggestions regarding our work's novelty. As far as we know, we are the first to investigate how the adversary can exploit selective forgetting mechanisms to wholly and sequentially delete unlearning samples in an attempt to undermine the integrity and performance of the victim model. In the Related Work Section of the main manuscript, we give detailed comparisons with existing related works.\n\nFrom the technical standpoint, different from traditional empirical sampling methods, the proposed static and sequential attack frameworks use discrete indication variables to formulate the complete deletion of targeted training samples, which are very hard to be directly solved due to the presence of non-differentiable loss functions. To address the challenges, we first design a continuous and differentiable function to approximate the discrete component in the formulated static attack framework, and then optimize the proposed sequential unlearning attack framework by training an adversarial policy network, which is specifically designed to target and attack a few critical data update requests in the sequential data update setting. We believe that these technical advancements strengthen the contributions made in this paper.\n\n**2: How could detect or filter out such requests to counter the attack. Could perturbation noise be added to mitigate the malicious impact of unlearning?**\n\nThanks for the discussion on countering the attack detection. One strategy is to identify potential unlearning requests by tracking model parameter changes before and after unlearning. However, our experiments show that even with this method, our malicious unlearning attacks maintain high success rates. One reason is that these attacks are targeted at specific samples, not overall performance reduction. Another reason is that effective malicious data update requests cause slight model parameter changes, as observed. Another strategy involves using the maliciously unlearned model with other prediction models for ensemble-based methods. Yet, this requires additional models and learning costs. Also, recent studies highlight vulnerabilities in existing prediction models, making them susceptible to subtle attacks like data poisoning. Therefore, simultaneous attacks on multiple models could deceive ensemble systems.\n\nAdditionally, in Section 9.1 of the Supplementary Material, we experimented with adding adversarial perturbation noises using adversarial training techniques to assess our attacks against robust deep learning models. Despite the success of adversarial training in enhancing adversarial robustness, our attacks remain potent against these robust models, as evident in Figure 1 and Figure 2 in Section 9.1 of the Supplementary Material. This further emphasizes the significance of our research, as no prior work has explored security risks introduced by selective forgetting.\n\n**3: Whether Anti-backdoor learning [r1] could be applied to mitigate the negative effect of proposed unlearning attack?**\n\nThanks for your insightful question. We want to clarify that applying Anti-backdoor learning [r1] to mitigate our proposed unlearning attack is not feasible. Anti-backdoor learning assumes the presence of backdoored samples with triggers, focusing on trigger pattern identification and correlation removal. Our unlearning attacks do not involve the injection of triggers; we remove specific training samples rather than adding triggers. Furthermore, Anti-backdoor learning identifies low-loss backdoor examples early in training and removes correlations, while our attacks target malicious deletion during unlearning (instead of training). Consequently, the strategies of Anti-backdoor learning are unsuitable for mitigating our unlearning attacks.\n\n**4: Discussions on the potential limitations of this attack.**\n\nDue to space limitations in our main submission, we deferred the discussions on the potential limitations of our attack in Supplementary Material (see Section 2 in Supplementary Material). In the final version, we will incorporate the discussions on the potential limitations of our attacks from the Supplementary Material to the main manuscript.\n\n**5: Adding more convincing baseline comparisons in the experiments.**\n\nThank you for your suggestions. We followed your advice and conducted experiments with two new baselines: input space clustering-based unlearning and representation space clustering-based unlearning. These clustering-based methods remove specific sample types, differing from the RandSearch baseline. In Table 1, we present the attack success rates of these clustering baselines on CIFAR-10 using first-order and second-order unlearning. We divided training data into 250 clusters using K-means for each baseline, based on input and representation space distances. Unlearning involved randomly selecting clusters and ensuring an equal number of unlearned samples as our methods. The results show that clustering-based unlearning baselines still perform poor low attack success rates. However, our proposed methods significantly outperform them by a large margin. This is because random cluster unlearning fails to guide importance scores for training samples to influence targeted loss in selective forgetting attacks.\n\nTable 1: Attack success rate of our proposed static forgetting attacks and new baselines.\n| Unlearning method  | RandSearch | Input space | Representation space | Ours |\n| :---: | :---: | :---: | :---: | :---: |\n| First-order | 0.08 ± 0.04 | 0.04 ± 0.03 | 0.14 ± 0.08 | 0.80 ± 0.04 |\n| Second-order | 0.10 ± 0.08 | 0.08 ± 0.05 | 0.12 ± 0.05 | 0.82 ± 0.04 |\n\n**Reference**\n\n[r1] “Anti-Backdoor Learning: Training Clean Models on Poisoned Data”, NeurIPS 2021.\n"}, {"review_id": "sGiasnD9lZ", "reviewer": "Reviewer_EC3c", "summary": "This paper studies the malicious data update in machine unlearning. The authors consider two strategies. The first is static selective forgetting attack framework, where the adversary exploits vulnerabilities in the unlearning systems by submitting a set of carefully crafted data update requests at once. The second is sequential selective forgetting attack framework that injects malicious update multiple times by considering the order and timing of data update requests. Theoretical and experimental analysis are provided to validate the proposed machine unlearning attacks.", "questions": "See weakness.", "limitations": "Not applicable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThe angle of the problem is novel as there is no study on malicious machine unlearning so far.\n•\tThe authors propose one-step and multi-step attacks, and study the attack effect in white/black box, targeted/untargeted settings.\n•\tThe two proposed methods are well-justified both theoretically and experimentally.\n", "weaknesses": "•\tExperiment detail missing:\n      o\tWhat is target class in the targeted setting? \n      o\tWhat is the class that the machine unlearning aims to forget? The entire class or a few samples? If latter(according to line 344-346), what does forgetting mean in this case?\n      o\tWhat does the attack success rate mean, especially in the targeted setting? Reaching the targeted class or no longer on the previous class? \n      o\tFor untargeted setting, will the random class be the class for the task of machine unlearning? If so, what is the impact?\n      o\tWhat is the setting (untargeted/targeted) for black-box attack? The authors should specify in the manuscript.\n•\tRelated work to model poisoning should be added. While the authors mentioned data poisoning attacks, the proposed attack in machine unlearning is closer to model poisoning, which embeds the attack goal with the benign training objectives. Likewise, the proposed attack embeds the attack goal into the unlearning objective.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your thoughtful comments and valuable suggestions. Below, we provide our response to the questions and concerns.\n\n**1: Providing more experiment details: (1) What is the target class in the targeted setting? (2) What is the class that the machine unlearning aims to forget? The entire class or a few samples? If the latter (according to lines 344-346), what does forgetting mean in this case? (3) What does the attack success rate mean, especially in the targeted setting? Reaching the targeted class or no longer on the previous class? (4) For an untargeted setting, will the random class be the class for the task of machine unlearning? If so, what is the impact? (5) What is the setting (untargeted/targeted) for black-box attack? The authors should specify in the manuscript.**\n\nSorry about the unclear descriptions regarding these mentioned experiment details. Below, we address your concerns one by one.\n\n(1) In the targeted setting, the \"target class\" refers to the class that the adversary aims to attack. For example, the motivated adversary could generate malicious data update requests to attack the targeted test samples and force the targeted test sample (e.g., the bird image) to be assigned as the attack targeted label (e.g., the dog label).\n\n(2) To be clarified, our goal is not to forget certain classes. Instead, we aim to use machine unlearning techniques to forget specific training samples and their corresponding influence on the pre-trained model. In lines 344-346 of the main manuscript, we conducted selective forgetting attacks to misclassify the targeted test samples by unlearning a subset of training data (the percentages indicate the total portion of training samples to be unlearned).\n\n(3) The attack success rate means the number of successful attacks achieved over all attack attempts. Specifically, in the targeted setting, a successful attack means the targeted test sample is misclassified into the adversary’s specified label on the victim model.\n\n(4) In the untargeted setting, the predicted label of the targeted test sample is changed to any label other than the true label, after unlearning a subset of training data. This can lead to misbehaviors on the victim model, such as degrading the classification accuracy of specific users. Consider an example where existing unlearning techniques are utilized to repair a face recognition system. During this process, in the untargeted setting, the adversary could make malicious data update requests to cause the repaired face recognition system to misidentify their intended target as anyone else other than the true identity. Lastly, it is important to emphasize that our task is not to select a random class and then perform machine unlearning to completely forget that specific class.\n\n(5) In the black-box attack, we consider a setting where we have no prior knowledge about the target pre-trained victim model. Therefore, we explore the transferability of selective forgetting attacks across various machine learning models. In detail, we can generate malicious data update requests in the targeted setting (where the predicted label is changed to a specified one) and the untargeted setting (where the label is changed to an incorrect one) on one surrogate model, and then transfer these malicious data update requests to attack the black-box victim model. \n\n**2: Adding the related work to model poisoning attacks.**\n\nThank you for your valuable suggestion. In the final version, we will incorporate related work on model poisoning attacks. Here, we want to clarify that our proposed selective forgetting attacks and model poisoning attacks differ significantly in terms of attack timing and attack mechanisms. Specifically, our attacks occur during the unlearning process, while model poisoning attacks usually take place during learning (e.g., [r1] assuming that the adversary directly attacks the model parameters during the learning process) or fine-tuning (e.g., [r2] involving a model fine-tuning step to attack the pre-trained model via the maliciously designed penalty term for fine-tuning). Additionally, our attacks utilize existing unlearning techniques to delete certain unlearning samples, whereas model poisoning attacks modify model parameters through means such as parameter manipulation [r1]. We appreciate this constructive suggestion and will ensure this suggested related work discussion in the final version.\n\n**Reference**\n\n[r1] “Local model poisoning attacks to Byzantine-Robust federated learning”, USENIX Security 2020.\n\n[r2] “Fooling Neural Network Interpretations via Adversarial Model Manipulation”, NeurIPS 2019.\n"}, {"review_id": "eWESZn5lcK", "reviewer": "Reviewer_MTyE", "summary": "This paper investigates the potential and viability of malicious data update requests in the context of the unlearning process. The authors put forward a malicious selective forgetting attack in a static scenario and present a framework for sequential forgetting attacks. And the framework is formulated as a stochastic optimal control problem.", "questions": "The attack goal of the proposed approach remains unclear. While the paper discusses constructing malicious data update requests during the unlearning process, it does not explicitly define the specific objectives pursued by the attacker. It is essential for the authors to provide concrete explanations regarding the attack goals, rather than merely stating them as desired outcomes. For instance, is the intention to diminish the fairness of the unlearning model or to deliberately misclassify specific samples?Furthermore, the paper lacks a high-level description of how the attack is executed, given a clear attack target. It is important for the authors to outline the general methodology employed by the attacker to carry out the attacks. \n\nThe attack methodology of the proposed approach lacks clarity. While the authors present multiple definitions, lemmas, and theorems in the methodology section, they do not provide an overall step-by-step explanation of how the attack is conducted. As a result, obtaining a clear understanding of the attack process remains challenging.\n\nAn overview image is also required to describe the proposed attacks.\n\nThe authors assert the effectiveness of their method in both white-box and black-box scenarios. However, the methodology section lacks a clear explanation of how the attack steps differ in these two scenarios. \n\nMore experiments on high-resolution datasets, such as ImageNet, VGG-Flower, that include more classes, should be evaluated. Otherwise, it is not sure whether the proposed method is applicable in the real-world scenario.\n\n", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper investigates the potential and viability of malicious data update requests in the context of the unlearning process. \n", "weaknesses": "The attack goal of the proposed approach remains unclear. \n\nThe attack methodology of the proposed approach lacks clarity.\n\nThe authors assert the effectiveness of their method in both white-box and black-box scenarios. However, the methodology section lacks a clear explanation of how the attack steps differ in these two scenarios. \n\nMore experiments on high-resolution datasets, such as ImageNet, VGG-Flower, that include more classes, should be evaluated. Otherwise, it is not sure whether the proposed method is applicable in the real-world scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your thoughtful comments and valuable suggestions. Below, we provide our response to the questions and concerns.\n\n**1: Clarifying the attack goal (e.g., diminishing the fairness of the unlearning model, or misclassifying specific samples?) of the proposed approach. Giving a high-level description of how the attack is executed.**\n\nFirst, we want to clarify that our proposed attacks are general and can offer great flexibility to accommodate the adversary's desired diverse attack objectives through the malicious unlearning samples (Please refer to lines 136-138 in Section 3.1 of the main manuscript). In this paper, we present novel static and sequential selective forgetting attack frameworks that empower the adversary to achieve desired attack goals, such as causing unfairness for specific groups of individuals or inducing misclassifications, by crafting malicious data update requests.\n\nAdditionally, the high-level descriptions of the proposed attacks are deferred to Section 4 in the Supplementary Material due to space limitations during the preparation of our submission. For example, in Algorithm 1 presented in the Supplementary Material, we illustrate the high-level descriptions of how to solve the proposed attack optimization framework using the second-order unlearning strategy described in Eqn. (1) and (3) of the main manuscript. In the final version, we will integrate these high-level descriptions for the attacks from the Supplementary Material into the main manuscript.\n\n**2: Clarifying the attack methodology of the proposed approach by providing the step-by-step explanations of how the attack is conducted.**\n\nDue to space limitations during the preparation of our submission, in Section 4 of the Supplementary Material, we provide the algorithms (e.g., Algorithm 1, 2, 3, and 4) to give detailed explanations of how the attacks are conducted in the static and sequential settings. In the final version, we will incorporate these high-level descriptions from the Supplementary Material into the main manuscript.\n\nAdditionally, in the one-page PDF in the \"global\" response, we have included the attack flowchart (please refer to Figure 2 and Figure 4 in this one-page PDF) for our attacks in the static and sequential settings.\n\n**3: Providing an overview image to describe the proposed attacks.**\n\nThanks for pointing this out. In the attached one-page PDF in the “global” response, following your suggestion, we provide an overview image (please refer to Figure 1 and Figure 3  in this one-page PDF) to describe the proposed attacks in the static and sequential attack settings.\n\n**4: Giving more explanations of how the attack steps differ in the black-box and white-box scenarios.**\n\nThanks for this suggestion. In our paper, we implement our selective forgetting attacks in both white-box and black-box scenarios. In the white-box setting, we assume that both the model architecture and parameters are known to us. Leveraging this knowledge, we can directly generate malicious data update requests for specific test samples on the pre-trained model. In the black-box attack, as we don't know any information about the model, we first need to train one or several surrogate models to substitute the pre-trained model, and then transfer selective forgetting attacks to the black-box victim model. In detail, we generate malicious data update requests on one model and apply these malicious data update requests to attack the black-box victim model. \n\n**5: Including more experiments on high-resolution datasets, such as ImageNet, and VGG-Flower.**\n\nWe greatly appreciate the suggestions for adding experiments on ImageNet and VGG-Flower. Following your suggestions, we conducted experiments on ImageNet [r1] and Flowers [r2], with high-resolution images and more classification classes. In experiments, we trained ResNet-18 on a sub-dataset of ImageNet and VGG-19 on Flowers. Following the same experimental setting in Section 4.1 of the main manuscript, we report the attack success rate of our proposed static forgetting attacks in the untargeted setting, and we compare the results with the RandSearch baseline. As shown in the below table, our proposed methods also achieve high attack success rates on these two datasets using the first-order and the rolling SGD unlearning methods. On the other hand, the RandSearch baseline demonstrates limited effort in misclassifying targeted test samples in an untargeted manner. In the final version, we will include the experiments on high-resolution datasets.\n\nTable 1: Attack success rate of our proposed static forgetting attacks on ImageNet and Flowers.\n| Dataset  | Unlearning method | RandSearch | Ours |\n| :---: | :---: | :---: | :---: |\n| ImageNet | First-order | 0.26 ± 0.08 | 0.96 ± 0.04 |\n| ImageNet | Unrolling SGD | 0.24 ± 0.07 | 0.94 ± 0.04 |\n| Flowers | First-order | 0.46 ± 0.13 | 1.00 ± 0.00 |\n| Flowers | Unrolling SGD | 0.35 ± 0.10 | 0.95 ± 0.05 |\n\t\n**Reference.**\n\n[r1] \"Imagenet: A large-scale hierarchical image database\", CVPR 2009.\n\n[r2] \"Automated flower classification over a large number of classes\", ICVGIP 2008.\n"}]

Author rebuttal:
 We would like to express our great gratitude to all the reviewers for their valuable time, comments, and questions. We highly appreciate all the feedback and suggestions, which further help us improve our paper. We are greatly encouraged that they found our ideas and contributions to be novel and significant (Reviewers a1jN, MEi8, EC3c, and MTyE), comprehensive (Reviewers a1jN, MEi8, and EC3c), and technically sound (Reviewers a1jN, MEi8, and EC3c). We are grateful that they recognized the effectiveness of our methods (Reviewers a1jN, MEi8, EC3c, and MTyE), and the comprehensiveness of our experiments (Reviewers a1jN, MEi8, and EC3c).
	
We carefully considered all concerns and questions raised by the reviewers and provided our detailed responses to these concerns and questions raised in individual comments. We hope that our responses will address the concerns and questions provided by the reviewers. We look forward to participating in further discussions and answering any further questions posed by the reviewers.
                   
Thank you for your time and consideration.

Best,

Authors of Paper5786


Meta-review: This paper studies a threat model where an adversary attacks a machine learning system via malicious data update requests during the unlearning process. Unlearning is an interesting and new protocol in machine learning with practical applications. This paper makes a first attempt at evaluating potential and viability of malicious unlearning attacks. The proposed methods are technically sound and systematically evaluated. Additional results on higher resolution datasets added during the rebuttal further strengthen the empirical support.
Decision: Accept (poster)