Title: Static and Sequential Malicious Attacks in the Context of Selective Forgetting
Venue: NeurIPS 2023
Accepted: yes
Keywords: Selective forgetting

static setting

sequential setting

security and robustness
Abstract: With the growing demand for the right to be forgotten, there is an increasing need for machine learning models to forget sensitive data and its impact. To address this, the paradigm of selective forgetting (a.k.a machine unlearning) has been extensively studied, which aims to remove the impact of requested data from a well-trained model without retraining from scratch. Despite its significant success, limited attention has been given to the security vulnerabilities of the unlearning system concerning malicious data update requests. Motivated by this, in this paper, we explore the possibility and feasibility of malicious data update requests during the unlearning process. Specifically, we first propose a new class of malicious selective forgetting attacks, which involves a static scenario where all the malicious data update requests are provided by the adversary at once. Additionally, considering the sequential setting where the data update requests arrive sequentially, we also design a novel framework for sequential forgetting attacks, which is formulated as a stochastic optimal control problem. We also propose novel optimization algorithms that can find the effective malicious data update requests. We perform theoretical analyses for the proposed selective forgetting attacks, and extensive experimental results validate the effectiveness of our proposed selective forgetting attacks. The source code is available in the supplementary material.

Static and Sequential Malicious Attacks in the Context of Selective Forgetting

Abstract

With the growing demand for the right to be forgotten, there is an increasing need for machine learning models to forget sensitive data and its impact. To address this, the paradigm of selective forgetting (a.k.a machine unlearning) has been extensively studied, which aims to remove the impact of requested data from a well-trained model without retraining from scratch. Despite its significant success, limited attention has been given to the security vulnerabilities of the unlearning system concerning malicious data update requests. Motivated by this, in this paper, we explore the possibility and feasibility of malicious data update requests during the unlearning process. Specifically, we first propose a new class of malicious selective forgetting attacks, which involves a static scenario where all the malicious data update requests are provided by the adversary at once. Additionally, considering the sequential setting where the data update requests arrive sequentially, we also design a novel framework for sequential forgetting attacks, which is formulated as a stochastic optimal control problem. We also propose novel optimization algorithms that can find the effective malicious data update requests. We perform theoretical analyses for the proposed selective forgetting attacks, and extensive experimental results validate the effectiveness of our proposed selective forgetting attacks. The source code is available in the supplementary material.

1 Introduction

Machine learning algorithms play a crucial role in diverse fields such as biology, speech recognition, agriculture, and medicine. To build pertinent models, these algorithms are frequently trained using a range of data sources, including third-party datasets, internal datasets, and customized subsets of publicly available user data. With recent demands for increased data privacy, the data users could erase the impact of their sensitive information from the trained models to ensure their privacy. Recent legislation (e.g., the General Data Protection Regulation from the European Union [60], the California Consumer Privacy Act [47], and the Canada's proposed Consumer Privacy Protection Act) requires the right to be forgotten, and grants users an unconditional right to request that their private data be removed from everywhere in the system within a reasonable time.

However, with the development of traditional machine learning techniques, this basic right is usually neglected or violated [70, 19, 38]. An illustrative instance is the inadvertent leakage of patients' genetic markers through machine learning methods employed for genetic data processing without the patients' awareness. Therefore, it is important to entitle data users the right to delete their personal data from trained machine learning models since machine learning models could memorize sensitive information of the training data and thus expose individual's privacy risk [54, 6, 8, 30, 59, 4, 40]. The most naive way is to retrain from the original data after removing the samples that need to be forgotten. Unfortunately, this naive retraining method can be prohibitive in terms of the computational and space cost-especially for large models and frequent deletion requests. To mitigate this, selective forgetting (a.k.a machine unlearning) [4, 9, 43, 22, 20, 28, 63] has been extensively researched in recent years to avoid the high computational cost associated with fully retraining a model from scratch.

However, existing studies on selective forgetting mainly focus on designing new forgetting algorithms to enable the right to be forgotten to be efficiently implemented, leaving the security issues during the unlearning process in adversarial settings largely unexplored. In practice, the motivated adversary could make use of the unlearning pipeline to craft malicious data update requests to achieve his/her desired attack goals. For example, the motivated adversary could increase a disadvantage against a specific group of individuals. In Figure 1, we present a toy example to highlight the impact of malicious data update requests on fairness using the COMPAS [32] dataset. Note that fairness in machine learning [67, 55] refers to the concept of ensuring that the decisions produced by machine learning algorithms are unbiased and equitable across different groups of individuals,

0%

10%

20%

30%

40%

50%

Percentage of unlearning samples

0.25

0.30

0.35

Fairness gap

Demographic Parity

Equalized Odds

irrespective of their protected attributes (e.g., race and ethnicity). Here we consider race (black/white) as the sensitive feature and just randomly unlearn varying percentages of samples from the minority group (white). For the fairness evaluation metrics, we adopt Demographic Parity [16] and Equalized Odds [24] (please refer to the supplementary material for details of the two evaluation metrics and more experimental results on attacking fairness). From this figure, we can easily see that even though we just randomly delete some individuals from the white group in the dataset, the fairness gap expands, indicating that the model's fairness is compromised. When the victim systems are employed for security-sensitive applications, such malicious data update requests can cause tremendous security threats to the unlearning system. Therefore, it is essential to understand the feasibility of malicious data requests in the unlearning systems.

In this paper, we aim to conduct a comprehensive study on the security vulnerability and robustness of the unlearning system to malicious data update requests during the unlearning process. Specifically, we first propose a novel static selective forgetting attack framework, where the adversary exploits vulnerabilities in the unlearning systems by submitting a set of carefully crafted data update requests at once. More specifically, the proposed static attack framework uses discrete indication variables to formulate the complete deletion of targeted training samples, which is very hard to be directly solved. To address this challenge, we design a continuous and differentiable function to approximate the discrete component. On the other hand, many real-world applications involve streaming data update requests that arrive in a sequential manner. The adversary could take advantage of this sequential interaction setting to strategically manipulate the unlearning process. However, launching attacks on all the received sequential data update requests indiscriminately can potentially lead to the detection of the adversary. To address this issue, we also design a novel sequential selective forgetting attack framework that takes into account the order and timing of data update requests. In this framework, the adversary focuses on attacking a few critical data update requests to maximize the impact of his/her malicious actions, potentially leading to severe security threats. We also conduct theoretical analyses of our proposed selective forgetting attacks. Further, we conduct extensive experiments in different scenarios to validate the general effectiveness of our proposed selective forgetting attacks.

2 Related Work

Selective forgetting (a.k.a machine unlearning and data deletion ) [20, 38, 19, 38, 49, 62] refers to removing the influence of the requested data from a trained model. However, existing data deletion works [19, 19, 8, 30, 59, 4, 10, 9, 43, 22, 20, 28, 63] ignore the risks of malicious data update requests during the unlearning process and fail to identify the vulnerabilities of machine learning models to selective forgetting attacks. It is worth mentioning that our proposed selective forgetting attacks are different from traditional evasion attacks [66, 68, 3, 34, 61, 27, 69, 26] and data poisoning attacks [46, 14, 51, 39]. Evasion attacks achieve adversarial goals by modifying test samples; however, our forgetting attacks do not change test data and only modify the well-trained models by making malicious data update requests. Traditional data poisoning attacks occur during training, and manipulate the original clean training data. Note that [13] adds carefully crafted samples to the training dataset, and assumes the exact and retraining tasks. The authors in [48] fail to address the issue of malicious whole data deletion in the static setting, nor do they consider the sequential attack setting involving various types of malicious data update requests.

3 Malicious Selective Forgetting Attacks

Without loss of generality, in this work, we consider the classification models. Let D = { z i = ( x i , y i ) } N i =1 denote the training dataset from Z = X ×Y , where x i ∈ R d 1 is a d 1 -dimensional sample and y i ∈ [ C ] denotes its associated class label. The model owner applies a learning algorithm A on D to learn a model f ( θ ) parameterized by θ , such that f ( θ ) achieves low empirical loss. We denote a loss function by a mapping ℓ : Θ ×Z → R that takes the parameters θ ∈ Θ and a sample z ∈ Z , outputs the loss of θ on z (i.e., ℓ ( z ; θ ) that we denote by ℓ z ( θ ) ). Let A ( D ) denote the distribution over the resulting model parameters θ when the learning algorithm A is applied on D .

Threat Model. As previously mentioned, the proposed malicious selective forgetting attacks are unlearning time attacks, i.e., the adversary interferes with the unlearning process of the well-trained model and cannot modify any test sample submitted to the victim model at testing time. In addition, the adversary is unable to modify the training samples during the training stage. This reflects the unlearning scenario in which the adversary can only generate the update requests to selectively forget certain data information during the unlearning process. In this paper, we study both the white-box and black-box settings. Specifically, in the white-box setting, we make the assumption that the adversary possesses complete information about the system (including the model architecture and parameters of the well-trained model). Note that many evasion and poisoning attacks in the literature [5, 42, 65, 52, 18, 33, 29] employ a white-box model to study the adversary's strong attack behaviors in such worst-case settings. In the black-box setting, we assume that the adversary does not have any prior knowledge about the target well-trained model.

3.1 Static Selective Forgetting Attacks

Here, we consider the static attack setting, where the model holder owns a well-trained classification model f ( θ ) parameterized by θ ∈ R d 2 on dataset D , and the adversary aims to make malicious update requests to deliberately forget some information to achieve his/her desired attack goals. In this case, all the malicious data update requests are provided at once, without consideration for the sequential order. We use D f ⊂ D and R A to denote the adversary's requested forget set and the coupled unlearning algorithm for A , respectively. Note that for the given forget set, machine unlearning method can return a model θ ∼ R A ( D,A ( D ) , D f ) , which posses no information about D f without influencing the contributions of other data. The following definition gives the definition of existing approximate and exact unlearning methods [64, 44, 23].

Definition 1 (Machine unlearning) . Let A and R A denote the learning algorithm and the unlearning method, respectively. The pair ( A,R A ) achieves exact unlearning if ∀ D,D f ⊂ D,A ( D r ) = d R A ( D,A ( D ) , D f ) , where D r = D \ D f and = d means the same distribution. This means that if the unlearned model from R A ( D,A ( D ) , D f ) has no information about D f , we cannot differentiate the model after forgetting from a model that is obtained on D r . The pair ( A,R A ) satisfies ( ϵ, δ ) -unlearning if ∀ D,D f ⊂ D, and E ⊂ R d 2 , P ( R A ( D,A ( D ) , D f ) ∈ E ) ≤ e ϵ P ( A ( D r ) ∈ E ) + δ , where A and D f denote the learning algorithm and the forget set, respectively.

As aforementioned, the adversary aims to make malicious update requests for his/her desired attack goals. For example, the adversary could attack the targeted test samples and force them to be assigned as the attack targeted label [7]. The adversary could also have a specific target model θ tar (e.g., the unfair and backdoored models) in mind and aim to induce a victim model as close as possible to that target model θ tar [56]. The effective unlearning samples (i.e., D f ⊂ D ) can be obtained by solving the following formulated optimization problem

In the above, D t represents a subset of D that is accessible to the adversary, and θ u ( D f ) are the parameters found by completely eliminating specific targeted samples D f ⊂ D t = { ( x p , y p ) } P p =1 . For each x p ∈ D t , we define a discrete indication parameter ω p ∈ { 0 , 1 } to indicate whether the sample x p should be completely deleted ( ω p = 1 ) or not ( ω p = 0 ). The forget set D f to be unlearned is denoted as D f = D t ◦ Ω = { x p | x p ∈ D t and ω p = 1 } , where Ω = { ω p ∈ { 0 , 1 }} P p =1 . Note that the above equation is a bi-level optimization problem - the minimization for D f involves the model parameters θ u ( D f ) , which are themselves the minimizer of the training problem,

Note that Eqn. (1) and (2) provide the high-level formulation for selecting subset D f such that the adversary's goals are maximized after unlearning. However, it is very difficult to optimize the effective update requests due to the introduced discrete indication parameters (i.e., Ω = { ω p ∈ { 0 , 1 }} P p =1 ) of the first constraint in Eqn. (2). Next, we take the second-order unlearning strategy proposed in [64] as an illustrative example to show how to solve the above formulated optimization problem. This unlearning strategy uses the inverse Hessian matrix of the second-order partial derivatives to change the original model's parameters to obtain the unlearned model [64]. For D t = Z = { z p } P p =1 ⊂ D , we use ˜ D t = ˜ Z = { ˜ z p } P p =1 to denote its corresponding unlearned versions, where ˜ z p = ( x p -ξ p , y p ) and ξ p is the unlearning modification for x p . Following [64], we calculate the second-order change ∆( Z, ˜ Z ) by calculating all the gradients difference between Z and ˜ Z with a weighting change from the inverse Hessian of the loss function, i.e., ∆( Z, ˜ Z ) = H -1 θ ∗ ( ∑ ˜ z p ∈ ˜ Z ω p ∗ ∇ θ ℓ (˜ z p ; θ ∗ ) -∑ z p ∈ Z ω p ∗∇ θ ℓ ( z p ; θ ∗ )) , where H -1 θ ∗ is the inverse Hessian matrix and ω p ∈ { 0 , 1 } denotes whether z p should be completely erased. To address the aforementioned challenge, we propose to relax each discrete variable ω p into a continuous one of range [0 , 1] , i.e., ω p ∈ [0 , 1] , and then approximate the original update objective for this second-order strategy by the following one

̸

where ℓ is a training loss, and ω p ∈ [0 , 1] . Here, we rewrite ω p as 1 2 (1+ sgn (2 ∗ ω p -1)) . Based on the fact that function h 1 ( x ) = 1 2 (1 -sgn ( x )) can be approximated by function h 2 ( x ) = 1 -1 1+exp( -φx ) , we can obtain the above equation. The parameter φ in h 2 represents the steepness of the curve. Additionally, the continuous property of h 2 allows us to solve the formulated optimization to perform selective forgetting attacks via the second-order unlearning strategy in [64]. From Section IV in [64], we can know that if ˜ Z = ∅ , the adversary has the option to completely remove the targeted sample or retain it, based on whether ω p ≥ 0 . 5 or ω p < 0 . 5 . When ˜ Z = ∅ , the adversary can intentionally and maliciously modify the targeted training samples via partially unlearning some data information based on Eqn. (3). Notably, following Lemma 1, we can easily generalize the above proposed attack framework to the scenario where the adversary wants to maliciously erase some features [64]. The algorithm and generalization to other unlearning methods are deferred to the supplementary material.

Lemma 1 ([64]) . Let F denote the features to be unlearned. Let θ ∗ -F denote the optimal model retrained on the new dataset that is derived by removing the features F from D . For learning models processing inputs x using the linear transformations of the form θ T x , we have θ ∗ -F ≡ θ ∗ F =0 , where θ ∗ F =0 is retrained by setting the values of the features F to zero in D .

Definition 2 (Strongly convexity) . A function ψ : R d 3 → R d 4 is said to be M -strongly convex for some M ≥ 0 if for any z 1 ∈ R d 3 , z 2 ∈ R d 3 , and any q ∈ (0 , 1) , ψ ( qz 1 + (1 -q ) z 2 ) ≤ qψ ( z 1 ) + (1 -q ) ψ ( z 1 ) -M 2 q (1 -q ) || z 1 -z 2 || 2 2 . Note that if the above condition is satisfied for M = 0 , we refer to the function ψ as convex.

Definition 3 (Lipschitzness continuity) . A general function ψ : R d 3 → R d 4 is L -globally Lipschitz continuous if for all z 1 ∈ R d 3 , z 2 ∈ R d 3 , || ψ ( z 1 ) -ψ ( z 2 ) || ≤ L || z 1 -z 2 || 2 .

Theorem 1. Let θ ∗ D = arg min θ ∈ Θ ℓ D ( θ ) for any given dataset D . Suppose that the loss function ℓ z is L -globally Lipschitz continuous and M -strongly convex for any z ∈ Z . For any integer N , dataset D of size N , and forget set D f , we can have that || θ ∗ D -θ ∗ D \ D f || 2 ≤ 2 L MN | D f | , where | D f | represents the size of the forget set.

The above lemma shows that we can erase features from many learning models by first setting them to zero [64]. The above theorem relates the difference of the model parameters to the forget set, and can be easily generalized to the defined data update requests in Definition 4.

3.2 Sequential and Dynamic Selective Forgetting Attacks

In practice, the model owner usually receives sequential update requests from one or more data owners at different times, and is asked to update the model from these sequential data update requests. Compelled by enticing incentives, the adversary could interact with this sequential update process to dynamically craft malicious update requests according to the model states, which can pose potential threats to the system. For the threat model, we here consider a very restricted setting where the adversary does not own any training data. Unlike the above static attacks, such a sequential update scenario presents a crucial challenge of transiency. Specifically, at each time step, the adversary needs to make an irrevocable decision on whether to attack, and if he/she fails, or opts not to attack, then that data point is no longer available for further attacks. In the sequential setting, the model owner receives data update requests (i.e., { u t } t ≥ 1 ) from the users sequentially, and is asked to update the model from these update requests. We let Z = {Z , ∅} with a slight abuse of notation. Here, the t -th update request u t is a tuple u t = ( o t , z tra t , z new t ) , where z tra t ∈ Z , z new t ∈ Z , and o t ∈ O = { 'Delete' , 'Add' , 'Modify' } is a update instruction. Using these update requests, we can sequentially update the dataset and model as defined below.

Definition 4 (Update sequences and sequentially updated models) . Let U = ( u 1 , u 2 , · · · , u t , · · · ) denote the update sequence, where u t ∈ O × Z × Z for all t . Given the dataset D and the t -th update request u t , the update operation for D t = D t -1 ◦ u t is defined as follows

We write D 0 = D . For any t ≥ 1 , we write θ t for the model input to the unlearning algorithm R A on time step t . We write θ 1 = A ( D 0 ) , and for any t ≥ 1 , θ t +1 = R A ( D t -1 , θ t , u t ) . We write { D t } t ≥ 0 to represent the sequence of updated datasets, { θ t } t ≥ 1 for the sequence of input models to R A .

The update sequences mentioned above align with the data update requests specified in [44, 23, 36], which focus on the add and delete requests. However, it is important to note that we extend these update requests to include the modify requests [64, 37] as well, and this extension is necessary because some existing works also incorporate the modify update requests to enable the deletion of partial data information [64, 37, 36, 2, 41, 17]. The adversary's goal is to force current updated model to satisfy certain desired properties at each time step while paying a small cost. For example, the adversary wants to force the current updated model to approach or maintain a target model θ tar . Then we can define the adversary's goal as L adv = || θ -θ tar || . We propose to formulate the sequential selective forgetting attacks as a Markov Decision Process (MDP) M = ( S , A , T , R , γ ) , where

S is the state space. The state s t at time step t is the stacked vector s t = [ θ t , u t ] T consisting of the current model θ t and the incoming update request u t , where u t ∈ O × Z × Z . The state space is S = Θ ×O×Z×Z . We assume that the initial model θ 0 is fixed and known to the adversary while the first update request u 1 is sampled from P , i.e., the initial state distribution is defined as µ 0 ( θ 0 , u 1 ) = P ( u 1 ) .

A is the adversary action space. For each update request u t , we define a discrete indication parameter k t ∈ { 0 , 1 } to denote whether at time step t the attack action should applied ( k t = 1 ) or not ( k t = 0 ). If the update instruction o t for u t is ' Add ', we assume that the adversary can only introduce imperceptible perturbations a t with the purpose of manipulating z new t for stealthiness. Thus, the attacked u t can be represented as ˜ u t = ( o t , z tra t = ∅ , ˜ z new t =

z new t + k t · a t ) . When o t = ' Modify ' and k t = 1 , we can write the attacked u t as ˜ u t = ( o t , z tra t , ˜ z new t = z new t + k t · a t ) , where a t is the imperceptible perturbations. When u t is a delete update request and the adversary chooses to attack this delete update request (i.e., k t = 1 ), we assume that the adversary has the ability to manipulate the update instruction by converting o t = ' Delete ' into ˜ o t = ' Modify ', and the attacked u t is ˜ u t = (˜ o t , z tra t , ˜ z new t = z tra t + k t · a t ) . Here, the adversary maliciously modifies z tra t ∈ D t -1 (instead of directly deleting z tra t ). Note that the adversary can also choose not to attack this delete request.

T denotes the state transition function. The state transition function T : S × A → ∆ S represents the conditional probability of the next state given the current state and attack action. We assume that the update function g is deterministic. Therefore, the stochasticity arises solely from u t +1 within s t +1 . To provide a concrete example, we consider the modification step where o t corresponds to the action ' Modify '. In this case, the transition function can be derived as

Note that in the above modify case, given the attack action a t , we can obtain ˜ u t = ( o t , z tra t , ˜ z new t = ( z new t + a t )) . Discussions on other data update cases (also the proposed sequential optimization approach) can be found in the supplementary material.

R is the cost function. We define the cost at the time step t as R ( s t = [ θ t , u t ] T , k t , a t ) := ∥ θ t +1 -θ tar ∥ 2 , which is determined by the current state and the attack action.

Apolicy is a function Φ M : S → A that the adversary uses to choose the attack action a t = Φ M ( s t = [ θ t , u t ] T ) based on the current victim model θ t and update request u t . Note that k t ∈ { 0 , 1 } denotes whether at time step t the attack action should be applied. Now, the problem is how the adversary can find an effective attack strategy { k t } t ≥ 1 with the corresponding attack action a t (generated by the policy network), which can maximize the adversary's goal (i.e., E M ∑ ∞ t =0 -γ t R ( s t , k t , Φ M ( s t )) ) and minimize the number of attacked time steps (i.e., ∑ T t =1 k t ) for stealthiness. A naive way is to attack each step. However, attacking all the time steps would cause suspicion and expose the identity of the adversary. To address this, we formulate the below optimization to obtain the effective policy

where Φ M is the policy network to be optimized. In the above first constraint, we use the notation u t without explicitly distinguishing whether the t -th update request is subjected to an attack or not.

However, directly solving the above optimization problem is highly challenging due to the involvement of numerous variables. In addition, the environment data distribution for the formulated MDP is fixed but unknown to the adversary. To address the first challenge, we propose training an adversarial policy network Φ adv that takes the current state s t as input and outputs the attack strategy ( p t , a ′ t ) , where p t is the probability for taking the malicious action a ′ t . Specifically, for each step, the adversary gets the attack strategy ( p t , a ′ t ) from the adversarial policy (i.e., Φ adv : s t → ( p t , a ′ t ) ). If p t ≥ 0 . 5 , the adversary designates step t as the critical point and introduces perturbations to mislead the model owner to trigger the action a ′ t . Otherwise, the adversary does not attack the current time step. Additionally, the adversary can construct a progressively refined empirical distribution ˆ P t based on the sequence of observations u 1: t . More precisely, at time t , by replacing P with ˆ P t and the model θ 0 with θ t , the attacker can construct a substitute MDP ˆ M t = ( S , A , ˆ T t , R , γ ) , solve for the optimal policy Φ ∗ ˆ M t on ˆ M t , and then apply the learned policy to perform the one-step attack. In this paper, we solve the substitute MDP using deep deterministic policy gradient (DDPG) [33, 35] to handle a continuous action space.

Theorem 2. Let Φ ∗ M and Φ ∗ ˆ M be the optimal policies for M and ˆ M respectively, with the same initial state distribution µ 0 . We assume that the 1-Wasserstein distance between the estimated distributions ̂ P and the true distribution P satisfies W 1 ( ̂ P,P ) ≤ υ . Additionally, we assume the

loss function ℓ : S ×Z → R exhibits Lipschitz continuity with respect to both s and z , with a constant L 1 , and Lipschitz smoothness with respect to z , with a constant L 2 . Moreover, we assume the loss function to possess strong convexity, strong smoothness, and twice continuous differentiability with respect to s . Let J M (Φ M ) be the cost in Eqn. (6). Then there exist two constants Ψ and Ω such that:

where ζ is a constant related with updating the model.

Discussions. The above theorem implies that the difference between J M (Φ ∗ M ) and J M ( Φ ∗ ˆ M ) depends on the distribution difference υ . In the above, we discuss our proposed selective forgetting attacks in the white-box setting. In the black-box setting, the adversary can randomly select substitute models and exploit the transferability property [11, 45, 58], which arises from the shared vulnerabilities or decision boundaries among different models. For instance, in approximate unlearning methods, the adversary can train one or several models to substitute the well-trained model θ ∗ . This allows the adversary to generate malicious update requests and effectively transfer them to the target black-box victim model.

4 Experiments

In this section, we conduct comprehensive experiments to evaluate the effectiveness of our proposed selective forgetting attacks. All the experiments are run for 10 individual trials with different random seeds. Due to space limitations, a detailed description of the experimental setup, parameter settings, and more experimental results are given in the supplementary material.

Datasets and models. In experiments, we adopt the following real-world datasets: CIFAR-10 [31], Adult [15], Diabetes [1], and MNIST [12]. CIFAR-10 consists of 60,000 color images across 10 classes. Adult is downsampled to 23,374 samples with 57 features. Diabetes comprises 70,692 survey responses with 21 numerical features. MNIST contains 70,000 grayscale images of handwritten digits. In addition, we adopt a synthetic dataset, which is a binary dataset with 10,000 samples and 20 features and is generated based on the normal distribution. We also use a range of machine learning models, including the logistic regression model, ResNet-18 [25], VGG-16 [53], MobileNetV2 [50], and a neural network with two fully connected layers.

Attack settings. In experiments, we implement malicious selective forgetting attacks using the following unlearning methods: first-order based [64], second-order based [64], unrolling SGD [57], amnesiac [21], and SISA [4]. In static forgetting attacks, we first pre-train the model and then perform selective forgetting attacks in targeted and untargeted settings. In the targeted setting, the adversary aims to misclassify the input as a specific target class, while in the untargeted setting, the adversary aims to mislead the model into predicting any incorrect class. In sequential forgetting attacks, we first generate the target model from static forgetting attacks and then force the victim model to be close to the target model. In experiments, we allow DDPG [35] to train once at the beginning to learn the optimal policy on the pre-attack data and then apply the learned policy to perform the one-step attack.

Baselines. In experiments, we adopt the RandSearch baseline, where we randomly select a set of training samples to be forgotten in static forgetting attacks. In sequential selective forgetting attacks, we consider the no attack and random attack baselines. Specifically, the no attack baseline keeps the incoming data update requests unchanged, and the random attack baseline adds random data noise to modify the update requests.

4.1 Experimental Results for Static Selective Forgetting Attacks

First, we conduct experiments to investigate the performance of malicious selective forgetting attacks in a static manner. We adopt the attack success rate as the evaluation metric, defined as the number of successful attacks achieved among all attack attempts. For L adv in Eqn. (1), we adopt the f 6 function in [7]. Table 1 summarizes the attack success rate of static forgetting attacks in the targeted setting via first-order, second-order, unrolling SGD, amnesiac, and SISA. Our proposed methods consistently achieve high attack success rates across diverse datasets and unlearning procedures. For example, our proposed methods hit an attack success rate of 1.0 on Diabetes and 0.8 on CIFAR-10. The reason is that our proposed methods can assign importance scores to training samples based on

the impact of the targeted loss and unlearn optimal samples, resulting in substantial enhancements in attack performance. In contrast, the RandSearch baseline performs poorly in identifying the training samples to be removed for misclassifying the targeted test samples. These experimental results show the applicability and effectiveness of our optimization framework for static selective forgetting attacks across various unlearning methods, enabling us to achieve desired attack goals.

Next, we study the impact of underrepresentation, which occurs when certain classes in a dataset have fewer instances than others. In Figure 2a, we unlearn 1% to 5% of training samples from CIFAR-10. In Figure 2b, we unlearn 5% to 25% of training samples from Diabetes. The unlearning samples are exclusively selected from the same class as the target class. We compare the attack performance with and without our proposed optimization framework. As illustrated, the attack success rates increase correspondingly with the percentage of

1%

2%

3%

4%

5%

Percentage of unlearning samples

0.2

0.4

0.6

0.8

1.0

Attack success rate

First-order w/ opt

First-order w/o opt

Second-order w/ opt

Second-order w/o opt

(a) CIFAR-10

5%

10%

15%

20%

25%

Percentage of unlearning samples

0.2

0.4

0.6

0.8

Attack success rate

First-order w/ opt

First-order w/o opt

Second-order w/ opt

Second-order w/o opt

(b) Diabetes

unlearning samples, as removing certain training data can hinder the model's ability to learn the target class. However, randomly removing a small portion of data without optimization has a minimal effect on the attack success rates. In contrast, our proposed methods can effectively identify the most critical training samples to be removed, even within a specific class, enhancing the attack performance. Therefore, malicious selective forgetting attacks can leverage data underrepresentation to achieve the desired attack goals.

Then, we examine the impact of subpopulations, which are groups of samples having similar features. Here, we focus on misclassifying a particular subpopulation of targeted test samples. Upon identifying a test sample for a successful attack, we create a cluster consisting of points most similar to that sample based on the final representation layer in the network. We then apply the same malicious update requests to attack subpopulations of varying sizes. As a baseline comparison, we randomly select group mem-

10

20

30

40

50

Subpopulation size

0.8

0.9

1.0

Attack success rate

Subpopulation, targeted

Random group, targeted

Subpopulation, untargeted

Random group, untargeted

(a) Diabetes

10

20

30

40

50

Subpopulation size

0.4

0.6

0.8

1.0

Attack success rate

Subpopulation, targeted

Random group, targeted

Subpopulation, untargeted

Random group, untargeted

(b) CIFAR-10

bers of the same sizes. In Figure 3, we adopt unrolling SGD on Diabetes and amnesiac on CIFAR-10. We observe that malicious selective forgetting attacks are highly effective in attacking subpopulations of test samples in both targeted and untargeted settings. For instance, when using the unrolling SGD to attack a subpopulation of size 50 on Diabetes, the targeted attack success rate remains at 1.0. Similarly, with the amnesiac on CIFAR-10, the targeted attack success rate only drops to 0.85 for a subpopulation size of 50. It is worth noting that while the baseline randomly generated groups are not as good as those formed based on similarity, the attack performance is still impressive.

Further, we explore the impact of data diversity on malicious selective forgetting attacks. We apply PCA projection on input features and cluster the training data for each class. Then we formulate diversity by sampling different percentages of data points for each cluster. In Figure 4, we unlearn the same number of data points in each cluster and evaluate the attack performance with random selection and our proposed optimization framework in targeted and untargeted settings. Firstly, our proposed methods demonstrate the ability to identify the most influential data points within each cluster, leading to substantial improvements in attack success rates compared to the random removal of data points. Secondly, removing an equal number of critical data points

2

4

6

8

10

# of unlearning samples per cluster

0.0

0.2

0.4

0.6

0.8

Attack success rate

10% w/ opt

50% w/ opt

10% w/ random

50% w/ random

(a) Diabetes, targeted

2

4

6

8

10

# of unlearning samples per cluster

0.0

0.2

0.4

0.6

0.8

Attack success rate

10% w/ opt

50% w/ opt

10% w/ random

50% w/ random

(b) Diabetes, untargeted

2

4

6

8

10

# of unlearning samples per cluster

0.0

0.2

0.4

0.6

0.8

Attack success rate

10% w/ opt

50% w/ opt

10% w/ random

50% w/ random

(c) CIFAR-10, targeted

2

4

6

8

10

# of unlearning samples per cluster

0.2

0.4

0.6

0.8

1.0

Attack success rate

10% w/ opt

50% w/ opt

10% w/ random

50% w/ random

(d) CIFAR-10, untargeted

from less redundant clusters has a greater impact on the unlearned model, resulting in higher attack success rates than more redundant clusters. Consequently, unlearning training data in different scopes of diversity can affect the performance of malicious selective forgetting attacks.

4.2 Experimental Results for Sequential Selective Forgetting Attacks

In this section, we evaluate the performance of malicious selective forgetting attacks in the sequential update setting. We adopt the Euclidean distance metric to quantify the similarity between the victim model and the target model. Firstly, we examine the model convergence over attack time steps, where we naively apply attacks at each time step in a sequence of update requests. In Figure 5, we incorporate add , delete , and modify requests in a sequence of 300 update requests using the first-order unlearning method. As shown in the figure, our attack steadily reduces the Euclidean distance between the victim model and the target model over time steps on each adopted dataset (we use digits 1 and 7 in MNIST). In contrast, the random attack baseline and the no attack baseline fail to approach the target model. Our learned adversarial policy demonstrates the ability to attack the sequential data update requests by selecting the most effective actions within a small difference to the requests, resulting in a minor difference between the updated model and the target model.

0

100

200

300

Time step

0.1

0.2

0.3

0.4

Euclidean distance

Our attack

Random attack

No attack

(a) Synthetic

0

100

200

300

Time step

0.1

0.2

0.3

0.4

0.5

Euclidean distance

Our attack

Random attack

No attack

(b) Adult

0

100

200

300

Time step

0.1

0.2

0.3

0.4

0.5

Euclidean distance

Our attack

Random attack

No attack

(c) MNIST

Next, we investigate the performance variations corresponding to different attack steps within the sequential update process. In Figure 6, we compare corresponding models under different numbers of attack times with the optimal model (which is obtained by attacking all time steps and converges to the target model as shown in Figure 5). The results show that our attack method consistently diminishes the Euclidean distance between the victim model and the optimal model as the number of attack times increase. Remarkably, our optimally leaned policy injects perturbations on some specific update requests only when necessary, leading to fewer attack times and achieving comparable performance as the optimal model. However, the random attack baseline does not contribute much to approaching the optimal model as the number of attack times increase. Therefore, our proposed

optimization framework proves stealthiness and effectiveness in identifying the critical attack time steps and inducing the effective attack actions to the sequential update requests.

0

100

200

300

Number of attack times

0.0

0.1

0.2

0.3

Euclidean distance

Our attack

Random attack

(a) Synthetic

0

100

200

300

Number of attack times

0.0

0.1

0.2

0.3

Euclidean distance

Our attack

Random attack

(b) Adult

0

100

200

300

Number of attack times

0.0

0.1

0.2

0.3

Euclidean distance

Our attack

Random attack

(c) MNIST

4.3 Black-box Experiments

In this section, we consider malicious selective forgetting attacks in the black-box setting. Firstly, we investigate the transferability of selective forgetting attacks across different machine learning models. In Table 2, we employ ResNet-18, VGG16, and MobileNetV2 in the untargeted setting. As illustrated, our proposed methods demonstrate the ability to transfer the generated malicious update requests to attack the blackbox model, even though the blackbox model is trained with a different model than the substitute model. For example, ResNet-18 can achieve attack success rates of 0.86 and 0.75 when transferred to VGG-16 and MobileNetV2, respectively. Furthermore,

we investigate the transferability of selective forgetting attacks between different unlearning methods. In Table 3, we apply first-order, second-order, and unrolling SGD unlearning methods in the targeted setting. We observe that malicious update requests generated by our proposed methods can also be effectively transferred to different unlearning methods in the black-box setting. The reason is that with the same update requests, the unlearned models share similar decision boundaries among different models and unlearning methods.

5 Conclusion

In this paper, we examine the security vulnerability and resilience of machine learning models against selective forgetting attacks during the unlearning process, without compromising the integrity of the training and testing procedures, as commonly seen in traditional evasion attacks and data poisoning attacks. Specifically, we first present the general framework for selective forgetting attacks in the static setting, enabling the adversary to generate malicious whole data deletion requests. Additionally, we propose a new approach for developing sequential selective forgetting attacks that can effectively compromise the unlearning system using the sequential data update requests. We also conduct theoretical analysis for the proposed selective forgetting attacks. The reported extensive experimental results demonstrate the effectiveness of our proposed selective forgetting attacks. We believe that our findings provide valuable insights into how to design secure and robust mechanisms to defend against selective forgetting attacks in the future.

References

Behavioral risk factor surveillance system survey data, 2015.

Manaar Alam, Hithem Lamri, and Michail Maniatakos. Get rid of your trail: Remotely erasing backdoors in federated learning. arXiv preprint arXiv:2304.10638 , 2023.

Maksym Andriushchenko and Matthias Hein. Provably robust boosted decision stumps and trees against adversarial attacks. Advances in Neural Information Processing Systems , 32, 2019.

Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In Proceedings of the 42nd IEEE Symposium on Security and Privacy , May 2021.

N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP) , pages 39-57. IEEE Computer Society, May 2017.

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium , volume 267, 2019.

Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) , pages 39-57. Ieee, 2017.

Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. In Proceedings of the 13th International Conference on Neural Information Processing Systems . MIT Press, 2000.

Chong Chen, Fei Sun, Min Zhang, and Bolin Ding. Recommendation unlearning. In Proceedings of the ACM Web Conference 2022 , WWW '22, page 2768-2777, New York, NY, USA, 2022. Association for Computing Machinery.

Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. Graph unlearning. In ACM SIGSAC Conference on Computer and Communications Security . ACM, November 2022.

Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In 28th USENIX security symposium (USENIX security 19) , pages 321-338, 2019.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Jimmy Z Di, Jack Douglas, Jayadev Acharya, Gautam Kamath, and Ayush Sekhari. Hidden poison: Machine unlearning enables camouflaged poisoning attacks. In NeurIPS ML Safety Workshop , 2022.

Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modification. Advances in Neural Information Processing Systems , 34:18944-18957, 2021.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference , pages 214-226, 2012.

Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. arXiv preprint arXiv:2303.07345 , 2023.

Jonas Geiping, Liam H. Fowl, W. Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. In International Conference on Learning Representations , April 2021.

Antonio A. Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. Making AI Forget You: Data Deletion in Machine Learning . 2019.

Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9301-9309, Seattle, WA, USA, June 2020. IEEE.

Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11516-11524, 2021.

Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3832-3842. PMLR, 13-18 Jul 2020.

Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi-Malvajerdi, and Chris Waites. Adaptive machine unlearning. Advances in Neural Information Processing Systems , 34:16319-16330, 2021.

Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems , 29, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Mengdi Huai, Jinduo Liu, Chenglin Miao, Liuyi Yao, and Aidong Zhang. Towards automating model explanations with certified robustness guarantees. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6935-6943, 2022.

Mengdi Huai, Tianhang Zheng, Chenglin Miao, Liuyi Yao, and Aidong Zhang. On the robustness of metric learning: an adversarial perspective. ACM Transactions on Knowledge Discovery from Data (TKDD) , 16(5):1-25, 2022.

Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics , pages 2008-2016. PMLR, 2021.

Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP) , pages 19-35. IEEE, 2018.

Masayuki Karasuyama and Ichiro Takeuchi. Multiple incremental decremental learning of support vector machines. IEEE Transactions on Neural Networks , 21(7):1048-1059, 2010.

Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar-10. Unpublished manuscript , 40(7):1-9, 2010.

Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. Compas dataset. 2017.

Henger Li, Xiaolin Sun, and Zizhan Zheng. Learning to attack federated learning: A modelbased reinforcement learning attack framework. In Advances in Neural Information Processing Systems , 2022.

Qizhang Li, Yiwen Guo, and Hao Chen. Practical no-box adversarial attacks against dnns. Advances in Neural Information Processing Systems , 33:12849-12860, 2020.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations , 2016.

Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents , pages 243-254. PMLR, 2022.

Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, and Jianfeng Ma. Backdoor defense with machine unlearning. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications , pages 280-289. IEEE, 2022.

Neil G Marchant, Benjamin IP Rubinstein, and Scott Alfeld. Hard to forget: Poisoning attacks on certified machine unlearning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7691-7700, 2022.

Chenglin Miao, Qi Li, Houping Xiao, Wenjun Jiang, Mengdi Huai, and Lu Su. Towards data poisoning attacks in crowd sensing systems. In Proceedings of the Eighteenth ACM International Symposium on Mobile Ad Hoc Networking and Computing , pages 111-120, 2018.

Fatemehsadat Mireshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh. Privacy in deep learning: A survey. arXiv preprint arXiv:2004.12254 , 2020.

Saemi Moon, Seunghyuk Cho, and Dongwoo Kim. Feature unlearning for generative models via implicit feedback. arXiv preprint arXiv:2303.05699 , 2023.

S. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2574-2582. IEEE Computer Society, Jun 2016.

Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. Descent-to-Delete: Gradient-Based Methods for Machine Unlearning. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory , pages 931-962. PMLR, March 2021. ISSN: 2640-3498.

Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory , pages 931-962. PMLR, 2021.

Ehsan Nowroozi, Yassine Mekdad, Mohammad Hajian Berenjestanaki, Mauro Conti, and Abdeslam El Fergougui. Demystifying the transferability of adversarial attacks in computer networks. IEEE Transactions on Network and Service Management , 19(3):3387-3400, 2022.

Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Accumulative poisoning attacks on real-time data. Advances in Neural Information Processing Systems , 34:2899-2912, 2021.

Stuart L Pardau. The california consumer privacy act: Towards a european-style privacy regime in the united states. J. Tech. L. & Pol'y , 23:68, 2018.

Wei Qian, Chenxu Zhao, Wei Le, Meiyi Ma, and Mengdi Huai. Towards understanding and enhancing robustness of deep learning models against malicious unlearning attacks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1932-1942, 2023.

Wei Qian, Chenxu Zhao, Huajie Shao, Minghan Chen, Fei Wang, and Mengdi Huai. Patient similarity learning with selective forgetting. In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) , pages 529-534. IEEE, 2022.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. Advances in neural information processing systems , 31, 2018.

Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems . Curran Associates Inc., 2018.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In USENIX Security Symposium , volume 1, page 4, 2021.

Megha Srivastava, Hoda Heidari, and Andreas Krause. Mathematical notions vs. human perception of fairness: A descriptive approach to fairness for machine learning. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2459-2468, 2019.

Fnu Suya, Saeed Mahloujifar, Anshuman Suri, David Evans, and Yuan Tian. Model-targeted poisoning attacks with provable convergence. In International Conference on Machine Learning , pages 10000-10010. PMLR, 2021.

Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) , pages 303-319. IEEE, 2022.

Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv:1704.03453 , 2017.

Cheng-Hao Tsai, Chieh-Yen Lin, and Chih-Jen Lin. Incremental and decremental training for linear classification. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . Association for Computing Machinery, 2014.

Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing , 10(3152676):10-5555, 2017.

Xingchen Wan, Henry Kenlay, Robin Ru, Arno Blaas, Michael A Osborne, and Xiaowen Dong. Adversarial attacks on graph classifiers via bayesian optimisation. Advances in Neural Information Processing Systems , 34:6983-6996, 2021.

Cheng-Long Wang, Mengdi Huai, and Di Wang. Inductive graph unlearning. arXiv preprint arXiv:2304.03093 , 2023.

Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. ArXiv , abs/2108.11577, 2021.

Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. Network and Distributed System Security Symposium , 2023.

Sandamal Weerasinghe, Tansu Alpcan, Sarah M. Erfani, and Christopher Leckie. Defending support vector machines against data poisoning attacks. IEEE Transactions on Information Forensics and Security , 16:2566-2578, 2021.

Cheng-Hsin Weng, Yan-Ting Lee, and Shan-Hung Brandon Wu. On the trade-off between adversarial and backdoor robustness. Advances in Neural Information Processing Systems , 33:11973-11983, 2020.

Xinyi Xu, Lingjuan Lyu, Xingjun Ma, Chenglin Miao, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Gradient driven rewards to guarantee fairness in collaborative machine learning. Advances in Neural Information Processing Systems , 34:16104-16117, 2021.

Xiang Zhang and Marinka Zitnik. Gnnguard: Defending graph neural networks against adversarial attacks. Advances in neural information processing systems , 33:9263-9275, 2020.

Xuezhou Zhang, Xiaojin Zhu, and Laurent Lessard. Online data poisoning attacks. In Learning for Dynamics and Control , pages 201-210. PMLR, 2020.

Juexiao Zhou, Haoyang Li, Xingyu Liao, Bin Zhang, Wenjia He, Zhongxiao Li, Longxi Zhou, and Xin Gao. Audit to forget: A unified method to revoke patients' private data in intelligent healthcare. bioRxiv , pages 2023-02, 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the potential and viability of malicious data update requests in the context of the unlearning process. The authors put forward a malicious selective forgetting attack in a static scenario and present a framework for sequential forgetting attacks. And the framework is formulated as a stochastic optimal control problem.\n\nSTRENGTHS:\nThis paper investigates the potential and viability of malicious data update requests in the context of the unlearning process.\n\nWEAKNESSES:\nThe attack goal of the proposed approach remains unclear. \n\nThe attack methodology of the proposed approach lacks clarity.\n\nThe authors assert the effectiveness of their method in both white-box and black-box scenarios. However, the methodology section lacks a clear explanation of how the attack steps differ in these two scenarios. \n\nMore experiments on high-resolution datasets, such as ImageNet, VGG-Flower, that include more classes, should be evaluated. Otherwise, it is not sure whether the proposed method is applicable in the real-world scenario.\n\nQUESTIONS:\nThe attack goal of the proposed approach remains unclear. While the paper discusses constructing malicious data update requests during the unlearning process, it does not explicitly define the specific objectives pursued by the attacker. It is essential for the authors to provide concrete explanations regarding the attack goals, rather than merely stating them as desired outcomes. For instance, is the intention to diminish the fairness of the unlearning model or to deliberately misclassify specific samples?Furthermore, the paper lacks a high-level description of how the attack is executed, given a clear attack target. It is important for the authors to outline the general methodology employed by the attacker to carry out the attacks. \n\nThe attack methodology of the proposed approach lacks clarity. While the authors present multiple definitions, lemmas, and theorems in the methodology section, they do not provide an overall step-by-step explanation of how the attack is conducted. As a result, obtaining a clear understanding of the attack process remains challenging.\n\nAn overview image is also required to describe the proposed attacks.\n\nThe authors assert the effectiveness of their method in both white-box and black-box scenarios. However, the methodology section lacks a clear explanation of how the attack steps differ in these two scenarios. \n\nMore experiments on high-resolution datasets, such as ImageNet, VGG-Flower, that include more classes, should be evaluated. Otherwise, it is not sure whether the proposed method is applicable in the real-world scenario.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the malicious data update in machine unlearning. The authors consider two strategies. The first is static selective forgetting attack framework, where the adversary exploits vulnerabilities in the unlearning systems by submitting a set of carefully crafted data update requests at once. The second is sequential selective forgetting attack framework that injects malicious update multiple times by considering the order and timing of data update requests. Theoretical and experimental analysis are provided to validate the proposed machine unlearning attacks.\n\nSTRENGTHS:\n•\tThe angle of the problem is novel as there is no study on malicious machine unlearning so far.\n•\tThe authors propose one-step and multi-step attacks, and study the attack effect in white/black box, targeted/untargeted settings.\n•\tThe two proposed methods are well-justified both theoretically and experimentally.\n\nWEAKNESSES:\n•\tExperiment detail missing:\n      o\tWhat is target class in the targeted setting? \n      o\tWhat is the class that the machine unlearning aims to forget? The entire class or a few samples? If latter(according to line 344-346), what does forgetting mean in this case?\n      o\tWhat does the attack success rate mean, especially in the targeted setting? Reaching the targeted class or no longer on the previous class? \n      o\tFor untargeted setting, will the random class be the class for the task of machine unlearning? If so, what is the impact?\n      o\tWhat is the setting (untargeted/targeted) for black-box attack? The authors should specify in the manuscript.\n•\tRelated work to model poisoning should be added. While the authors mentioned data poisoning attacks, the proposed attack in machine unlearning is closer to model poisoning, which embeds the attack goal with the benign training objectives. Likewise, the proposed attack embeds the attack goal into the unlearning objective.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the malicious forgetting issue in model unlearning and proposes two attack strategies: static attack and dynamic sequential attack. The authors also present a theoretical framework for selective forgetting attacks. Experimental results on multiple benchmark datasets demonstrate that the proposed attack method poses a significant security threat to model unlearning.\n\nSTRENGTHS:\n- Model unlearning as an effective strategy for data forgetting has garnered significant attention, making the exploration of potential malicious attacks during the unlearning process an interesting research direction. \n- The paper has a clear research motivation, well-structured writing, and provides ample theoretical support.\n\nWEAKNESSES:\n- It is unclear whether this topic has been previously studied. The core of the proposed unlearning attack lies in a reasonable data sampling strategy for the victim model, which may not be novel from a technical standpoint. \n- It would be interesting to investigate if the defender, being aware of the malicious unlearning procedure, how could detect or filter out such requests to counter the attack. For example, could perturbation noise be added to mitigate the malicious impact of unlearning?\n- Recent study like Anti-backdoor Learning [1] uses two-stage unlearning techniques against backdoor attacks, whether this method could be applied to mitigate the negative effect of proposed unlearning attack?\n- The authors should discuss the potential limitations of this attack.\n\nI would be happy to improve the score if the authors address the aforementioned issues.\n\n[1] Yige Li, Xixiang Lyu, Xingjun Ma, et al, Anti-Backdoor Learning: Training Clean Models on Poisoned Data, NeurIPS, 2021\n\nQUESTIONS:\nRefer to the weaknesses mentioned above.\n\nLIMITATIONS:\nThe method does not exhibit apparent limitations. One main concern lies in the lack of convincing baseline comparisons in the experiments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identified a novel class of machine learning attacks, i.e.,  ML models can be manipulated with malicious data update requests during the machine unlearning process.\nThe authors study two threat scenarios: (1)  selective forgetting attacks and (2) sequential selective forgetting attacks.\nSpecifically, in the static setting, an adversary can first select a subset of accessible training data and then launch the unlearning request with the selected subset in order to induce the victim model to become close to a target model.\nIn this setting, the authors achieve the attack goal by casting the discrete data selection process into a continuous optimization process.\nOn the other hand, in the sequential setting, data update requests occur sequentially and the adversary can modify any update request before it is received by the victim.\nIn this setting, the authors propose to train a policy network via reinforcement learning to output attack strategies according to the current environment state.\nComprehensive experiments verify the effectiveness of the proposed selective forgetting attacks.\n\nSTRENGTHS:\n1. This paper identifies a novel and important ML threat, selective forgetting attacks. Such a threat is realistic in the real world and therefore worth the ML security community to continue further research.\n\n2. For the static version of the attack, how to efficiently select a subset to launch an attack is a challenging problem.\nThe authors address the problem by approximating the discrete indication function with a continuous function to perform GD-based optimization. This is a technically non-trivial solution.\n\n3. The problem formalization for sequential attacks is comprehensive, as it considers (almost) all possible types of update requests(\"delete\", \"add\", and \"modify\").\n\nWEAKNESSES:\n1. Two concerns about the sequential selective forgetting attacks:\n    - In the black-box setting, According to this paper, the adversary could not access any training data. Therefore, to let the substitute model better imitate the behavior of the victim model (in order to train an effective policy network), the adversary may need to collect data that are similar to the real data. However, this could be difficult in the real world when the adversary does not have any prior knowledge about the private training data.\n\n    - The adversary performs sequential attacks via modifying incoming update requests before they are received by the victim model, in which the modification strategy is by adding perturbations to the requested examples. This may result in a strange \"paradox\": if you modify the data that needs to be unlearned, is it still the original data? Wouldn't it result in the victim model \"unlearning\" something that it has never learned? Please comment.\n\n2. Suggestion: including results of membership inference attacks (MIA) would be interesting. It would be interesting to see how selective forgetting attacks will affect the success rate of MIA.\n\nQUESTIONS:\n1. What is the usage of Theorem 1? It seems like removing Theorem 1 would not affect the conclusion of Section 3.1.\n\n2. In Section 4, how to calculate the \"attack success rate\"? Is it the case that the authors first collect a set of data and then evaluate the victim's misclassification rate every time after processing an unlearn request?\n\n3. The setting of selective forgetting attacks seems a little similar to that in [r1]. It would be interesting to compare [r1] with this paper.\n\n**Reference**\n\n[r1] Shumailov et al. \"Manipulating SGD with Data Ordering Attacks\". arXiv 2021.\n\nLIMITATIONS:\nThe main limitation of this paper is that the threatening scenario of sequential attacks may not be realistic to some extent, which has been explained in Section \"Weaknesses\".\nNevertheless, I believe the merits of this paper suppress its disadvantages and would significantly benefit the ML security community.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}