SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $S$  is the state space. The state  $s_t$  at time step  $t$  is the stacked vector  $s_t = [\theta_t, u_t]^T$  consisting of the current model  $\theta_t$  and the incoming update request  $u_t$ , where  $u_t \in \mathcal{O} \times \mathcal{Z} \times \mathcal{Z}$ . The state space is  $S = \Theta \times \mathcal{O} \times \mathcal{Z} \times \mathcal{Z}$ . We assume that the initial model  $\theta_0$  is fixed and known to the adversary while the first update request  $u_1$  is sampled from  $P$ , i.e., the initial state distribution is defined as  $\mu_0(\theta_0, u_1) = P(u_1)$ .
-  $\mathcal{A}$  is the adversary action space. For each update request  $u_{t}$ , we define a discrete indication parameter  $k_{t} \in \{0,1\}$  to denote whether at time step  $t$  the attack action should apply ( $k_{t} = 1$ ) or not ( $k_{t} = 0$ ). If the update instruction  $o_{t}$  for  $u_{t}$  is "Add", we assume that the adversary can only introduce imperceptible perturbations  $a_{t}$  with the purpose of manipulating  $z_{t}^{new}$  for stealthiness. Thus, the attacked  $u_{t}$  can be represented as  $\tilde{u}_{t} = (o_{t}, z_{t}^{tra} = \emptyset, \tilde{z}_{t}^{new} =$
$z_{t}^{new} + k_{t}\cdot a_{t})$ . When  $o_t = "Modify"$  and  $k_{t} = 1$ , we can write the attacked  $u_{t}$  as  $\tilde{u}_t = (o_t,z_t^{tra},\tilde{z}_t^{new} = z_t^{new} + k_t\cdot a_t)$ , where  $a_{t}$  is the imperceptible perturbations. When  $u_{t}$  is a delete update request and the adversary chooses to attack this delete update request (i.e.,  $k_{t} = 1$ ), we assume that the adversary has the ability to manipulate the update instruction by converting  $o_t = "Delete"$  into  $\tilde{o}_{t} = "Modify"$ , and the attacked  $u_{t}$  is  $\tilde{u}_{t} = (\tilde{o}_{t},z_{t}^{tra},\tilde{z}_{t}^{new} = z_{t}^{tra} + k_{t}\cdot a_{t})$ . Here, the adversary maliciously modifies  $z_{t}^{tra}\in D_{t - 1}$  (instead of directly deleting  $z_{t}^{tra}$ ). Note that the adversary can also choose not to attack this delete request.
-  $\mathcal{T}$  denotes the state transition function. The state transition function  $\mathcal{T}: S \times \mathcal{A} \to \Delta_S$  represents the conditional probability of the next state given the current state and attack action. We assume that the update function  $g$  is deterministic. Therefore, the stochasticity arises solely from  $u_{t+1}$  within  $s_{t+1}$ . To provide a concrete example, we consider the modification step where  $o_t$  corresponds to the action "Modify". In this case, the transition function can be derived as
$$
\begin{array}{l} \mathcal {T} \left(s _ {t + 1} \mid s _ {t}, a _ {t}\right) = \mathcal {T} \left(u _ {t + 1}, \theta_ {t + 1} \mid u _ {t}, \theta_ {t}, a _ {t}\right) = P \left(u _ {t + 1} \mid u _ {t}, \theta_ {t}, a _ {t}\right) P r \left(\theta_ {t + 1} = g \left(u _ {t}, \theta_ {t}, a _ {t}\right)\right) \\ = P \left(u _ {t + 1} \mid u _ {t}, \theta_ {t}, a _ {t}\right) P r \left(\theta_ {t + 1} = g ^ {\prime} \left(\widetilde {u} _ {t}, \theta_ {t}\right)\right) = P \left(u _ {t + 1} \mid u _ {t}, \theta_ {t}, a _ {t}\right) P r \left(\theta_ {t + 1} = R _ {A} \left(D _ {t - 1}, \right. \right. \\ \end{array}
$$
$$
\left. \theta_ {t}, \widetilde {u} _ {t}\right)) = P \left(u _ {t + 1} \mid u _ {t}, \theta_ {t}, a _ {t}\right) = P \left(u _ {t + 1}\right). \tag {5}
$$
Note that in the above modify case, given the attack action  $a_{t}$ , we can obtain  $\widetilde{u}_{t} = (o_{t}, z_{t}^{tra}, \tilde{z}_{t}^{new} = (z_{t}^{new} + a_{t}))$ . Discussions on other data update cases (also the proposed sequential optimization approach) can be found in the supplementary material.
-  $\mathcal{R}$  is the cost function. We define the cost at the time step  $t$  as  $\mathcal{R}(s_t = [\theta_t, u_t]^T, k_t, a_t) \coloneqq \|\theta_{t+1} - \theta^{\mathrm{tar}}\|_2$ , which is determined by the current state and the attack action.
A policy is a function  $\Phi_{\mathcal{M}}: \mathcal{S} \to \mathcal{A}$  that the adversary uses to choose the attack action  $a_t = \Phi_{\mathcal{M}}(s_t = [\theta_t, u_t]^T)$  based on the current victim model  $\theta_t$  and update request  $u_t$ . Note that  $k_t \in \{0,1\}$  denotes whether at time step  $t$  the attack action should be applied. Now, the problem is how the adversary can find an effective attack strategy  $\{k_t\}_{t \geq 1}$  with the corresponding attack action  $a_t$  (generated by the policy network), which can maximize the adversary's goal (i.e.,  $\mathbb{E}_{\mathcal{M}} \sum_{t=0}^{\infty} -\gamma^t \mathcal{R}(s_t, k_t, \Phi_{\mathcal{M}}(s_t))$ ) and minimize the number of attacked time steps (i.e.,  $\sum_{t=1}^{T} k_t$ ) for stealthiness. A naive way is to attack each step. However, attacking all the time steps would cause suspicion and expose the identity of the adversary. To address this, we formulate the below optimization to obtain the effective policy
$$
\begin{array}{l} \min  _ {\Phi_ {\mathcal {M}}, \left\{k _ {t} \right\} _ {t \geq 1}} \mathbb {E} _ {s \sim \mu_ {0}} \mathbb {E} _ {\mathcal {M}} \sum_ {t = 0} ^ {\infty} \gamma^ {t} \mathcal {R} \left(s _ {t} = \left[ \theta_ {t}, u _ {t} \right] ^ {T}, k _ {t}, a _ {t} = \Phi_ {\mathcal {M}} (s _ {t})\right) + \sum_ {t = 1} ^ {\infty} k _ {t} \\ \text {s . t .}, \theta_ {t} = R _ {A} \left(D _ {t - 2}, \theta_ {t - 1}, u _ {t - 1}\right), k _ {t} \in \{0, 1 \}, \tag {6} \\ \end{array}
$$
where  $\Phi_{\mathcal{M}}$  is the policy network to be optimized. In the above first constraint, we use the notation  $u_{t}$  without explicitly distinguishing whether the  $t$ -th update request is subjected to an attack or not.
However, directly solving the above optimization problem is highly challenging due to the involvement of numerous variables. In addition, the environment data distribution for the formulated MDP is fixed but unknown to the adversary. To address the first challenge, we propose training an adversarial policy network  $\Phi^{adv}$  that takes the current state  $s_t$  as input and outputs the attack strategy  $(p_t, a_t')$ , where  $p_t$  is the probability for taking the malicious action  $a_t'$ . Specifically, for each step, the adversary gets the attack strategy  $(p_t, a_t')$  from the adversarial policy (i.e.,  $\Phi^{adv}: s_t \to (p_t, a_t')$ ). If  $p_t \geq 0.5$ , the adversary designates step  $t$  as the critical point and introduces perturbations to mislead the model owner to trigger the action  $a_t'$ . Otherwise, the adversary does not attack the current time step. Additionally, the adversary can construct a progressively refined empirical distribution  $\hat{P}_t$  based on the sequence of observations  $u_{1:t}$ . More precisely, at time  $t$ , by replacing  $P$  with  $\hat{P}_t$  and the model  $\theta_0$  with  $\theta_t$ , the attacker can construct a substitute MDP  $\hat{\mathcal{M}}_t = (\mathcal{S}, \mathcal{A}, \hat{\mathcal{T}}_t, \mathcal{R}, \gamma)$ , solve for the optimal policy  $\Phi_{\hat{\mathcal{M}}_t}^*$  on  $\hat{\mathcal{M}}_t$ , and then apply the learned policy to perform the one-step attack. In this paper, we solve the substitute MDP using deep deterministic policy gradient (DDPG) [33, 35] to handle a continuous action space.
Theorem 2. Let  $\Phi_{\mathcal{M}}^{*}$  and  $\Phi_{\hat{\mathcal{M}}}^{*}$  be the optimal policies for  $\mathcal{M}$  and  $\hat{\mathcal{M}}$  respectively, with the same initial state distribution  $\mu_0$ . We assume that the 1-Wasserstein distance between the estimated distributions  $\hat{P}$  and the true distribution  $P$  satisfies  $W_{1}(\hat{P}, P) \leq v$ . Additionally, we assume the
loss function  $\ell : S \times \mathcal{Z} \to \mathbb{R}$  exhibits Lipschitz continuity with respect to both  $s$  and  $z$ , with a constant  $L_{1}$ , and Lipschitz smoothness with respect to  $z$ , with a constant  $L_{2}$ . Moreover, we assume the loss function to possess strong convexity, strong smoothness, and twice continuous differentiability with respect to  $s$ . Let  $\mathcal{J}_{\mathcal{M}}(\Phi_{\mathcal{M}})$  be the cost in Eqn. (6). Then there exist two constants  $\Psi$  and  $\Omega$  such that:
$$
\left| \mathcal {J} _ {\mathcal {M}} \left(\Phi_ {\mathcal {M}} ^ {*}\right) - \mathcal {J} _ {\mathcal {M}} \left(\Phi_ {\hat {\mathcal {M}}} ^ {*}\right) \right| \leq v \Omega \left(L _ {1} \left(L _ {2} \zeta + 2\right) + \Psi L _ {2} \zeta\right), \tag {7}
$$
where  $\zeta$  is a constant related with updating the model.
**Discussions.** The above theorem implies that the difference between  $\mathcal{I}_{\mathcal{M}}(\Phi_{\mathcal{M}}^{*})$  and  $\mathcal{I}_{\mathcal{M}}(\Phi_{\hat{\mathcal{M}}}^{*})$  depends on the distribution difference  $v$ . In the above, we discuss our proposed selective forgetting attacks in the white-box setting. In the black-box setting, the adversary can randomly select substitute models and exploit the transferability property [11, 45, 58], which arises from the shared vulnerabilities or decision boundaries among different models. For instance, in approximate unlearning methods, the adversary can train one or several models to substitute the well-trained model  $\theta^{*}$ . This allows the adversary to generate malicious update requests and effectively transfer them to the target black-box victim model.
In this section, we conduct comprehensive experiments to evaluate the effectiveness of our proposed selective forgetting attacks. All the experiments are run for 10 individual trials with different random seeds. Due to space limitations, a detailed description of the experimental setup, parameter settings, and more experimental results are given in the supplementary material.
Datasets and models. In experiments, we adopt the following real-world datasets: CIFAR-10 [31], Adult [15], Diabetes [1], and MNIST [12]. CIFAR-10 consists of 60,000 color images across 10 classes. Adult is downsampled to 23,374 samples with 57 features. Diabetes comprises 70,692 survey responses with 21 numerical features. MNIST contains 70,000 grayscale images of handwritten digits. In addition, we adopt a synthetic dataset, which is a binary dataset with 10,000 samples and 20 features and is generated based on the normal distribution. We also use a range of machine learning models, including the logistic regression model, ResNet-18 [25], VGG-16 [53], MobileNetV2 [50], and a neural network with two fully connected layers.
Attack settings. In experiments, we implement malicious selective forgetting attacks using the following unlearning methods: first-order based [64], second-order based [64], unrolling SGD [57], amnesiac [21], and SISA [4]. In static forgetting attacks, we first pre-train the model and then perform selective forgetting attacks in targeted and untargeted settings. In the targeted setting, the adversary aims to misclassify the input as a specific target class, while in the untargeted setting, the adversary aims to mislead the model into predicting any incorrect class. In sequential forgetting attacks, we first generate the target model from static forgetting attacks and then force the victim model to be close to the target model. In experiments, we allow DDPG [35] to train once at the beginning to learn the optimal policy on the pre-attack data and then apply the learned policy to perform the one-step attack.
Baselines. In experiments, we adopt the RandSearch baseline, where we randomly select a set of training samples to be forgotten in static forgetting attacks. In sequential selective forgetting attacks, we consider the no attack and random attack baselines. Specifically, the no attack baseline keeps the incoming data update requests unchanged, and the random attack baseline adds random data noise to modify the update requests.
4.1 Experimental Results for Static Selective Forgetting Attacks
First, we conduct experiments to investigate the performance of malicious selective forgetting attacks in a static manner. We adopt the attack success rate as the evaluation metric, defined as the number of successful attacks achieved among all attack attempts. For  $\mathcal{L}_{adv}$  in Eqn. (1), we adopt the  $f_{6}$  function in [7]. Table 1 summarizes the attack success rate of static forgetting attacks in the targeted setting via first-order, second-order, unrolling SGD, amnesiac, and SISA. Our proposed methods consistently achieve high attack success rates across diverse datasets and unlearning procedures. For example, our proposed methods hit an attack success rate of 1.0 on Diabetes and 0.8 on CIFAR-10. The reason is that our proposed methods can assign importance scores to training samples based on
Table 1: Attack success rate of static forgetting attacks in the targeted setting.
Unlearning method	Diabetes	CIFAR-10
RandSearch	Ours	RandSearch	Ours
First-order	0.04 ± 0.03	1.00 ± 0.00	0.08 ± 0.04	0.80 ± 0.04
Second-order	0.06 ± 0.04	1.00 ± 0.00	0.10 ± 0.08	0.82 ± 0.04
Unrolling SGD	0.04 ± 0.03	1.00 ± 0.00	0.06 ± 0.03	0.78 ± 0.06
Amnesiac	0.08 ± 0.04	0.98 ± 0.02	0.04 ± 0.03	0.74 ± 0.08
SISA	0.47 ± 0.12	0.74 ± 0.07	0.13 ± 0.05	0.67 ± 0.08
the impact of the targeted loss and unlearn optimal samples, resulting in substantial enhancements in attack performance. In contrast, the RandSearch baseline performs poorly in identifying the training samples to be removed for misclassifying the targeted test samples. These experimental results show the applicability and effectiveness of our optimization framework for static selective forgetting attacks across various unlearning methods, enabling us to achieve desired attack goals.
Next, we study the impact of underrepresentation, which occurs when certain classes in a dataset have fewer instances than others. In Figure 2a, we unlearn  $1\%$  to  $5\%$  of training samples from CIFAR-10. In Figure 2b, we unlearn  $5\%$  to  $25\%$  of training samples from Diabetes. The unlearning samples are exclusively selected from the same class as the target class. We compare the attack performance with and without our proposed optimization framework. As illustrated, the attack success rates increase correspondingly with the percentage of
(b) Diabetes
Figure 2: Attack success rate of selective forgetting attacks when data is underrepresented, comparing with and without optimization. The shaded area represents the standard error.
unlearning samples, as removing certain training data can hinder the model's ability to learn the target class. However, randomly removing a small portion of data without optimization has a minimal effect on the attack success rates. In contrast, our proposed methods can effectively identify the most critical training samples to be removed, even within a specific class, enhancing the attack performance. Therefore, malicious selective forgetting attacks can leverage data underrepresentation to achieve the desired attack goals.
Then, we examine the impact of subpopulations, which are groups of samples having similar features. Here, we focus on misclassifying a particular subpopulation of targeted test samples. Upon identifying a test sample for a successful attack, we create a cluster consisting of points most similar to that sample based on the final representation layer in the network. We then apply the same malicious update requests to attack subpopulations of varying sizes. As a baseline comparison, we randomly select group mem
(a) Diabetes
Figure 3: Attack success rate of selective forgetting attacks across subpopulations of varying sizes.
bers of the same sizes. In Figure 3, we adopt unrolling SGD on Diabetes and amnesiac on CIFAR-10. We observe that malicious selective forgetting attacks are highly effective in attacking subpopulations of test samples in both targeted and untargeted settings. For instance, when using the unrolling SGD to attack a subpopulation of size 50 on Diabetes, the targeted attack success rate remains at 1.0. Similarly, with the amnesiac on CIFAR-10, the targeted attack success rate only drops to 0.85 for a subpopulation size of 50. It is worth noting that while the baseline randomly generated groups are not as good as those formed based on similarity, the attack performance is still impressive.
Further, we explore the impact of data diversity on malicious selective forgetting attacks. We apply PCA projection on input features and cluster the training data for each class. Then we formulate diversity by sampling different percentages of data points for each cluster. In Figure 4, we unlearn the same number of data points in each cluster and evaluate the attack performance with random selection and our proposed optimization framework in targeted and untargeted settings. Firstly, our proposed methods demonstrate the ability to identify the most influential data points within each cluster, leading to substantial improvements in attack success rates compared to the random removal of data points. Secondly, removing an equal number of critical data points
from less redundant clusters has a greater impact on the unlearned model, resulting in higher attack success rates than more redundant clusters. Consequently, unlearning training data in different scopes of diversity can affect the performance of malicious selective forgetting attacks.
(a) Diabetes, targeted
(b) Diabetes,untargeted
(c) CIFAR-10, targeted
(d) CIFAR-10,untargeted
Figure 4: Attack success rate of selective forgetting attacks with clusters of  $10\%$  data points (less redundant) and with clusters of  $50\%$  data points (more redundant).
4.2 Experimental Results for Sequential Selective Forgetting Attacks
In this section, we evaluate the performance of malicious selective forgetting attacks in the sequential update setting. We adopt the Euclidean distance metric to quantify the similarity between the victim model and the target model. Firstly, we examine the model convergence over attack time steps, where we naively apply attacks at each time step in a sequence of update requests. In Figure 5, we incorporate add, delete, and modify requests in a sequence of 300 update requests using the first-order unlearning method. As shown in the figure, our attack steadily reduces the Euclidean distance between the victim model and the target model over time steps on each adopted dataset (we use digits 1 and 7 in MNIST). In contrast, the random attack baseline and the no attack baseline fail to approach the target model. Our learned adversarial policy demonstrates the ability to attack the sequential data update requests by selecting the most effective actions within a small difference to the requests, resulting in a minor difference between the updated model and the target model.
(a) Synthetic
Figure 5: Euclidean distance of the victim model to the target model at each time step.
Next, we investigate the performance variations corresponding to different attack steps within the sequential update process. In Figure 6, we compare corresponding models under different numbers of attack times with the optimal model (which is obtained by attacking all time steps and converges to the target model as shown in Figure 5). The results show that our attack method consistently diminishes the Euclidean distance between the victim model and the optimal model as the number of attack times increase. Remarkably, our optimally leaned policy injects perturbations on some specific update requests only when necessary, leading to fewer attack times and achieving comparable performance as the optimal model. However, the random attack baseline does not contribute much to approaching the optimal model as the number of attack times increase. Therefore, our proposed
optimization framework proves stealthiness and effectiveness in identifying the critical attack time steps and inducing the effective attack actions to the sequential update requests.
(b) Adult
Figure 6: Euclidean distance of the victim model to the target under different numbers of attack times.
4.3 Black-box Experiments
In this section, we consider malicious selective forgetting attacks in the black-box setting. Firstly, we investigate the transferability of selective forgetting attacks across different machine learning models. In Table 2, we employ ResNet-18, VGG-16, and MobileNetV2 in the untargeted setting. As illustrated, our proposed methods demonstrate the ability to transfer the generated malicious update requests to attack the blackbox model, even though the blackbox model is trained with a different model than the substitute model. For example, ResNet-18 can achieve attack success rates of 0.86 and 0.75 when transferred to VGG-16 and MobileNetV2, respectively. Furthermore,
Table 2: Attack success rates in the black-box setting using substitute models on CIFAR-10.
Substitute\Black-box	ResNet-18	VGG-16	MobileNetV2
ResNet-18	1.00 ± 0.00	0.86 ± 0.05	0.75 ± 0.07
VGG-16	1.00 ± 0.00	1.00 ± 0.00	0.68 ± 0.05
MobileNetV2	0.98 ± 0.02	0.72 ± 0.09	1.00 ± 0.00
Table 3: Attack success rate in the black-box setting using substitute unlearning methods on Diabetes.
Substitute\Black-box	First-order	Second-order	Unrolling SGD
First-order	1.00 ± 0.00	0.88 ± 0.08	0.96 ± 0.03
Second-order	1.00 ± 0.00	1.00 ± 0.00	0.94 ± 0.06
Unrolling SGD	1.00 ± 0.00	0.94 ± 0.03	1.00 ± 0.00
we investigate the transferability of selective forgetting attacks between different unlearning methods. In Table 3, we apply first-order, second-order, and unrolling SGD unlearning methods in the targeted setting. We observe that malicious update requests generated by our proposed methods can also be effectively transferred to different unlearning methods in the black-box setting. The reason is that with the same update requests, the unlearned models share similar decision boundaries among different models and unlearning methods.
In this paper, we examine the security vulnerability and resilience of machine learning models against selective forgetting attacks during the unlearning process, without compromising the integrity of the training and testing procedures, as commonly seen in traditional evasion attacks and data poisoning attacks. Specifically, we first present the general framework for selective forgetting attacks in the static setting, enabling the adversary to generate malicious whole data deletion requests. Additionally, we propose a new approach for developing sequential selective forgetting attacks that can effectively compromise the unlearning system using the sequential data update requests. We also conduct theoretical analysis for the proposed selective forgetting attacks. The reported extensive experimental results demonstrate the effectiveness of our proposed selective forgetting attacks. We believe that our findings provide valuable insights into how to design secure and robust mechanisms to defend against selective forgetting attacks in the future.