SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chenxu Zhao*
cxzhao@iastate.edu
Wei Qian*
wqi@iastate.edu
Mengdi Huai
mdhuai@iastate.edu
With the growing demand for the right to be forgotten, there is an increasing need for machine learning models to forget sensitive data and its impact. To address this, the paradigm of selective forgetting (a.k.a machine unlearning) has been extensively studied, which aims to remove the impact of requested data from a well-trained model without retraining from scratch. Despite its significant success, limited attention has been given to the security vulnerabilities of the unlearning system concerning malicious data update requests. Motivated by this, in this paper, we explore the possibility and feasibility of malicious data update requests during the unlearning process. Specifically, we first propose a new class of malicious selective forgetting attacks, which involves a static scenario where all the malicious data update requests are provided by the adversary at once. Additionally, considering the sequential setting where the data update requests arrive sequentially, we also design a novel framework for sequential forgetting attacks, which is formulated as a stochastic optimal control problem. We also propose novel optimization algorithms that can find the effective malicious data update requests. We perform theoretical analyses for the proposed selective forgetting attacks, and extensive experimental results validate the effectiveness of our proposed selective forgetting attacks. The source code is available in the supplementary material.
Machine learning algorithms play a crucial role in diverse fields such as biology, speech recognition, agriculture, and medicine. To build pertinent models, these algorithms are frequently trained using a range of data sources, including third-party datasets, internal datasets, and customized subsets of publicly available user data. With recent demands for increased data privacy, the data users could erase the impact of their sensitive information from the trained models to ensure their privacy. Recent legislation (e.g., the General Data Protection Regulation from the European Union [60], the California Consumer Privacy Act [47], and the Canada's proposed Consumer Privacy Protection Act) requires the right to be forgotten, and grants users an unconditional right to request that their private data be removed from everywhere in the system within a reasonable time.
*The first two authors contribute equally to this work.
However, with the development of traditional machine learning techniques, this basic right is usually neglected or violated [70, 19, 38]. An illustrative instance is the inadvertent leakage of patients' genetic markers through machine learning methods employed for genetic data processing without the patients' awareness. Therefore, it is important to entitle data users the right to delete their personal data from trained machine learning models since machine learning models could memorize sensitive information of the training data and thus expose individual's privacy risk [54, 6, 8, 30, 59, 4, 40]. The most naive way is to retrain from the original data after removing the samples that need to be forgotten. Unfortunately, this naive retraining method can be prohibitive in terms of the computational and space cost—especially for large models and frequent deletion requests. To mitigate this, selective forgetting (a.k.a machine unlearning) [4, 9, 43, 22, 20, 28, 63] has been extensively researched in recent years to avoid the high computational cost associated with fully retraining a model from scratch.
However, existing studies on selective forgetting mainly focus on designing new forgetting algorithms to enable the right to be forgotten to be efficiently implemented, leaving the security issues during the unlearning process in adversarial settings largely unexplored. In practice, the motivated adversary could make use of the unlearning pipeline to craft malicious data update requests to achieve his/her desired attack goals. For example, the motivated adversary could increase a disadvantage against a specific group of individuals. In Figure 1, we present a toy example to highlight the impact of malicious data update requests on fairness using the COMPAS [32] dataset. Note that fairness in machine learning [67, 55] refers to the concept of ensuring that the decisions produced by machine learning algorithms are unbiased and equitable across different groups of individuals,
Figure 1: Fairness gap (Demographic Parity and Equalized Odds) with random unlearning samples in the minority group on COMPAS.
irrespective of their protected attributes (e.g., race and ethnicity). Here we consider race (black/white) as the sensitive feature and just randomly unlearn varying percentages of samples from the minority group (white). For the fairness evaluation metrics, we adopt Demographic Parity [16] and Equalized Odds [24] (please refer to the supplementary material for details of the two evaluation metrics and more experimental results on attacking fairness). From this figure, we can easily see that even though we just randomly delete some individuals from the white group in the dataset, the fairness gap expands, indicating that the model's fairness is compromised. When the victim systems are employed for security-sensitive applications, such malicious data update requests can cause tremendous security threats to the unlearning system. Therefore, it is essential to understand the feasibility of malicious data requests in the unlearning systems.
In this paper, we aim to conduct a comprehensive study on the security vulnerability and robustness of the unlearning system to malicious data update requests during the unlearning process. Specifically, we first propose a novel static selective forgetting attack framework, where the adversary exploits vulnerabilities in the unlearning systems by submitting a set of carefully crafted data update requests at once. More specifically, the proposed static attack framework uses discrete indication variables to formulate the complete deletion of targeted training samples, which is very hard to be directly solved. To address this challenge, we design a continuous and differentiable function to approximate the discrete component. On the other hand, many real-world applications involve streaming data update requests that arrive in a sequential manner. The adversary could take advantage of this sequential interaction setting to strategically manipulate the unlearning process. However, launching attacks on all the received sequential data update requests indiscriminately can potentially lead to the detection of the adversary. To address this issue, we also design a novel sequential selective forgetting attack framework that takes into account the order and timing of data update requests. In this framework, the adversary focuses on attacking a few critical data update requests to maximize the impact of his/her malicious actions, potentially leading to severe security threats. We also conduct theoretical analyses of our proposed selective forgetting attacks. Further, we conduct extensive experiments in different scenarios to validate the general effectiveness of our proposed selective forgetting attacks.
Selective forgetting (a.k.a machine unlearning and data deletion) [20, 38, 19, 38, 49, 62] refers to removing the influence of the requested data from a trained model. However, existing data deletion works [19, 19, 8, 30, 59, 4, 10, 9, 43, 22, 20, 28, 63] ignore the risks of malicious data update requests during the unlearning process and fail to identify the vulnerabilities of machine learning models to selective forgetting attacks. It is worth mentioning that our proposed selective forgetting attacks are different from traditional evasion attacks [66, 68, 3, 34, 61, 27, 69, 26] and data poisoning attacks [46, 14, 51, 39]. Evasion attacks achieve adversarial goals by modifying test samples; however, our forgetting attacks do not change test data and only modify the well-trained models by making malicious data update requests. Traditional data poisoning attacks occur during training, and manipulate the original clean training data. Note that [13] adds carefully crafted samples to the training dataset, and assumes the exact and retraining tasks. The authors in [48] fail to address the issue of malicious whole data deletion in the static setting, nor do they consider the sequential attack setting involving various types of malicious data update requests.
3 Malicious Selective Forgetting Attacks
Without loss of generality, in this work, we consider the classification models. Let  $D = \{z_{i} = (x_{i},y_{i})\}_{i = 1}^{N}$  denote the training dataset from  $\mathcal{Z} = \mathcal{X}\times \mathcal{Y}$ , where  $x_{i}\in \mathbb{R}^{d_{1}}$  is a  $d_{1}$ -dimensional sample and  $y_{i}\in [C]$  denotes its associated class label. The model owner applies a learning algorithm  $A$  on  $D$  to learn a model  $f(\theta)$  parameterized by  $\theta$ , such that  $f(\theta)$  achieves low empirical loss. We denote a loss function by a mapping  $\ell :\Theta \times \mathcal{Z}\to \mathbb{R}$  that takes the parameters  $\theta \in \Theta$  and a sample  $z\in \mathcal{Z}$ , outputs the loss of  $\theta$  on  $z$  (i.e.,  $\ell (z;\theta)$ ) that we denote by  $\ell_z(\theta)$ ). Let  $A(D)$  denote the distribution over the resulting model parameters  $\theta$  when the learning algorithm  $A$  is applied on  $D$ .
Threat Model. As previously mentioned, the proposed malicious selective forgetting attacks are unlearning time attacks, i.e., the adversary interferes with the unlearning process of the well-trained model and cannot modify any test sample submitted to the victim model at testing time. In addition, the adversary is unable to modify the training samples during the training stage. This reflects the unlearning scenario in which the adversary can only generate the update requests to selectively forget certain data information during the unlearning process. In this paper, we study both the white-box and black-box settings. Specifically, in the white-box setting, we make the assumption that the adversary possesses complete information about the system (including the model architecture and parameters of the well-trained model). Note that many evasion and poisoning attacks in the literature [5, 42, 65, 52, 18, 33, 29] employ a white-box model to study the adversary's strong attack behaviors in such worst-case settings. In the black-box setting, we assume that the adversary does not have any prior knowledge about the target well-trained model.
3.1 Static Selective Forgetting Attacks
Here, we consider the static attack setting, where the model holder owns a well-trained classification model  $f(\theta)$  parameterized by  $\theta \in \mathbb{R}^{d_2}$  on dataset  $D$ , and the adversary aims to make malicious update requests to deliberately forget some information to achieve his/her desired attack goals. In this case, all the malicious data update requests are provided at once, without consideration for the sequential order. We use  $D_f \subset D$  and  $R_A$  to denote the adversary's requested forget set and the coupled unlearning algorithm for  $A$ , respectively. Note that for the given forget set, machine unlearning method can return a model  $\theta \sim R_A(D, A(D), D_f)$ , which possesses no information about  $D_f$  without influencing the contributions of other data. The following definition gives the definition of existing approximate and exact unlearning methods [64, 44, 23].
Definition 1 (Machine unlearning). Let  $A$  and  $R_A$  denote the learning algorithm and the unlearning method, respectively. The pair  $(A, R_A)$  achieves exact unlearning if  $\forall D, D_f \subset D, A(D_r) =_d R_A(D, A(D), D_f)$ , where  $D_r = D \setminus D_f$  and  $=_d$  means the same distribution. This means that if the unlearned model from  $R_A(D, A(D), D_f)$  has no information about  $D_f$ , we cannot differentiate the model after forgetting from a model that is obtained on  $D_r$ . The pair  $(A, R_A)$  satisfies  $(\epsilon, \delta)$ -unlearning if  $\forall D, D_f \subset D$ , and  $E \subset \mathbb{R}^{d_2}$ ,  $P(R_A(D, A(D), D_f) \in E) \leq e^\epsilon P(A(D_r) \in E) + \delta$ , where  $A$  and  $D_f$  denote the learning algorithm and the forget set, respectively.
As aforementioned, the adversary aims to make malicious update requests for his/her desired attack goals. For example, the adversary could attack the targeted test samples and force them to be assigned as the attack targeted label [7]. The adversary could also have a specific target model  $\theta^{tar}$  (e.g., the unfair and backdoored models) in mind and aim to induce a victim model as close as possible to that target model  $\theta^{tar}$  [56]. The effective unlearning samples (i.e.,  $D_{f} \subset D$ ) can be obtained by solving the following formulated optimization problem
$$
D _ {f} = \arg \min  _ {D _ {f} \subset D _ {t}} \mathcal {L} _ {a d v} (\cdot ; \theta^ {u} (D _ {f})), \text {w h e r e} D _ {t} = \left\{\left(x _ {p}, y _ {p}\right) \right\} _ {p = 1} ^ {P} \subset D. \tag {1}
$$
In the above,  $D_{t}$  represents a subset of  $D$  that is accessible to the adversary, and  $\theta^{u}(D_{f})$  are the parameters found by completely eliminating specific targeted samples  $D_{f} \subset D_{t} = \{(x_{p},y_{p})\}_{p = 1}^{P}$ . For each  $x_{p} \in D_{t}$ , we define a discrete indication parameter  $\omega_{p} \in \{0,1\}$  to indicate whether the sample  $x_{p}$  should be completely deleted  $(\omega_{p} = 1)$  or not  $(\omega_{p} = 0)$ . The forget set  $D_{f}$  to be unlearned is denoted as  $D_{f} = D_{t} \circ \Omega = \{x_{p}|x_{p} \in D_{t} \text{ and } \omega_{p} = 1\}$ , where  $\Omega = \{\omega_{p} \in \{0,1\} \}_{p = 1}^{P}$ . Note that the above equation is a bi-level optimization problem - the minimization for  $D_{f}$  involves the model parameters  $\theta^{u}(D_{f})$ , which are themselves the minimizer of the training problem,
$$
\theta^ {u} \left(D _ {f}\right) = R _ {A} \left(D, f _ {D} \left(\theta^ {*}\right), D _ {f} = D _ {t} \circ \Omega\right), \text {w h e r e} D _ {t} = \left\{\left(x _ {p}, y _ {p}\right) \right\} _ {p = 1} ^ {P} \text {a n d} \Omega = \left\{\omega_ {p} \right\} _ {p = 1} ^ {P}. \tag {2}
$$
Note that Eqn. (1) and (2) provide the high-level formulation for selecting subset  $D_{f}$  such that the adversary's goals are maximized after unlearning. However, it is very difficult to optimize the effective update requests due to the introduced discrete indication parameters (i.e.,  $\Omega = \{\omega_p\in \{0,1\} \}_{p = 1}^P$ ) of the first constraint in Eqn. (2). Next, we take the second-order unlearning strategy proposed in [64] as an illustrative example to show how to solve the above formulated optimization problem. This unlearning strategy uses the inverse Hessian matrix of the second-order partial derivatives to change the original model's parameters to obtain the unlearned model [64]. For  $D_{t} = Z = \{z_{p}\}_{p = 1}^{P}\subset D$ , we use  $\tilde{D}_t = \tilde{Z} = \{\tilde{z}_p\}_{p = 1}^P$  to denote its corresponding unlearned versions, where  $\tilde{z}_p = (x_p - \xi_p,y_p)$  and  $\xi_{p}$  is the unlearning modification for  $x_{p}$ . Following [64], we calculate the second-order change  $\Delta (Z,\tilde{Z})$  by calculating all the gradients difference between  $Z$  and  $\tilde{Z}$  with a weighting change from the inverse Hessian of the loss function, i.e.,  $\Delta (Z,\tilde{Z}) = H_{\theta^{*}}^{-1}(\sum_{\tilde{z}_p\in \tilde{Z}}\omega_p*\nabla_\theta \ell (\tilde{z}_p;\theta^*) - \sum_{z_p\in Z}\omega_p*\nabla_\theta \ell (z_p;\theta^*))$ , where  $H_{\theta^{*}}^{-1}$  is the inverse Hessian matrix and  $\omega_{p}\in \{0,1\}$  denotes whether  $z_{p}$  should be completely erased. To address the aforementioned challenge, we propose to relax each discrete variable  $\omega_{p}$  into a continuous one of range [0, 1], i.e.,  $\omega_{p}\in [0,1]$ , and then approximate the original update objective for this second-order strategy by the following one
$$
\theta^ {u} \leftarrow \theta^ {*} - H _ {\theta^ {*}} ^ {- 1} \left[ \sum_ {p = 1} ^ {P} \left(\frac {1}{1 + \exp \left(- \varphi \left(2 * \omega_ {p} - 1\right)\right)} * \left(\nabla_ {\theta} \ell \left(\tilde {z} _ {p}; \theta^ {*}\right) - \nabla_ {\theta} \ell \left(z _ {p}; \theta^ {*}\right)\right) \right], \right. \tag {3}
$$
where  $\ell$  is a training loss, and  $\omega_{p} \in [0,1]$ . Here, we rewrite  $\omega_{p}$  as  $\frac{1}{2}(1 + \mathrm{sgn}(2*\omega_p - 1))$ . Based on the fact that function  $h_1(x) = \frac{1}{2}(1 - \mathrm{sgn}(x))$  can be approximated by function  $h_2(x) = 1 - \frac{1}{1 + \exp(-\varphi x)}$ , we can obtain the above equation. The parameter  $\varphi$  in  $h_2$  represents the steepness of the curve. Additionally, the continuous property of  $h_2$  allows us to solve the formulated optimization to perform selective forgetting attacks via the second-order unlearning strategy in [64]. From Section IV in [64], we can know that if  $\tilde{Z} = \emptyset$ , the adversary has the option to completely remove the targeted sample or retain it, based on whether  $\omega_p \geq 0.5$  or  $\omega_p < 0.5$ . When  $\tilde{Z} \neq \emptyset$ , the adversary can intentionally and maliciously modify the targeted training samples via partially unlearning some data information based on Eqn. (3). Notably, following Lemma 1, we can easily generalize the above proposed attack framework to the scenario where the adversary wants to maliciously erase some features [64]. The algorithm and generalization to other unlearning methods are deferred to the supplementary material.
Lemma 1 ([64]). Let  $F$  denote the features to be unlearned. Let  $\theta_{-F}^{*}$  denote the optimal model retrained on the new dataset that is derived by removing the features  $F$  from  $D$ . For learning models processing inputs  $x$  using the linear transformations of the form  $\theta^T x$ , we have  $\theta_{-F}^{*} \equiv \theta_{F=0}^{*}$ , where  $\theta_{F=0}^{*}$  is retrained by setting the values of the features  $F$  to zero in  $D$ .
Definition 2 (Strongly convexity). A function  $\psi : \mathbb{R}^{d_3} \to \mathbb{R}^{d_4}$  is said to be  $M$ -strongly convex for some  $M \geq 0$  if for any  $z_1 \in \mathbb{R}^{d_3}, z_2 \in \mathbb{R}^{d_3}$ , and any  $q \in (0,1)$ ,  $\psi(qz_1 + (1 - q)z_2) \leq q\psi(z_1) + (1 - q)\psi(z_1) - \frac{M}{2}q(1 - q)\|z_1 - z_2\|_2^2$ . Note that if the above condition is satisfied for  $M = 0$ , we refer to the function  $\psi$  as convex.
Definition 3 (Lipschitzness continuity). A general function  $\psi : \mathbb{R}^{d_3} \to \mathbb{R}^{d_4}$  is  $L$ -globally Lipschitz continuous if for all  $z_1 \in \mathbb{R}^{d_3}, z_2 \in \mathbb{R}^{d_3}$ ,  $||\psi(z_1) - \psi(z_2)|| \leq L||z_1 - z_2||_2$ .
Theorem 1. Let  $\theta_D^* = \arg \min_{\theta \in \Theta} \ell_D(\theta)$  for any given dataset  $D$ . Suppose that the loss function  $\ell_z$  is  $L$ -globally Lipschitz continuous and  $M$ -strongly convex for any  $z \in \mathcal{Z}$ . For any integer  $N$ , dataset  $D$  of size  $N$ , and forget set  $D_f$ , we can have that  $\| \theta_D^* - \theta_{D \setminus D_f}^* \|_2 \leq \frac{2L}{MN} |D_f|$ , where  $|D_f|$  represents the size of the forget set.
The above lemma shows that we can erase features from many learning models by first setting them to zero [64]. The above theorem relates the difference of the model parameters to the forget set, and can be easily generalized to the defined data update requests in Definition 4.
3.2 Sequential and Dynamic Selective Forgetting Attacks
In practice, the model owner usually receives sequential update requests from one or more data owners at different times, and is asked to update the model from these sequential data update requests. Compelled by enticing incentives, the adversary could interact with this sequential update process to dynamically craft malicious update requests according to the model states, which can pose potential threats to the system. For the threat model, we here consider a very restricted setting where the adversary does not own any training data. Unlike the above static attacks, such a sequential update scenario presents a crucial challenge of transciency. Specifically, at each time step, the adversary needs to make an irrevocable decision on whether to attack, and if he/she fails, or opts not to attack, then that data point is no longer available for further attacks. In the sequential setting, the model owner receives data update requests (i.e.,  $\{u_t\}_{t\geq 1}$ ) from the users sequentially, and is asked to update the model from these update requests. We let  $\mathcal{Z} = \{\mathcal{Z},\emptyset \}$  with a slight abuse of notation. Here, the  $t$ -th update request  $u_{t}$  is a tuple  $u_{t} = (o_{t},z_{t}^{tra},z_{t}^{new})$ , where  $z_{t}^{tra}\in \mathcal{Z}$ ,  $z_{t}^{new}\in \mathcal{Z}$ , and  $o_{t}\in \mathcal{O} = \{"Delete", "Add", "Modify"\}$  is a update instruction. Using these update requests, we can sequentially update the dataset and model as defined below.
Definition 4 (Update sequences and sequentially updated models). Let  $\mathcal{U} = (u_1, u_2, \dots, u_t, \dots)$  denote the update sequence, where  $u_t \in \mathcal{O} \times \mathcal{Z} \times \mathcal{Z}$  for all  $t$ . Given the dataset  $D$  and the  $t$ -th update request  $u_t$ , the update operation for  $D_t = D_{t-1} \circ u_t$  is defined as follows
$$
\left\{ \begin{array}{l} D _ {t} = D _ {t - 1} \backslash z _ {t} ^ {t r a}, \quad i f \left(o _ {t} = “ D e l e t e ”\right) \wedge \left(z _ {t} ^ {t r a} \in D _ {t - 1}\right) \wedge \left(z _ {t} ^ {n e w} = \emptyset\right) \\ D _ {t} = D _ {t - 1} \cup z _ {t} ^ {n e w}, \quad i f \left(o _ {t} = “ A d d ”\right) \wedge \left(z _ {t} ^ {t r a} = \emptyset\right) \wedge \left(z _ {t} ^ {n e w} \notin D _ {t - 1}\right) \\ D _ {t} = \left(D _ {t - 1} \backslash z _ {t} ^ {t r a}\right) \cup z _ {t} ^ {n e w}, \quad i f \left(o _ {t} = “ M o d i f y ”\right) \wedge \left(z _ {t} ^ {t r a} \in D _ {t - 1}\right) \wedge \left(z _ {t} ^ {n e w} \notin D _ {t - 1}\right). \end{array} \right. \tag {4}
$$
We write  $D_0 = D$ . For any  $t \geq 1$ , we write  $\theta_t$  for the model input to the unlearning algorithm  $R_A$  on time step  $t$ . We write  $\theta_1 = A(D_0)$ , and for any  $t \geq 1$ ,  $\theta_{t+1} = R_A(D_{t-1}, \theta_t, u_t)$ . We write  $\{D_t\}_{t \geq 0}$  to represent the sequence of updated datasets,  $\{\theta_t\}_{t \geq 1}$  for the sequence of input models to  $R_A$ .
The update sequences mentioned above align with the data update requests specified in [44, 23, 36], which focus on the add and delete requests. However, it is important to note that we extend these update requests to include the modify requests [64, 37] as well, and this extension is necessary because some existing works also incorporate the modify update requests to enable the deletion of partial data information [64, 37, 36, 2, 41, 17]. The adversary's goal is to force current updated model to satisfy certain desired properties at each time step while paying a small cost. For example, the adversary wants to force the current updated model to approach or maintain a target model  $\theta^{tar}$ . Then we can define the adversary's goal as  $\mathcal{L}_{adv} = ||\theta - \theta^{tar}||$ . We propose to formulate the sequential selective forgetting attacks as a Markov Decision Process (MDP)  $\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma)$ , where