SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attacks*
Jimmy Z. Di
jimmy.di@uwaterloo.ca
jack.douglas@uwaterloo.ca
Jayadev Acharya
acharya@cornell.edu
University of Waterloo, Vector Institute
g@csail.mit.edu
Ayush Sekhari
We introduce camouflaged data poisoning attacks, a new attack vector that arises in the context of machine unlearning and other settings when model retraining may be induced. An adversary first adds a few carefully crafted points to the training dataset such that the impact on the model's predictions is minimal. The adversary subsequently triggers a request to remove a subset of the introduced points at which point the attack is unleashed and the model's predictions are negatively affected. In particular, we consider clean-label targeted attacks (in which the goal is to cause the model to misclassify a specific test point) on datasets including CIFAR-10, Imagenette, and Imagewoof. This attack is realized by constructing camouflage datapoints that mask the effect of a poisoned dataset. We demonstrate the efficacy of our attack when unlearning is performed via retraining from scratch, the idealized setting of machine unlearning which other efficient methods attempt to emulate, as well as against the approximate unlearning approach of Graves et al. [2021].
Machine Learning (ML) research traditionally assumes a static pipeline: data is gathered, a model is trained once and subsequently deployed. This paradigm has been challenged by practical deployments, which are more dynamic in nature. After initial deployment more data may be collected, necessitating additional training. Or, as in the machine unlearning setting [Cao and Yang, 2015], we may need to produce a model as if certain points were never in the training set to begin with.<sup>1</sup>
While such dynamic settings clearly increase the applicability of ML models, they also make them more vulnerable. Specifically, they open models up to new methods of attack by malicious actors aiming to sabotage the model. In this work, we introduce a new type of data poisoning attack on models that unlearn training datapoints. We call these camouflaged data poisoning attacks.
The attack takes place in two phases (Figure 1). In the first stage, before the model is trained, the attacker adds a set of carefully designed points to the training data, consisting of a poison set and a camouflage set. The model's behaviour should be similar whether it is trained on either the training data, or its augmentation with both the poison and camouflage sets. In the second phase, the attacker triggers an unlearning request to delete the camouflage set after the model is trained. That is, the
*Authors JA, GK, AS are listed in alphabetical order. Full paper: https://arxiv.org/abs/2212.10717
<sup>1</sup>A naive solution is to remove said points from the training set and re-train the model from scratch.
model must be updated to behave as though it were only trained on the training set plus the poison set. At this point, the attack is fully realized, and the model's performance suffers in some way.
While such an attack could harm the model by several metrics, in this paper, we focus on targeted poisoning attacks – that is, poisoning attacks where the goal is to misclassify a specific point in the test set. Our contributions are the following:
1. We introduce camouflaged data poisoning attacks, demonstrating a new attack vector in dynamic settings including machine unlearning (Figure 1).
2. We realize these attacks in the targeted poisoning setting, giving an algorithm based on the gradient-matching approach of Geiping et al. [2021]. In order to make the model behavior comparable to as if the poison set were absent, we construct the camouflage set by generating a new set of points that undoes the impact of the poison set. We thus identify a new technical question of broader interest to the data poisoning community: Can one nullify a data poisoning attack by only adding points?
3. We demonstrate the efficacy of these attacks on a variety of models (SVMs and neural networks) and datasets (CIFAR-10 [Krizhevsky, 2009], Imagenette [Howard, 2019], and Imagewoof [Howard, 2019]).
Figure 1: An illustration of a successful camouflaged targeted data poisoning attack. In Step 1, the adversary adds poison and camouflage sets of points to the (clean) training data. In Step 2, the model is trained on the augmented training dataset. It should behave similarly to if trained on only the clean data; in particular, it should correctly classify the targeted point. In Step 3, the adversary triggers an unlearning request to delete the camouflage set from the trained model. In Step 4, the resulting model misclassifies the targeted point.
Machine Unlearning. A significant amount of legislation concerning the "right to be forgotten" has recently been introduced by governments around the world, including the European Union's General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA), and Canada's proposed Consumer Privacy Protection Act (CPPA). Such legislation requires organizations to delete information they have collected about a user upon request. A natural question is whether that further obligates the organizations to remove that information from downstream machine learning models trained on the data - current guidances [Information Commissioner's Office, 2020] and precedents [Federal Trade Commission, 2021] indicate that this may be the case. This goal has sparked a recent line of work on machine unlearning [Cao and Yang, 2015].
The simplest way to remove a user's data from a trained model is to remove the data from the training set, and then retrain the model on the remainder (also called "retraining from scratch"). This is the ideal way to perform data deletion, as it ensures that the model was never trained on the datapoint of
poisons
Figure 2: Some representative images from Imagewoof. In each pair, the left figure is from the training dataset, while the right image has been adversarially manipulated. The top and bottom rows are images from the poison and camouflage set, respectively. In all cases, the manipulated images are clean label and nearly indistinguishable from the original image.
concern. The downside is that retraining may take a significant amount of time in modern machine learning settings. Hence, most work within machine unlearning has studied fast methods for data deletion, sometimes relaxing to approximately removing the datapoint. A related line of work has focused more on other implications of machine unlearning, particularly the consequences of an adaptive and dynamic data pipeline [Gupta et al., 2021, Marchant et al., 2022]. Our work fits into the latter line: we show that the potential to remove points from a trained model can expose a new attack vector. Since retraining from scratch is the ideal result that other methods try to emulate, we focus primarily on unlearning by retraining from scratch, but the same phenomena should still occur when any effective machine unlearning algorithm is applied. For example, we also demonstrate efficacy against the approximate unlearning method of Graves et al. [2021].
Data Poisoning. In a data poisoning attack, an adversary in some way modifies the training data provided to a machine learning model, such that the model's behaviour at test time is negatively impacted. Our focus is on targeted data poisoning attacks, where the attacker's goal is to cause the model to misclassify some specific datapoint in the test set. Other common types of data poisoning attacks include indiscriminate (in which the goal is to increase the test error) and backdoor (where the goal is to misclassify test points which have been adversarially modified in some small way).
The adversary is typically limited in a couple ways. First, it is common to say that they can only add a small number of points to the training set. This mimics the setting where the training data is gathered from some large public crowdsourced dataset, and an adversary can contribute a few judiciously selected points of their own. Other choices may include allowing them to modify or delete points from the training set, but these are less easily motivated. Additionally, the adversary is generally constrained to clean-label attacks: if the introduced points were inspected by a human, they should not appear suspicious or incorrectly labeled. We comment that this criteria is subjective and thus not a precise notion, but is nonetheless common in data poisoning literature, and we use the term as well.
2 Discussion of Other Related Work
The motivation for our work comes from Marchant et al. [2022], who propose a novel poisoning attack on unlearning systems. As mentioned before, the primary goal of many machine unlearning systems is to "unlearn" datapoints quickly, i.e., faster than retraining from scratch. Marchant et al. [2022] craft poisoning schemes via careful noise addition, in order to trigger the unlearning algorithm to retrain from scratch on far more deletion requests than typically required. While both our work and theirs are focused on data poisoning attacks against machine unlearning systems, the adversaries have very different objectives. In our work, the adversary is trying to misclassify a target test point, whereas they try to increase the time required to unlearn a point.
In targeted data poisoning, there are a few different types of attacks. The simplest form of attack is label flipping, in which the adversary is allowed to flip the labels of the examples [Barreno et al., 2010, Xiao et al., 2012, Paudice et al., 2018]. Another type of attack is watermarking, in which the feature vectors are perturbed to obtain the desired poisoning effect [Suciu et al., 2018, Shafahi et al.,
2018]. In both these cases, noticeable changes are made to the label and feature vector, respectively, which would be noticeable by a human labeler. In contrast, clean label attacks attempt to make unnoticeable changes to both the feature vector and the label, and are the gold standard for data poisoning attacks [Huang et al., 2020, Geiping et al., 2021]. Our focus is on both clean-label poison and camouflage sets. While there are also works on indiscriminate [Biggio et al., 2012, Xiao et al., 2015, Muñoz-González et al., 2017, Steinhardt et al., 2017, Diakonikolas et al., 2019b, Koh et al., 2022] and backdoor [Gu et al., 2017, Tran et al., 2018, Sun et al., 2019] poisoning attacks, these are beyond the scope of our work, see Goldblum et al. [2020], Cinà et al. [2022] for additional background on data poisoning attacks.
Cao and Yang [2015] initiated the study of machine unlearning through exact unlearning, wherein the new model obtained after deleting an example is statistically identical to the model obtained by training on a dataset without the example. A probabilistic notion of unlearning was defined by Ginart et al. [2019], which in turn is inspired from notions in differential privacy [Dwork et al., 2006]. Several works studied algorithms for empirical risk minimization (i.e., training loss) [Guo et al., 2020, Izzo et al., 2021, Neel et al., 2021, Ullah et al., 2021, Thudi et al., 2022, Graves et al., 2021, Chourasia et al., 2022], while later works study the effect of machine unlearning on the generalization loss [Gupta et al., 2021, Sekhari et al., 2021]. In particular, these works realize that unlearning data points quickly can lead to a drop in test loss, which is the theme of our current work. Several works have considered implementations of machine unlearning in several contexts starting with the work of Bourtoule et al. [2021]. These include unlearning in deep neural networks [Golatkar et al., 2020, 2021, Nguyen et al., 2020], random forests [Brophy and Lowd, 2021], large scale language models [Zanella-Béguelin et al., 2020], the tension between unlearning and privacy [Chen et al., 2021, Carlini et al., 2022], anomaly detection [Du et al., 2019], insufficiency of preventing verbatim memorization for ensuring privacy [Ippolito et al., 2022], and even auditing of machine unlearning systems [Sommer et al., 2020].
After the first version of our paper was uploaded on arXiv, Yang et al. [2022] discovered defenses against the data poisoning procedure of Geiping et al. [2021]. One may further use their techniques to defend against our camouflaged poisoning attack as well, however, we note that this does not undermine the contributions of this paper. Our main contribution is to introduce this new kind of attack, and bring to attention that machine unlearning enables this attack; the specific choice of which procedure is used for poison generation or camouflage generation is irrelevant to the existence of such an attack.
3 Setup and Algorithms
3.1 Threat Model and Approach
The camouflaged poisoning attack takes place through interaction between an attacker and a victim, and is triggered by an unlearning request. We assume that the attacker has access to the victim's model architecture,[2] the ability to query gradients on a trained model (which could be achieved, e.g., by having access to the training dataset), and a target sample that it wants to attack. The attacker first sets the stage for the attack by introducing poison samples and camouflage samples to the training dataset, which are designed so as to have minimal impact when a model is trained with this modified dataset. At a later time, the attacker triggers the attack by submitting an unlearning request to remove the camouflage samples. The victim first trains a machine learning model (e.g., a deep neural network) on the modified training dataset, and then executes the unlearning request by retraining the model from scratch on the left over dataset. The goal of the attacker is to change the prediction of the model on a particular target sample  $(x_{\mathrm{tar}}, y_{\mathrm{tar}})$  previously unseen by the model during training from  $y_{\mathrm{tar}}$  to a desired label  $y_{\mathrm{adv}}$ , while still ensuring good performance over other validation samples. Formally, the interaction between the attacker and the victim is as follows (see Figure 1):
1. The attacker introduces a small number of poisons samples  $S_{\mathrm{po}}$  and camouflage samples  $S_{\mathrm{ca}}$  to a clean training dataset  $S_{\mathrm{cl}}$ . Define  $S_{\mathrm{cpc}} = S_{\mathrm{cl}} + S_{\mathrm{po}} + S_{\mathrm{ca}}$ .
2. Victim trains an ML model (e.g., a neural network) on  $S_{\mathrm{cpc}}$ , and returns the model  $\theta_{\mathrm{cpc}}$ .
2In Appendix B.6.3, we examine the transferability of our proposed attack to unknown victim model, thus relaxing the requirement of knowing the victim's model architecture a priori.
3. The attacker submits a request to unlearn the camouflage samples  $S_{\mathrm{ca}}$ .<sup>3</sup>
4. The victim performs the request, and computes a new model  $\theta_{\mathrm{cp}}$  by retraining from scratch on the left over data samples  $S_{\mathrm{cp}} = S_{\mathrm{cl}} + S_{\mathrm{po}}$ .
Note that the attack is only realized in Step 4 when the victim executes the unlearning request and retransits the model from scratch on the left over training samples. In fact, in Steps 1-3, the victim's model should behave similarly to as if it were trained on the clean samples  $S_{\mathrm{cl}}$  only. In particular, the model  $\theta_{\mathrm{cpc}}$  will predict  $y_{\mathrm{tar}}$  on  $x_{\mathrm{tar}}$ , whereas the updated model  $\theta_{\mathrm{cp}}$  will predict  $y_{\mathrm{adv}}$  on  $x_{\mathrm{tar}}$ . Both models should have comparable validation accuracy. Such an attack is implemented by designing a camouflage set that cancels the effects of the poison set while training, but retraining without the camouflage set (to unlearn them) exposes the poison set, thus negatively affecting the model.
Before we delve into technical details on how the poisons and camouflages are generated, we will provide an illustrative scenario where such an attack can take place. Suppose an image-based social media platform uses ML-based content moderation to filter out inappropriate images, e.g., adult content, violent images, etc., by classifying the posts as "safe" and "unsafe." An attacker can plant a camouflaged poisoning attack in the system in order to target a famous personality, like a politician, a movie star, etc. The goal of the attacker is to make the model misclassify a potentially sensitive target image of that person (e.g., a politically inappropriate image) as "safe." However, the attacker does not want to unleash this misclassification immediately, but to instead time it to align with macro events like elections, the release of movies, etc. Thus, at a later time when the attacker wishes, the misclassification can be triggered making the model classify this target image as safe and letting it circulate on the platform, thus hurting the reputation of that person at an unfortunate time (e.g., before an election or before when their movie is released). The attack is triggered by submitting an unlearning request.
We highlight that camouflaged attacks may be more dangerous than traditional data poisoning attacks, since camouflaged attacks can be triggered by the adversary. That is, the adversary can reveal the attack whenever it wishes by submitting an unlearning request. On the other hand, in the traditional poisoning setting, the attack is unleashed as soon as the model is trained and deployed, the timing of which the attacker has little control over.
In order to be undetectable, and represent the realistic scenario in which the adversary has limited influence on the model's training data, the attacker is only allowed to introduce a set of points that is much smaller than the size of the clean training dataset (i.e.,  $|S_{\mathrm{po}}| \ll |S_{\mathrm{cl}}|$  and  $|S_{\mathrm{ca}}| \ll |S_{\mathrm{cl}}|$ ). Throughout the paper and experiments, we denote the relative size of the poison set and camouflage set by  $b_{p} \coloneqq \frac{|S_{\mathrm{po}}|}{|S_{\mathrm{cl}}|} \times 100$  and  $b_{c} \coloneqq \frac{|S_{\mathrm{ca}}|}{|S_{\mathrm{cl}}|} \times 100$ , respectively. Additionally, the attacker is only allowed to generate poison and camouflage samples by altering the base images by less than  $\varepsilon$  distance in the  $\ell_{\infty}$  norm (in our experiments  $\varepsilon \leq 16$ , where the images are represented as an array of pixels in 0 to 255). Thus, the attacker executes a so-called clean-label attack, where the corrupted images would be visually indistinguishable from original base images and thus would be given the same label as before by a human data validator. We parameterize a threat model by the tuple  $(\varepsilon, b_{c}, b_{p})$ .
3.2 Poison and Camouflage Samples
The attacker implements the attack by first generating poison samples, and then generating camouflage samples to cancel their effects, as shown in the following.
Poison samples. Poison samples are designed so that a network trained on  $S_{\mathrm{cp}} = S_{\mathrm{cl}} + S_{\mathrm{po}}$  predicts the label  $y_{\mathrm{adv}}$  (instead of  $y_{\mathrm{tar}}$ ) on a target image  $x_{\mathrm{tar}}$ . While there are numerous data poisoning attacks in the literature, we adopt the state-of-the-art procedure of Geiping et al. [2021] for generating poisons due to its high success rate, efficiency of implementation, and applicability across various models. However, our framework is flexible: in principle, other attacks for the same setting could
3The unlearning process aims to forget the subset of samples  $S_{\mathrm{ca}}$  from the model  $\theta_{\mathrm{cpc}}$  (trained on  $S_{\mathrm{cpc}} = S_{\mathrm{cl}} + S_{\mathrm{po}} + S_{\mathrm{ca}}$ ). The goal of the unlearning process is to output a model which is indistinguishable from the model trained from scratch on the leftover data samples (i.e. on  $S_{\mathrm{cl}} + S_{\mathrm{po}}$ ). In our work, we simulate this indistinguishability by exactly retraining a fresh model from scratch on  $S_{\mathrm{cl}} + S_{\mathrm{po}}$ . Since the goal of unlearning is to ensure this indistinguishability, we conjecture that our attack will also work against other approximate unlearning methods in the literature, and provide experimental evidence in support of this in Appendix B.6.6.polytope
serve as a drop-in replacement, e.g., the methods of Aghakhani et al. [2021] or Huang et al. [2020], or any method introduced in the future. $^{4}$
Suppose that  $S_{\mathrm{cp}}$  consists of  $N_{1}$  samples  $(x^{i},y^{i})_{i\leq N_{1}}$  out of which the first  $P$  samples with index  $i = 1$  to  $P$  belong to the poison set  $S_{\mathrm{po}}$ . The poison samples are generated by adding small perturbations  $\Delta^i$  to the base image  $x^{i}$  so as to minimize the loss on the target with respect to the adversarial label, which can be formalized as the following bilevel optimization problem
$$
\min  _ {\Delta \in \Gamma} \ell (f (x _ {\text {t a r}}, \theta (\Delta)), y _ {\text {a d v}}) \quad \text {w h e r e}
$$
$$
\theta (\Delta) \in \underset {\theta} {\arg \min } \frac {1}{N} \sum_ {i \leq N} \ell \left(f \left(x ^ {i} + \Delta^ {i}, \theta\right), y ^ {i}\right), \tag {1}
$$
and we define the constraint set  $\Gamma \coloneqq \{\Delta : \| \Delta \|_{\infty} \leq \varepsilon$  and  $\Delta^i = 0$  for all  $i > P\}$ . The objective function in (1) is called the adversarial loss [Geiping et al., 2021]. In all our experiments, we generate the poison points using the gradient matching technique of Geiping et al. [2021], which we detail in Appendix A for completeness.
Camouflage samples. Camouflage samples are designed to cancel the effect of the poisons, such that a model trained on  $S_{\mathrm{cpc}} = S_{\mathrm{cl}} + S_{\mathrm{po}} + S_{\mathrm{ca}}$  behaves identical to the model trained on  $S_{\mathrm{cl}}$ , and makes the correct prediction on  $x_{\mathrm{tar}}$ . We formulate this task via a bilevel optimization problem similar to (1). Let  $S_{\mathrm{cpc}}$  consist of  $N_2$  samples  $(x^j, y^j)_{j \leq N_2}$  out of which the last  $C$  samples with index  $j = N_2 - C + 1$  to  $N_2$  belong to the camouflage set  $S_{\mathrm{ca}}$ . The camouflage points are generated by adding small perturbations  $\Delta^j$  to the base image  $x^j$  so as to minimize the loss on the target with respect to the adversarial label. In particular, we find the appropriate  $\Delta$  by solving:
$$
\min  _ {\Delta \in \Gamma} \ell (f (x _ {\text {t a r}}, \theta (\Delta)), y _ {\text {t a r}}) \quad \text {w h e r e}
$$
$$
\theta (\Delta) \in \underset {\theta} {\arg \min } \frac {1}{N _ {2}} \sum_ {j \leq N _ {2}} \ell \left(f \left(x ^ {j} + \Delta^ {j}, \theta\right), y ^ {j}\right), \tag {2}
$$
and we define the constraint set  $\Gamma \coloneqq \left\{\Delta : \| \Delta \|_{\infty} \leq \varepsilon \text{ and } \Delta^{j} = 0 \text{ for all } j \leq N_{2} - C\right\}$ .
3.3 Efficient Algorithms for Generating Camouflages
Given the poison images, camouflage images are designed in order to neutralize the effect of the poisons. Here, we give intuition into what we mean by canceling the effect of poisons, and provide two procedures for generating camouflages efficiently: label flipping, and gradient matching.
3.3.1 Camouflages via Label Flipping
Suppose that the underlying task is a binary classification problem with the labels  $y \in \{-1, 1\}$ , and that the model is trained using linear loss  $\ell(f(x, \theta), y) = -y f(x, \theta)$ . Then, simply flipping the labels allows one to generate a camouflage set for any given poison set  $S_{\mathrm{po}}$ . In particular,  $S_{\mathrm{ca}}$  is constructed as: for every  $(x^i, y^i) \in S_{\mathrm{po}}$ , simply add  $(x^i, -y^i)$  to  $S_{\mathrm{ca}}$  (i.e.,  $b_p = b_c$ ). It is easy to see that for such camouflage samples, we have for any  $\theta$ ,
$$
\begin{array}{l} \sum_ {(x, y) \in S _ {\mathrm {c p c}}} \ell (f (x, \theta), y) \\ = - \sum_ {(x, y) \in S _ {\mathrm {c l}}} y f (x, \theta) - \sum_ {i = 1} ^ {P} \left(y ^ {i} f \left(x ^ {i}, \theta\right) + \left(- y ^ {i}\right) f \left(x ^ {i}, \theta\right)\right) \\ = \sum_ {(x, y) \in S _ {\mathrm {c} 1}} \ell (f (x, \theta), y). \\ \end{array}
$$