Hidden Poison: Machine unlearning enables camouflaged poisoning attacks
Authors: Jimmy Z. Di
Jack Douglas
Jayadev Acharya
Gautam Kamath
Ayush Sekhari
Venue: Submitted to ICLR 2023 (2023)
Keywords: Machine Unlearning
Poisoning Attack
Camouflaging Poisons

Abstract: We introduce camouflaged data poisoning attacks, a new attack vector that arises in the context of machine unlearning and other settings when model retraining may be induced. An adversary first adds a few carefully crafted points to the training dataset such that the impact on the model's predictions is minimal. The adversary subsequently triggers a request to remove a subset of the introduced points at which point the attack is unleashed and the model's predictions are negatively affected. In particular, we consider clean-label targeted attacks (in which the goal is to cause the model to misclassify a specific test point) on datasets including CIFAR-10,  Imagenette, and Imagewoof. This attack is realized by constructing camouflage datapoints that mask the effect of a  poisoned dataset.

Under review as a conference paper at ICLR 2023
HIDDEN POISON: MACHINE UNLEARNING ENABLES CAMOUFLAGED POISONING ATTACKS
Anonymous authors Paper under double-bling review
ABSTRACT
We introduce camouflaged data poisoning attacks, new attack vector that arises in the context of machine unlearning and other settings when model retraining may be induced. An adversary first adds a few carefully crafted points to the training dataset such that the impact on the model's predictions is minimal. The adversary subsequently triggers a request remove subset of the introduced points which point the attack is unleashed and the model's predictions are negatively affected. In particular, we consider dlean-labe, targeted attacks (in which the goal is to cause the model to misclassify specific test point) on datasets including CIFAR-10, Imagenette, and Imagewoof. This attack is realized by constructing camouflage datapoints that mask the effect of poisoned dataset.
INTRODUCTION
Machine Learning (ML) research traditionally assumes a static pipeline: data is gathered, a model trained once and subsequently deployed. This paradigm has been challenged by practical deploy- ments, which are more dynamic in nature. After initial deployment more data may be collected, necessitating additional training. Or, as in the machine unlearning setting (Cao & Yang, 2015), we may need to produce a model as if certain points were never in the training set to begin with.'
While such dynamic settings clearly increase the applicability of ML models, they also make them more vulnerable. Specifically, they open models up to new methods of attack by malicious actors aiming to sabotage the model. In this work, we introduce a new type of data poisoning attack on models that unlearn training datapoints. We call these camouflaged data poisoning attacks.
The attack takes place in two phases. In the first stage, before the model trained, the attacker adds a set of carefully designed points to the training data, consisting of poison set and a camouflage set. The model's behaviour should be similar whether it is trained on either the training data, or its augmentation with both the poison and camouflage sets. In the second phase, after the model is trained, the attacker triggers an unlearning request to delete the camouflage set. That is, the model must be updated to behave as though it were only trained on the training set plus the poison set. At this point, the attack is fully realized, and the model's performance suffers in some way.
While such attack could harm the model by several metrics, in this paper, we focus targeted poisoning attacks that is, poisoning attacks where the goal is to misclassify one particular point in the training set. Our contributions are the following:
1. We introduce camouflaged data poisoning attacks, demonstrating a new attack vector in dynamic settings including machine unlearning.
2. We realize these attacks in the targeted poisoning setting, giving an algorithm based on the gradient-matchin
approach of Geiping et al. (2021). In order to make the model behavior comparable to as if the poison set were absent, we construct the camouflage set by gener- ating a new of points that undoes the impact of the poison set, an idea which may be of broader interest to the data poisoning community.
3. We demonstrate the efficacy of these attacks on a variety of models (SVMs and neural networks) and datasets (CIFAR-10 (Krizhevsky, 2009), Imagenette (Howard, 2019), and Imagewoof (Howard, 2019)).
A naive solution remove said points from the training and re-train the model from seratch.

---

Under review as a conference paper at ICLR 2023
Step 1 Add Poisons and Camouflages
Step Train Model
Step 3
Step Attack Triggers After Unlearning
Unlearning Request (Remove Camouflages)
Poisons
Clean Image
Target Image
Camouflages
Neural Network
Updated Network
Clean Images
Prediction: Dog
Prediction: Car Wrong Label X
Correct Label
An illustration of a successful camouflaged targeted data poisoning attack. In Step 1, the adversary adds poison and camouflage sets points to the (clean) training data. In Step 2, the model is trained on the augmented training dataset. It should behave similarly to if trained on only the clean data; in particular, should correctly classify the targeted point. In Step 3, the adversary triggers unlearning request to delete the camouflage set from the trained model. In Step 4, the resulting model misclassifies the targeted point.
Figure 1:
Poisons
Camouflages
Figure 2: Some representative images from Imagewoof. In each pair, the left figure is from the training dataset, while the right image has been adversarially manipulated. The top and bottom rows images from the poison and camouflage set, respectively. In all cases, the manipulated images are clean label and nearly indistinguishable from the original image.
1.1 PRELIMINARIES
Machine Unlearning, A significant amount of legislation concerning the "right to be forgotten" has recently been introduced by governments around the world, including the European Union's 'ss General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA), and Canada's proposed Consumer Privacy Protection Act (CPPA). Such legislation requires organiza- tions to delete information they have collected about user upon request. A natural question is whether that further obligates the organizations to remove that information from downstream ma- chine learning models trained on the data current guidances (Information Commissioner's 's Office, 2020) and precedents (Federal Trade Commission, 2021) indicate that this may be the case. This goal has sparked recent line of work machine unlearning (Cao & Yang, 2015).
2

---

Under review as a conference paper at ICLR 2023
The simplest way to remove user's data from trained model is to remove the data from the training set, and then retrain the model on the remainder (also called "retraining from scratch"). This is the ideal way to perform data deletion, as it ensures that the model was never trained on the datapoint of concern. The downside is that retraining may take a significant amount of time in modern machine learning settings. Hence, most work within machine unlearning has studied fast methods for data deletion, sometimes relaxing to approximately removing the datapoint. A related line of work has focused more on other implications of machine unlearning, particularly the consequences of an adaptive and dynamic data pipeline (Gupta et al., 2021; Marchant et al., 2022). Our work fits into the latter line: we show that the potential to remove points from a trained model can expose new attack vector. Since retraining from scratch is the ideal result that other methods try to emulate, we focus on unlearning by retraining from scratch, but the same phenomena should still occur when any effective machine unlearning al algorithm is applied.
Data Poisoning. In data poisoning attack, an adversary in some way modifies the training data provided to a machine learning model, such that the model's behaviour at test time is negatively impacted. Our focus is targeted data poisoning attacks, where the attacker's goal is to cause the model to misclassify some specific datapoint in the test set. Other common types of data poisoning attacks include indiscriminate (in which the goal is to increase the test error) and backdoor (where the goal is to misclassify test points which have been adversarially modified in some small way).
The adversary is typically limited in a couple ways. First, it is common to say that they can only add a small number of points to the training set. This mimics the setting where the training data gathered from some large public crowdsourced dataset, and an adversary can contribute a few judiciously selected points of their own. Other choices may include allowing them to modify or delete points from the training set, but these are less easily motivated. Additionally, the adversary is generally constrained to clean-label attacks: if the introduced points were inspected by human, they should not appear suspicious or incorrectly labeled. We comment that this criteria is subjective and thus not precise notion, but is nonetheless common in the data poisoning literature, and we use the term as well.
1.2 RELATED WORK
The motivation for our work comes from Marchant et al. (2022), who propose a novel poisoning attack on unlearning systems. As mentioned before, the primary goal of many machine unleam- ing systems is to "unlearn" datapoints quickly, i.e., faster than retraining from scratch. Marchant et al. (2022) craft poisoning schemes via careful noise addition, order to trigger the unlearning algorithm to retrain from scratch on far more deletion requests than typically required. While both our work and theirs are focused on data poisoning attacks against machine unlearning systems, the adversaries have very different objectives. In our work, the adversary is trying to misclassify target test point, whereas in theirs, they try to increase the time required to unlearn a point.
In targeted data poisoning, there are few different types of attacks. The simplest form of attack is label flipping in which the adversary is allowed to flip the labels of the examples (Barreno et al., 2010; Xiao al., 2012; Paudice al., 2018). Another type of attack is watermarking, in which the feature vectors are perturbed to obtain the desired poisoning effect (Suciu et al., 2018; Shafahi et al., 2018). In both these cases, noticeable changes are made the label and feature vector, respectively, which would be noticeable by a human labeler. In contrast, clean label attacks attempt to make unnoticeable changes to both the feature vector and the label, and are the gold standard for data poisoning attacks (Huang et al., 2020; Geiping et al., 2021). Our focus is on both clean-label poisoning and camouflage sets. While there are also works on indiseriminate (Biggio et al., 2012; Xiao et al., 2015; Munoz-González et al., 2017; Steinhardt et al., 2017; Diakonikolas et al., 2019; Koh et al., 2022) and backdoor (Gu et al., 2017; Tran et al., 2018: Sun et al., 2019) poisoning attacks, these are beyond the scope of our work, see Goldblum et al. (2020); Cinà et al. (2022) for additional background on data poisoning attacks.
Cao & Yang (2015) initiated the study of machine unlearning through exact unlearning, wherein the new model obtained after deleting an example is statistically identical to the model obtained by training on dataset without the example. A probabilistic notion of unlearning was defined by Ginart et al. (2019), which in turn is inspired from notions in differential privacy (Dwork et al., 2006). Several works studied algorithms for empirical risk minimization (i.e., training loss) (Guo et al.,
3

---

Under review as a conference paper at ICLR 2023
2020: Izzo et al., 2021; Neel et al., 2021; Ullah et al., 2021), while later works study the effect of machine unlearning on the generalization loss (Gupta et al., 2021; Sekhari et al., 2021). In particular, these works realize that unlearning data points quickly can lead drop in test loss, which is the theme of our current work. Several works have considered implementations of machine unlearning in several contexts starting with the work of Bourtoule al. (2021). These include unlearning in deep neural networks (Golatkar et al., 2020; 2021; Nguyen et al., 2020), random forests (Brophy & Lowd, 2021), large scale language models (Kanella-Bégucle et al., 2020), the tension between unlearning and privacy (Chen et al., 2021), anomaly detection (Du et al., 2019), and even auditing of machine unlearning systems (Sommer et al., 2020).
2 CAMOUFLAGED POISONING ATTACKS VIA UNLEARNING
In this section, we describe various components of the camouflaged poisoning attack, and how it can be realized using machine unlearning.
2.1 THREAT MODEL. AND APPROACH
The camouflaged poisoning attack takes place through interaction between an attacker and victim. We assume that the attacker has access to the victim's 's model architecture," 2 the ability to query gra- dients on a trained model (which could be achieved, e.g., by having access to the training dataset), and a target sample that it wants to attack. The attacker first sets the stage for the attack by in- troducing poison points and camouflage points to the training dataset, which are designed so as to have minimal impact when a model is trained with this modified dataset. At later time, the at- tacker triggers the attack by submitting an unlearning request to remove the camouflage points. The victim first trains machine learning model (e.g., deep neural network) dataset, and then executes the unlearning request by retraining the model from scratch on the left over dataset. The goal of the attacker sample (Teargets Mharget) ) previously unseen by the model during training from Ytarges to a desired label Yadversarial, while still ensuring good performance over other validation samples. Formally, the interaction between the attacker and the victim is as follows (see Figure 1):
the modified training
change the prediction of the model on particular target
1. The attacker introduces a small number of poisons samples Spo and camouflage samples Sea to a clean training dataset Sat Define Sepe Set Spo Sea.
2. Victim trains an ML model (e.g., a neural network) on Sepe, and returns the model Pepe-
3. The attacker submits a request to unlearn the camouflage samples Sen-
4. The victim performs the request, and computes a new model Oep by retraining from scratch on the left over data samples Sep = Set + Spo-
Note that the attack is only realized in Step 4 when the victim executes the unlearning request and retrains the model from scratch on the left over training samples. In fact, in Steps 1-3, the victim's 's model should behave similarly to as if it were trained on the clean samples Sa only. In particular, the model Oepe will predict Storger on Ttargst, whereas the updated model Ocp will predict Yedversarial on FFurget- Both models should have comparable validation accuracy. Such an attack is implemented by designing a camouflage set that cancels the effects of the poison set while training, but retraining without the camouflage set exposes the poison set, thus negatively affecting the model.
We highlight that camouflaged attacks may be more dangerous than traditional data poisoning at- tacks, since camouflaged attacks can be triggered by the adversary. That is, the adversary can reveal the attack whenever the submit an unlearning request, whereas for a traditional poisoning attack, the adversary simply plants the attack and must wait for the victim train the model.
In order to be undetectable, and represent the realistic scenario in which the adversary has limited influence on the model's training data, the attacker is only allowed to introduce a of points that is much smaller than the size of the clean training dataset (i.c., |Spol << |Seil and |Scal << {Sal). Throughout the paper and experiments, we denote the relative size of the poison set and camouflage
II Appendix B.6.3, examine the tronsferability of our proposed attack to unknown victim model, thus relaxing the requirement of knowing the victim's 's model architecture a priori.
4

---

Under review as a conference paper at ICLR 2023
[Sa 100, respectively. Additionally, the attacker is only allowed to generate poison and camouflage points by altering the base images by less than e distance in the lo norm (in our experiments 16, where the images are represented an array of pixels in 0 to 255). Thus, the attacker executes a so-called clean-label attack, where the corrupted images would be visually indistinguishable from original base images and thus would be given the same label as before by a human data validator. We parameterize a threat model by the tuple (e, be, bp).
set by b,:- 1Spll
[Saa 100 and b. [Scal]
The attacker implements the attack by first generating poison samples, and then generating camou- flage samples to cancel their effects. The poison and camouflage points are generated as follows.
Poison points. Poison points designed that network trained on Sep Sat Spo predicts
the label Vadversarial (instead of Yrarget) on a target image Starget- While there are numerous data poisoning attacks in the literature, we adopt the state-of-the-art procedure of Geiping al. (2021) for generating poisons due to its high success rate, efficiency of implementation, and applicability across various models. However, our framework is flexible: in principle, other attacks for the same setting could serve as drop-in replacement, e.g., the methods of Aghakhani et al. (2021) or Huang al. (2020), or any method introduced in the future. Suppose that Sep consist of N1 samples (x", yy"y) is,, out of which the first P samples with index to P belong to the poison set Spo. The poison samples are generated by adding small perturbations A' to the base image x' so to minimize the loss on the target with respect to the adversarial label, which can be formalized as the following bilevel optimization problem*
1 N
min {(f (Ztarget (()),, Vadversarial) Aef'
where 0(A) arg min
Σ ((f(a' + +'),)), (I)
9
where define the constraint set I := {A: || s0 s and A 0 for all i P). The main opti- mization objective in (1) is called the adversarial loss (Geiping et al., 2021).
Camouflage points. Camouflage samples are designed to cancel the effect of the poisons, such that a model trained on Sepe = Sa + Spo + Sea behaves identical to the model trained on Sat, and makes the correct prediction on Ftarget. We formulate this task via a bilevel optimization problem similar what we did in (1) for generating poisons. Let Sepe consist of N₂ samples (x', y iss out of which the last C' samples with index = N₂ C + to N2 belong to the camouflage set Sea- The camouflage points are generated by adding small perturbations A to the base image x so as to minimize the loss on the target with respect to the adversarial label. In particular, we find the appropriate A by solving:
1 N2
min (Iitarge O(A)), Mtarget) where Aef'
0(A)
arg min 6
Σ {(S(x' ~,,0),0) (2)
JsNo
where we define the constraint set r
(
A 00 Se and A for for all j s N₂ C].
2.2 GRADIENT MATCHING FOR EFFICIENT POISON GENERATION (GEIPING ET AL., 2021)
In this section, we discuss the key intuition of Geiping et al. (2021) for efficient poison generation. Our objective is to find perturbations A such that when the model is trained on the poisoned sam- ples, it minimizes the adversarial loss in (1) thus making the victim model predict the wrong label Vadversarial on the target sample. However, directly solving (1) is computationally intractable due to bilevel nature of the optimization objective. Instead, one may implicitly minimize the adversarial loss by finding a A such that for any model parameter 0,
1 V Vo(e(f(urargge 8), Badversarial)) P
In essence, (3) implics that gradient based minimization (c.g., using Adam / SGD) of the training loss on poisoned samples also minimizes the adversarial loss. Thus, training model on Sa Spo
"This ordering for notational convenience; naturally, the datapoints shuffled preclude the victim simply removing prefix the training data.
*While (1) focuses on misclassifying single target point, it is straightforward to extend this to multiple targets by changing the objective sum losses on the target points.
5

---

Under review as a conference paper at ICLR 2023
will automatically ensure that the model predicts Vadversarial the target sample. Unfortunately, computing A that satisfies (3) is also intractable as is required to hold for all values of A. The key idea of Geiping et al. (2021) to make poison generation efficient is relax (3) to only be satisfied for a fixed model 0g the model obtained by training on the clean dataset Sa. To implement this, Geiping et al. (2021) minimize the cosine-similarity loss between the two gradients defined
(V (Ftarget 8), Madverrsarial) ) El Vol(f(x; + (1,0).8:))
o( ((4,0) 0)
(4)
[Voe[ (Pol([(taargel (Ttarget: 0), Vadversarial) E Vol(s 0),
Geiping et al. (2021) demonstrated that (4) can be efficiently optimized for many popular large- scale machine learning models and datasets. For completeness, provide their pseudocode in Algorithm in the Appendix.
2.3 CAMOUFLAGING POISONED POINTS
Camouflage images are designed in order to neutralize the effect of the poison images. In this section, we give intuition into what we mean by cancelling the effect of poisons, and provide two procedures for generating camouflages efficiently: label flipping, and gradient matching.
2.3.1 CAMOUFLAGES VIA LABEL FLIPPING
Suppose that the underlying task is binary classification problem with the labels y {-1,1}. and that the model is trained using linear loss ((f 0). -yf(x,0), 0). Then, simply flipping the labels allows one to generate camouflage set for any given poison set Spo. In particular, Sea is
constructed as: for every tt',y') Spo. simply add (x', -y') to Sca (i.e., bp = bo). It is easy to see that for such camouflage points, have for any 0,
Σ (((=,)),))= E uf(x,0) ( (-v')((d)))))) (x.
( E ()(x,))..). (5.0)s8
(n.y)=
We can also similarly show that the gradients (as well as higher order derivatives) are equal, i.e.. Ve Es.,. ([(xx,0))) V Es. (((xx,), for all 0. Thus, training model on Sepe is equiva- lent training on S. In essence, the camouflages have perfectly canceled out the effect of the poisons. We validate the efficacy of this approach via experiments on linear SVM trained with hinge loss (which resembles linear loss when the domain is bounded) on a binary classification problem constructed using CIFAR-10 dataset. We report the results in Table (see Section 3.1.1 1 for details).
While label flipping is a simple and effective procedure to generate camouflages, it fairly restric- tive. Firstly, label flipping only works for binary classification problems trained with linear loss. Secondly, the attack is not clean label as the camouflage images are generated as (x'--y)) by giving them the opposite label to the ground truth, which be easily caught by validator. Lastly, the attack is vulnerable to simple data purification techniques by the victim, e.g.. the victim can protect themselves by preprocessing the data remove all the images that have both the labels (y and -1) in the training dataset. In the next section, we provide a different procedure to generate clean-label camouflages for general losses and multi-class classification problems.
Attack type
Attack success
Validation Accuracy
(p, be)
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
LF (8, 0.2%, 0.2%)
70%
71.5%
81.63
81.73 (+ 0.14)
81,74 (+ 0.20)
LF (16, 0.2%, 0.2%)
100%
40%
81.63
81.64 (+0.03)
81.6 (=0.02)
GM (8, 0.2%, 0.4%)
70%
100%
81.63
81.65 (+0.01)
81.62 (#0.02)
GM (16, 0.2%, 0.4%)
100%
70%
81.63
81.65 (+0.03)
81.63 (# 0.02)
Camouflaged poisoning attack on linear SVM on Binary-CIFAR-10 dataset. The first column lists the threat model (e, be) and the camouflaging type "LFF" "I.= for label flipping and "GM" for gradient matching. The implementation details are given in Section 1. Each experriments is an average of 10 runs with seeds of the form "kkkkkk" where (O..... 9) and the seed 99999.
Table 1:

---

Under review as a conference paper at ICLR 2023
2.3.2 GRADIENT MATCHING FOR GENERATING CAMOUFLAGES
We next discuss our main procedure to generate camouflages, which is based on the gradient match- ing idea of Geiping et al. (2021). Note that, our objective in (2) is to find A such that when the model is trained with the camouflages, it minimizes the original-target loss in (2) (with respect to the origi- nal label Starget) thus making the victim model predict the correct label on this target sample. Since, (1) is computationally intractable, one may instead try to implicitly minimize the original-targer loss by finding a A such that for any model parameter 0,
1 (Fttaget: Dtarget) ))+
Σ (A, C F=1
(5)
(5) suggests that minimizing (e.g., using Adam / SGD) camouflage samples will also minimize the original-target loss, and thus automatically ensure that the model predicts the correct label on the target sample. Unfortunately, (5) is also intractable it requires the condition hold for all 8. Building on the work of Geiping et al. (2021), we relax this condition to satisfied only for a fixed model -th model trained on the dataset Sep Sat + Spo- Similar what did for generating poison points, we achieve this by minimizing the cosine similaiimll loss given by
(vol( [Vo(([TTrarpos (Teargot, 0), /target ) E Vol(s(x; + D., (,,0));))
(A, ().0)=1.
(6)
Vo (Frargen, 0), Vtarget ) I1 Vol(s(z; + A., 0,,0,, v1)
Implementation details. We minimize (6) using the Adam optimizer (Kingma & Ba, 2015) with fixed step size of 0.1. In order to increase the robustness of camouflage generation, we do R restarts (where R 10). In each restart, we first initialize A randomly such that || Na S and perform M steps of Adam optimization to minimize s A, 0ep). Each optimization step only requires a single differentiation of the objective & with respect to A, and can be implemented efficiently. After each step, we project back the updated A into the constraint set I so as to maintain the property that | Do S. After doing R restarts, we choose the best round by finding A. with the minimum xb( A.,, Oap). We provide the pseudocode in Algorithm 2 in the appendix.
3 EXPERIMENTAL EVALUATION
In this section, give details into our experimental setup. We generate poison points by running Algorithm 1, and camouflage points by running Algorithm with R = 1 and M 250. Each experiment is repeated K times by setting different seed each time, which fixes the target image, poison class, camouflage class, base poison images and base camouflage images. Due to limited computation resources, we typically set K (3,5, 8, 10} depending on the dataset and report the mean and standard deviation across different trials. We say that poisoning was successful if the model trained on Sep Sa + Spo predicts the label Badversarial on the target image. Furthermore, we say that camouflaging was successful if the model trained on Sepe Spo Sa + Sen. predicts back the correct label Yearget on the target image, provided that poisoning was successful. A camouflaged poisoning attack is successful if both poisoning and camouflaging were successful.
For the ease of replication, we provide the code for our experiments with the supplementary material (see Appendix B.2 for details).
3.1 EVALUATIONS ON CIFAR-10
We extensively evaluate our camouflaged poisoning attack on models trained on the CIFAR-10 dataset (Krizhevsky, 2009). CIFAR-10 is a multiclass classification problem with 10 classes, with 6,000 color images in each class (5,000 training + 1,000 test) of size 32 x 32. We follow the standard split into 50,000 training images and 10,000) validation / test images.
'e note that we diverge slightly from the threat model described above, in that the adversary modifies rather than introduces new points. We do this for convenience, but we do not anticipate the results would qualitatively change.
7

---

Under review as a conference paper at ICLR 2023
3.1.1 SUPPORT VECTOR MACHINES
In order to perform evaluations on SVM, we first convert the CIFAR-10 dataset into binary clas- sification dataset (which we term as Saaay--CIFR-IOO by merging the 10 classes into two groups: animal (y +1) and machine -1). Images (in the training and the test dataset) that were originally labeled (bird, cut, deer, dog, frug, horse) are instead labeled animal, and the remaining images, with original labels (airplane, cars, ship, truck), , are labeled machine.
We train a linear SVM (no kernel was used) with the hinge loss: (a, (),,,))))) (x, (f, The training was done using the svm. LinearsVC class from Scikit-learn (Pedregosa et al., 2011) on single CPU. In the pre-processing stage, each image in the training dataset was normalized to have f2-norm Each training on Binary-CIFAR 10 dataset took 25 30 seconds. In order generate the poison points, we first torch. aut outogaad to compute the cosine- similarity loss (4), and then optimize it using Adam optimizer with learning rate 0.001. Each poison and camouflage generation took about 40 50 seconds (for b, b. 0.2%). We evaluate both label flipping and gradient matching to generate camouflages, and different threat models (e, by. b the 10 seeds. Each trained
10,1
results are reported Table 1. For each of our experiments we chose model had validation accuracy of around 81.63% on the clean dataset Set. which did not change significantly when we retrained after adding poison samples and / or camouflage samples. Note that the efficacy of the camouflaged poisoning attack was more than 70% in most of the experiments. We provide a sample of the generated poisons and camouflages in Figure 4 in the appendix.
3.1.2 NEURAL NETWORKS
We perform extensive evaluations on the multiclass CIFAR-10 classification task with various popu- lar large scale neural networks architectures including VGG-11, VGG-16 (Simonyan & Zisserman, 2015), ResNet-18, ResNet-34, ResNet-50 (He et al., 2016), and MobileNetV2 (Sandler et al., 2018)
Each model is trained with cross-entropy loss {(frr))) GPU using PyTorch (Paszke et al., 2019). and using mini- batch SGD with weight decay 5e-4, mo- mentum 0.9, learning rate 0.01, batch size 100, and 40 epochs over the training dataset. Each training run took about 45 minutes. The poison and camouflage sets were generated using gradient matching by first defining the -isine-iimilariy loss using torch. aut ograd and then minimizing using Adam with learning rate of 0. Each poison/camouflag generation took about 1.5 hours.
(Pg((r fxx f(x,0))
single
We report the efficacy of our camouflaged poisoning attack across different models and threat models (s, bp.bb) be) in Figure 3; also see Appendix B.4 for detailed results and performance drops on the validation dataset after adding poison and camouflage set. Each model was trained to have validation accuracy between 81-87% (depending on the architecture), which changed minimally when the model was retrained with poison and camouflage samples. Poisoning was successful least 80% of the time in most of the experiments. Camouflaging 11, VGG-16, Resnet-18, and Resnet-34 but was not as successful for MobileNetV2 and Resnet-50. Furthermore, camouflaging succeeded at least 75% of times when b. 6p. but did not perform as well when we set bp > b. in the thread model (more poison images than camouflage images).
successful least 70% of the time for VGG-
1.0
Aotack Success
Attack Success
0.0
0.6
0.4
0.0
V9G-11
VGG-16
ResNet- 30MobileNet22
1-18
<-B by Nim1%
4-18
X-8
bu 0,-2%
Model Architecture
Threat Model
Figure 3: Efficacy of the proposed camouflaged poisoning attack on CIFAR-10 dataset. The left plot gives the success for the threat model 16, 16,6, 0.6%, b, 0.6% across different neural network architectures. The right plot gives the success for ResNet-18 architecture across different threat models. See Appendix B.4 for the experiment details.
8

---

Under review as a conference paper at ICLR 2023
3.2 EVALUATIONS ON IMAGENETTE AND IMAGEWOOF
We evaluate the efficacy of our attack vector on the challenging multiclass classification problem on the Imagenette and Imagewoof datasets (Howard, 2019). Imagenette is a subset of 10 classes (Tench, English springer, Cassette player, Chain saw, Building/church, French horn, Truck, Gas pump, Golf ball, Parachute) from the Imagenet dataset (Russakovsky et al., 2015). The Imagenette dataset consists of around 900 images of various sizes for each class. In total, we have 13394 images which divided into a training dataset of size 9469 and test dataset of size 3925. To perform training, all images are resized and centrally cropped down to 224 x 224 pixels.
Threat Model
Attack Success
Dataset
Model
bp
be
Poisoning
Camouflaging
Imagenette
VGG-16
16
6.3%
6.3%
25%
100%
Imagenette
Resnet- 18
16
6.3%
6.3%
40%
50%
Imagewoof
Resnet- 18
16
6.6%
6.6%
50%
75%
Table 2: Evaluation of camouflaged poisoning attack on Imagenette and Imagewoof datasets over 5 seeds (with restart per seed). Note that camouflaging succeeded in most of the experiments in which poisoning succeeded. Prior works (e.g.. Geiping et al. (2021)) set a large number of restarts R, and then choose the most effective attack among them. Due to computational constraints, we only one restart (i.c., R = 1) for each experiment. Given additional computational resources, could inflate the success rate both the poisoning and camouflaging.
Imagewoof (Howard, 2019) is another subset of Imagenet dataset consisting of 10 classes (Shih- Tzu, Rodesian Ridgeback, Beagle, English Foxhound, Border Terrier, Austrailian Terrier, Golden Retriever. Old English Sheep Dog. Samoyed, Dingo). Imagewoof consists of around 900 images of various sizes for each class, and in total 1 12954 images which are divided into training dataset of size 9025 and test dataset of size 3929. Similar to Imagenette, we resize all images and crop to the central 224 224 pixels before training.
We evaluate our camouflaged poisoning attack on two different neural network architectures- 16 and ResNet- 18, and different threat models (s, b,, b.) listed in Table 3.2. Each model is trained on a single GPU with cross-entropy loss, that is minimized using SGD algorithm with weight decay 5e- 4, momentum 0.9 and batch size 20. We start with learning rate of 0.01, and exponentially decay it with y = 0.9 after every epoch, for total of 50 epochs over the training dataset. The poisons and camouflages were generated using gradient matching by first defining the cosine-similarity loss using torch. autograd and then optimizing it using Adam optimizer with learning rate 0.1. In our experiments, camouflaging was successful for at least 50% of the time when poisoning was successful. However, because we modified about 13% of the training dataset when adding poisons/ / camouflages, we observe that the fluctuation in the model's 's validation accuracy can be up to 7% for both Imagenette and Imagewoof, as expected on making such large change in the training set.
Additional experiments. In Appendix B.6, we provide additional experiments on CIFAR-10 showing that our attack is robust to data augmentation, and successfully transfers when the victim model is different from the model on which poison and camouflage samples were generated.
Conclusion and discussion. We demonstrated new attack vector, camouflaged poisoning attacks, against machine learning pipelines where training points can be unlearned. This shows that as we introduce new functionality to machine learning systems, we must be aware of novel threats that emerge. We outline a few interesting directions for further research: It is important to understand how to defend against camouflaged attacks. As observed by Geiping et al. (2021), it is unlikely that differential privacy (Dwork et al., 2006) would be an effective defense, preventing attacks in the non-camouflaged setting incurs too significant a loss in accuracy. Another direction is to reduce the knowledge needed by the adversary, thereby creating stronger attacks, E.g., while our setting requires grxy-box knowledge, one could instead consider a black-box model to attack ML APIs. Fi- nally, it is interesting to determine what other types of threats can be camouflaged, e.g., indiscrimi- nate or backdoor poisoning attacks. Beyond exploring this new attack vector, it is also independently interesting to understand how can neutralize the effect of attack by adding points.
9

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
For the purposes of reproducibility, we include our code in the supplement. Descriptions of the var- ious files are given in Appendix B.2. Hyperparametens and experimental details for all experiments are provided in Section Finally, we report random seeds, target points and classes, poison and camouflage classes for all experiments in Appendix B.3.
REFERENCES
Hojjat Aghakhani. Dongyu Meng, Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye polytope: A scalable clean-label poisoning attack with improved transferability. In 202/ IEEE European Symposium on Security and Privacy (FuroS&P), pp. 159-178. IEEE, 2021.
Marco Barreno, Blaine Nelson, Anthony D Joseph, and J Doug Tygar. The security of machine learning. Machine Learning, 81(2):21-144, 2010.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector ma- chines. In Proceedings of the 29th International Conference Machine Learning, ICML '12, pp. 1467-1474. JMLR, Inc., 2012.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In proceedings of the 42nd IEEE Symposium on Security and Privacy, SP '21, Washington, DC, USA, 2021. IEEE Computer Society.
Jonathan Brophy and Daniel Lowd. Machine unlearning for random forests. In Proceedings of the 38th International Conference on Machine Learning, ICML, '21, pp. 1992-1104. JMLR, Inc., 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In Pro- ceedings of the 36th IEEE Symposium Security and Privacy, SP 15, pp. 463-480, Washington, DC, USA, 2015. IEEE Computer Society.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC Con- ference on Computer and Communications Security, pp. 896-911,2033 2021.
Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon. Werner Zellinger, Berhaard Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, and Fabio Roli. Wild patterns reloaded: A survey of machine learning security against training data poisoning. arXiv preprint 0:X1:22005.01992. 2022.
Ilias Diakonikolas, Gautam Kamath, Daniel M. Kane, Jerry Li, Jacob Steinhardt, and Alistair Stew- art. Sever: A robust meta-algorithm for stochastic optimization. In Proceedings of the 36th International Conference on Machine Learning, ICML 19, pp. 1596-1606. JMLR, Inc., 2019.
Min Du, Zhi Chen, Chang Liu, Rajvardhan Oak, and Dawn Song. Lifelong anomaly detection through unlearning. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 1283-1297, 2019.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Conference Theory of Cryptography, TCC '06, pp. 265, Berlin, Heidelberg, 2006. Springer.
Federal Trade Commission. California company settles ftc allegations it deceived consumers about use of facial recognition in photo storage app. January 2021.
Jonas Geiping, Liam Fowl, W Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. In Pro- ceedings of the 9th International Conference on Learning Representations, ICLR '21, 2021.
General Data Protection Regulation. Regulation (EU) 2016/679 the European parliament and of
the council of 27 April 2016, 2016.
10

---

Under review as a conference paper at ICLR 2023
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making AI forget you: Data deletion in machine learning. In Advances in Neural Information Processing Systems 32, NeurIPS '19, pp.
Curran Associates, Inc., 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selec- tive forgetting in deep networks. In Proceedings of the 2020 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR '20, pp. 9304-9312, Washington, DC, USA, 2020. IEEE Computer Society.
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the 2021 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR '21, pp. 792-801. IEEE Com- puter Society, 2021.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. arXiv preprint arXiv:2012. 2020.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in machine learning model supply chain. arXiv preprint arXiv: /tti::700..00 010..7788.03333 2017.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning. ICML '20, pp. 3832-3842. JMLR, Inc., 2020.
Varun Gupta, Christopher Jung. Seth Neel, Aaron Roth, Saeed Sharíf-Maavajerdii. and Chris Waites. Adaptive machine unlearning. In Advances in Neural Information Processing Systems 34, NeurlPS 22,, pp. 16319-16330. Curran Associates, Inc., 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the 2016 IEEE Computer Society Conference on Computer Vision and Puttern Recognition, CVPR '16, pp. 770-778, Washington, DC, USA, 2016. IEEE Computer Society.
Jeremy Howard. Imagenette, 2019. URL http://///ggttubb.cc///s/aa/a//s
W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Prac- tical general-purpose clean-label data poisoning. In Advances in Neural Information Processing Systems 33, NeurIPS '20. pp. 12080-12091. Curran Associates, Inc., 2020.
Information Commissioner's Office. Guidance on the ai auditing framework, February 2020.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models: Algorithms and evaluation. In Proceedings of the 24th Interna- tional Conference on Artificial Intelligence and Statistics, AISTATS '21, pp. 2008-20182 JMLR, Inc., 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR '1, 2015.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning, 1 111(1):1-47, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Neil G Marchant, Benjamin IP Rubinstein, and Scott Alfeld. Hard forget: Poisoning attacks on certified machine unlearning. In Proceedings of the Thirh AAAI Conference on Artificial Intelligence, volume of AAAI '22, pp. 7691-7700, 2022.
Luis Muñoz-Gonzále Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back -raddgraiinnt optimization. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp. 27-3773 2017.
11

---

Under review as a conference paper at ICLR 2023
Seth Neel, Aaron Roth, and Saeed for machine unlearning. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, ALT 21. JMLR, Inc., 2021.
Gradient-based methods
Quoe Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Variational bayesian unlearning. Advances in Neural Information Processing Systems, 33: 33:16025-16036 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32, NeurlPS '19, pp. 8036-8037. Curran Associates, Inc., 2019.
Andrea Paudice, Luis Muñoz-Gonzzlez, and Emil C Lupu. Label sanitization against label flipping poisoning attacks. In Joint European conference machine learning and knowledge discovery in databases, pp. 5-15. Springer, 2018.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duch- esnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85): 2825 2830, 2011.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision. 1 (5(3):211-252) 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the 2018 IEEE Computer Society Conference on Computer Vision and Patlrnn Recognition, CVPR '18, pp. 4510-4520, Washington, DC, USA, 2018. IEEE Computer Society.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Advances in Neural Information Processing Systems 34, NeurlPS '21, pp. 18075-18086. Curran Associates, Inc., 2021.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks neural networks. Advances in neural information processing systems, 31, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale visual recogmition. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 15, 2015.
David Marco Sommer, Liwei Song, Sameer Wagh, and Prateek Mittal. Towards probabilistic verifi- cation of machine unlearning. arXiv preprint wxXi:2203.00427,, 2020.
Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. In Advances in Neural Information Processing Systems 30, NeurIPS '17, pp. 1520-3532. Curran Associates, Inc., 2017.
Octavian Suciu, Radu Marginean, Yigitcan Kaya, Hal Daume III, and Tudor Dumitras. When does machine learning (FAIL)? generalized transferability for evasion and poisoning attacks. 27th USENIX Security Symposium (USENIX Security 18), pp. 1299-1316, 2018.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint (0ti::/911.0000 2019.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems 31, NeurIPS '18, pp. 1011-0021. Curran Associates, Inc., 2018
12

---

Under review as a conference paper at ICLR 2023
Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pp. 4126-4142. PMLR, 2021.
Han Xiao, Huang Xiao, and Claudia Bekert. Adversarial label flips attack on support vector ma- chines. In ECAI 2012, pp. 870-875, IOS Press, 2012.
Huang Xiao, Battista Biggio, Blaine Nelson, Han Xiao, Claudia Eckert, and Fabio Roli. Support vector machines under adversarial label contamination. Neurocomputing, 160:33662, 2015.
Santiago Zanella-Béguee Béguelli, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models. In Proceedings of the 2020 ACM SIGSAC Conference Computer and Communications Security, pp. 363--755,20 2020.
13

---

Under review as a conference paper at ICLR 2023
A POISON GENERATION ALGORITHM
Algorithm Gradient Matching to generate poisons (Geiping et al., 2021)
Require: Clean network f(= trained on uncorrupted base images Sal, target (Starget Starget) ) and adversarial label Yadversarial: Poison budget P, perturbation bound s, number of restarts R, optimization steps M
1: Collect dataset Spo = (z', .y' 9
of P many images whose true label is Yadversarinl-
2: for r 1.... R restarts do
3: Randomly initialize perturbations A s.t. | Se.
4: for k = .,, M optimization steps do
Compute the loss ( Octean) as in (4) using the base poison images in Spo. 6: Update A using Adam update to minimize and project onto the constraint r.
5:
7: end for
8: Amongst the restarts, choose the A, with the smallest value of Φ( (AA,, feleaa).
9: end for
10: Return the poisoned set Spo = fx² A 1
P
1==1
Algorithm 2 Gradient Matching generate camouflages
Require: Network f( :8p)) trained on Se + Spo the target (Itarget: Vtarget); Camouflage budget C, perturbation bound number of restarts R, optimization steps M
C 1: Collect dataset Sea = (x) y² f=1 of C many images whose true label is Vtarget-
2: for I,... restarts do
3: Randomly initialize perturbations A s.t. |4| Se.
4: for k I,..., optimization steps do
5: Compute the loss (A, 0cp) as in (4) using the base camoufaage images in Sea.
6: Update A using an Adam update to minimize vb and project onto the constraint set
7: end for
8: Amongst the restarts, choose the A. with the smallest value of (AA,; Oep).
9:
end for
Return the poisoned Sea fxr A,,10 1
B EXPERIMENT DETAILS
B. HARDWARE
All our experiments were executed on Google Colab with a Google Colab Pro+ subscription.
B.2 CODE
We provide code for experiments ready-to-deploy Colab notebooks (as well as Python files). In the supplementary material, the code can be found in:
1. www.llaaeyyciaallloooooeeeeeiiio www.oodde..mmssssiio.. ipynk: Experiments for Binary-CIFAR-
10 dataset with linear SVM.
2. Neuzal_network.cifaau wtt.araroooodduudoeaoeeeeeemeeee
ipynh: Experiments for CIFAR-
10 dataset with various neural network models.
3. periments for Imagenette/ Imagewoof dataset with various neural network models.
Imagewoff.codee wtt.:cooee..bmssssiio... iitiisssiiiinnnns...... ipynb:
Ex-
B.3 EXPERIMENTAL SETUP
For the case of replication, we report the corresponding poison class, target class, camouflage class and Target ID for various seeds in different experiments.
14

---

Under review as a conference paper at ICLR 2023
Random Seed
Target Class
Poison Class
Camouflage Class
Target ID
2000000000
Deer
Bird
Deer
9621
2000000001
Cat
Horse
Cat
1209
2000000011
Frog
Bird
Frog
6503
20000001
Bird
Cat
Bird
124
2000001 111
Plane
Deer
Plane
7649
200001 1111
Cat
Dog
Cat
4423
20001 IIIII
Truck
Car
Truck
8117
2001
Bird
Truck
Bird
3686
201
Cat
Bird
Cat
642
21
Frog
Ship
Frog
97
Table 3: Target, poison and camouflage class corresponding to different initial random seeds used for CIFAR-10 experiments. The reported Target ID) is relative to the CIFAR-10 10 validation dataset.
Random Seed
Target Class
Poison Class
Camouflage Class
Target ID
2000000000
Building
Cassette player
Building
1559
2000000001
Chain saw
Gas pump
Chain saw
1266
2000000011
Truck
Cassette player
Truck
2460
200000011
Cassette player
Chain saw
Cassette player
792
2000001 111
Tench
Building
Tench
2500
2000011111
Chain
French horn
Chain
1162
20001 11111
Parachute
English springer
Parachute
3826
Cassette player
Parachute
Cassette player
1121
201
Chain saw
Cassette player
Chain saw
1198
21
Truck
Golf ball
Truck
2343
Table 4: Target class, poison class and camouflage class corresponding to different random seeds used for Imagenette experiments. The reported target ID is relative to the Imagenette validation set.
Random Seed
Target Class
Poison Class
Camouflage Class
Target ID
2000000000
Border Terrier
Beagle
Border Terrier
1493
2000000001
English Foxhound
Old English Sheep Dog
English Foxhound
1362
2000000011
Golden Retriever
Beagle
Golden Retriever
2399
2000000111 11
Beagle
English Foxhound
Beagle
827
2000001 111
Shih-Tzu
Border Terrier
Shih-Tzu
250
200001
English Foxhound
Austrailian Terrier
English Foxhound
1405
20001
Dingo
Rodesian Ridgeback
Dingo
3810
2001
Beagle
Dingo
Beagle
1204
201
English Foxhound
Beagle
English Foxhound
1294
21111111111111111111111111111111
Golden Retriever
Samoeyed
Golden Retriever
2282
Table 5: Target class, poison class and camouflage class corresponding to different random seeds used for Imagewoof experiments. The reported target ID is relative to the Imagewoof validation set.
B.4 ADDITIONAL DETAILS ON CIFAR- EXPERIMENTS
We elaborate on the results reported in Figure 3. Table 6, we report the efficacy of the proposed camouflaged poisoning attack on different neural network architectures where the threat model is given by & = 16, bp 0.6%, b. 0.6%. The reported results are an average over 5 seeds from 0000000000-000000 111. In the first column under attack success, report the number of times poisoning was successful amongst the run trials, and in the second column, we report the number of successful.
times camouflaging successful for the trials for which poisoning
In Table 7, we report the success of the proposed attack when we change the threat model, but fix the network architecture be R Each experiment repeated times 5 times with 8 restarts
15

---

Under review as a conference paper at ICLR 2023
each time, and the mean success rate is reported. These experiments were conducted with 5 seeds from 2000011 100001111121111 11100111111111111111111
Attack success
Validation Accuracy
Network Architecture
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
VGG-11
100%
80%
85.01
85.03 (# 0.37)
85.10 (+ 0.29)
VGG-16
80%
75%
87.68
87.42 (# 0.17)
87.45 (# 0.26)
ResNet-18
80%
75%
82.13
81.88 (+ 0.15)
81.80 (+ 0.12)
ResNet-34
80%
50%
82.45
82.61 (+ 0.30)
83.12 (+ 0.93)
ResNet-50
80%
25%
81.02
81.76 (# 0.13)
84.62 (+ 0.71)
MobileNetV2
60%
33%
82.79
83.26 (+ 0.25)
85.47 (4: 0.27)
Table 6: Evaluating proposed camouflaged poisoning attack on various model architectures the CIFAR-10 dataset with the threat model & 16, by 0.6%, b 0.6%.
Threat model
Attack success
Validation Accuracy
bp
b
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
16
1%
100%
80%
82.13
81.98 (+ 0.16)
82.12 (+ 0.21)
8
1%
1%
80%
75%
82.13
82.21 (# 0.21)
82.09 (# 0.23)
16
2%
1%
100%
20%
82.13
82.31 (a 0.26)
82.19 (# 0.24)
8
2%
1%
100%
40%
82.13
82.43 (# 0.30)
82.34 (+ 0.27)
Table 7: Evaluating our proposed camouflaged poisoning attack on various threat models with
CIFAR- 10 dataset trained on ResNet-18.
16

---

Under review as a conference paper at ICLR 2023
B.5 VISUALIZATIONS
Target
Poisons
Camouflages
Figure 4: Visualization of poisons and camouflages Binary-CIFAR-II 10 dataset (animal vs ma- chine classification). The top row shows the original images and the bottom row shows the corre- sponding poisoned camouflaged images (with the added A). The shown images generated for a camouflaged poisoning attack on SVM, with Seed 555555, e 16, bp 0.2, b. 0.4 and the target ID 6646.
Target
Poisons
Camouflages
Figure 5: Visualization of poisons and camouflages on CIFAR-I0 dataset (multiclass classification task). The top row shows the original images and the bottom row shows the corresponding poisoned / camouflaged images (with the added A). The shown images were generated for a camouflaged poisoning attack ResNet-18, with Seed 2000000000, 8, bp 0.2, b 0.4, poison class bird, target class deer, and the target ID) 9621.
17

---

Under review as a conference paper at ICLR 2023
STILL
STILL
Target
Poisons
Camouflages
Figure 6: Visualization of poisons and camouflages on Imagenette dataset. The first and the third columns shows the original images, and the second and the fourth columns shows the corrupted images (with added A). The shown images were generated for a camouflaged poisoning attack on ResNet- 18, with Seed 2000011111 and 8. The target and camouflage class is chain saw, and the poison class is French horn.
Poisons
Camouflages
Figure 7: Some representative poison and camouflage images for attack on Imagewoof dataset. In each pair, the left figure is the original picture from the training dataset and the right figure has been adversarially manipulated by adding A. The shown images were generated for camouflaged poisoning attack on Resnet-18, with Seed 10000005, bp b. 6.6% and & 16. The target and camouflage class is English Springer, and the poison class is Building (Church).
18

---

Under review as a conference paper at ICLR 2023
Target
P
Poisons
Camouflages
Figure 8: Visualization of poisons and camouflages Imagewoof dataset. The first and the third columns shows the original images, and the second and the fourth columns shows the corrupted images (with added A). The shown images were generated for a camouflaged poisoning attack ResNet-18 18, with Seed = 2111111110, hp bh = 4.2%, & = 16. The target and camouflage class is Austrailian Terrier, and the poison class is Golden Retriever.
19

---

Under review as a conference paper at ICLR 2023
B.6 ADDITIONAL EXPERIMENTS
B.6.1 ABLATION STUDY DIFFERENT VALUES by AND b.
In this section, we explore the effect of changing the relative sizes of the poison samples and cam- ouflage samples. The experiments are performed ResNet-18 for CIFAR-10 dataset with by 1% and b. (0.5, 1, 1.55% % respectively, and vice-versa. We report the results in Table 8.
One may also wonder what is the price that attacker has pay in terms of the success rate of poisoning (only) attack if it chooses to devote part of the budget for camouflages. In Table 9, we report the results for comparing the success rate % poisons and 1 camouflages, success rate with 2% poisons (and no camouflages.)
Problem parameters
Attack success
Validation Accuracy
bp
be
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
1%
1.5%
83%
80%
0.822
0.8191
0.8264
1%
0.5%
83%
40%
0.845
0.8395
0.8432
1%
1%
83%
80%
0.822
0.8274
0.8244
0.5%
1%
33%
50%
0.822
0.8261
0.8194
1.5%
1%
83%
20%
0.822
0.8156
0.8118
Table 8: Effect of different sizes of poison and camouflage datasets on the success of the proposed camouflaged poisoning attack on CIFAR-10 dataset trained on ResNet-18, 18, with & 16. The reported success rates are averaged over six different trials with seeds 200000111-21111111 experiment, we do 4 restarts for every poison and camouflage generation.
For each
Problem parameters
Attack success
Validation Accuracy
by
be
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
1%
1%
83%
80%
0.822
0.8274
0.8244
2%
83%
N/A
0.822
0.817
N/A
Table 9: Comparison of success rate when the allocated budget of 2% is split as: (a) 1% poisons and 1% camouflage, and (b) 2% poison samples and 0% camouflages. The experiment is performed on CIFAR-10 dataset trained on ResNet- 18 with 16. The reported success rates averaged six different trials with seeds 200000111 211111111. For each experiment, we do 4 restarts per poison camouflage generation.
B.6.2 ROBUSTNESS OUR ATTACK TO RANDOM DELETIONS
We next explore the effect of random removal the generated poison and camouflage samples on the success of our attack. In data-scraping scenario. victim may scrape all the data points modified by an attacker. In Table 10. we report the result where different amounts of the generated poison and camouflage samples are deleted uniformly at random.
20

---

Under review as a conference paper at ICLR 2023
Amount removed
Attack success
Validation Accuracy
Poisons
Camouflages
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
5%
5%
80%
75%
0.822
0.823
0.819
10%
10%
60%
33%
0.822
0.817
0.825
20%
20%
80%
100%
0.822
0.826
0.820
30%
30%
100%
80%
0.822
0.8215
0.818
40%
40%
20%
100%
0.822
0.825
0.817
Table 10: Effect of random removal of the generated poison and camouflage samples the success of the proposed camouflaged poisoning attack on CIFAR-IO dataset trained on ResNet- 18, with 16. The reported success rates are averaged over 5 different trials with seeds 200001111 211111111. For each experiment, we do 4 restarts for every poison and camouflage generation.
B.6.3 TRANSFER EXPERIMENTS
In this section, we show that the poison and camouflage samples generated by the proposed approach transfer across models. Thus, attacker successfully execute the camouflaged poisoning attack, even if the victim trains a different model than the one on which the poison and camouflage samples generated. We show the transfer success in Figure The brewing network denotes the network architecture on which poison and camouflage samples were generated (we adopt the same notation as Geiping al. (2021)). The victim network denotes the model architecture used by the victim for training on the manipulated dataset.
We ran total of experiments per (brewing model, victim model) pair using the seeds 2000000000- 2000000011. 1. Each reported number denotes the fraction of times when both poisoning and camou- flaging were successful the transfer experiment, and thus the attack could take place.
Resterr-
Brewing Network
Restet--
0.3
0.5
0.5
VGG-16
0.5
ResNet-18
ResNet-3 34
VGG-16 16
Victim Network
Figure 9: Transfer experiments CIFAR-10 dataset.
B.6.4 ROBUSTNESS DATA AUGMENTATION
Data augmentation is commonly used to avoid overfitting in deep neural networks. order to be applicable in the real life, our poisoning and camouflaging attacks must be successful even when the model is trained with data augmentation. In order to validate this, we evaluate our approach on CIFAR- 10 dataset trained with data augmentation ResNet-18 18 in the threat model 16, b, by 1%: the results are Table 11. The considered data augmentations are:
1. No Augmentation: Exact images from the training dataset used.
21

---

Under review as a conference paper at ICLR 2023
2. Augmentation Set I: 50% chance that the image will be horizontally flipped, but rota- tions.
Augmentation Set 2: 50% chance that the image will be horizontally flipped, and random rotations in Uniform(-10, 10) degrees.
The reported results in Table 11 are an average over random seeds from "kkkkkk" where s k s 5. As expected, the validation accuracy for the model trained on clean dataset increased from 82% per- cent when trained without augmentation, to 86% for augmentation Set and 88% for augmentation set 2. The addition of data augmentation during training and re-training stages make harder for poisoning to succeed and at the same time makes it easier for camouflaging to succeed.
Attack success
Validation Accuracy
Data Augmentation
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
No Augmentation
100%
20%
82%
82%
82%
Augmentation Set
86%
33%
86%
85%
86%
Augmentation Set
60%
100%
88%
86
86%
Table 11: Effect of data augmentation on our proposed camouflaged poisoning attack.
B.6.5 SIMILARITY OF THE FEATURE SPACE DISTANCE
natural approach defend against dataset manipulation attacks is to try to identify the modified images, and then remove them from the training dataset (i.e., data sanitization). For instance, one could cluster images based on their distance from their class mean image, or from the target image. This type of defense could potentially thwart watermarking poisoning attacks such as Poison Frogs (Shafahi et al., 2018). As we show in Figure 10, such a defense would not be effective against our proposed poison and camouflage generation procedures, as the data distribution for the poison set and the camouflage set is similar to that of the clean images from the respective classes
22

---

Under review as a conference paper at ICLR 2023
Distance to Poison Class Mean
L2 Distance to Target Image Features
0.30
total poison class poisons
0.025
total poison class
poisons
0.25
0.020
0.20
Count
Count
0.015
0015
0.010
0010
0.005
0.055
60
80
100
40
60
200
120
140
L2 Distance Camouflage Class Mean
Distance to Target Image Features
0.035
total camouflage class
total camouflage class
0.000
camouflages
0.020
camouflages
0.025
Count
0.200
Count
0.015
0.055
0.000
0.10
0.005
0.000
100
100
Figure 10: Feature space distance for our generated poison and camouflage set. The reported data was collected by successful camouflaged poisoning attack on Resnet- 18 model trailed on CIFAR- 10 with seed 2000000000, = 16 and bp = b. = 1%.
23


Reviews:
[{"review_id": "go6g-Ye9B5Q", "reviewer": "Reviewer_yC6u", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors present a clean-label machine-unlearning-triggered poisoning attack that targets a specific test point. Given access to the victim model’s architecture, the attacker crafts a poison set that makes the victim model misclassify the target sample and a camouflage set that neutralizes the poison set. Both sets are crafted by adding optimized noise (hard to detect by humans) to the images using the gradient-matching approach from previous work. The poisoning attack is then only triggered when the attacker requests to delete the camouflage set and the victim model unlearns by retraining. The authors show the method is effective on several models (SVMs and a few deep neural networks) and datasets (CIFAR-10, Imagenette, Imagewoof).\n", "summary_of_the_review": "see strength and weakness", "strength_and_weaknesses": "Strengths:\n\n- Novel idea\n- The attack is only triggered when needed\n- Can be difficult to detect\n\nWeaknesses:\n\n- Unclear attack scenario\n- Strong attack assumption\n- Limited scalability\n\nComments for the authors:\n\nThe authors present a novel poisoning attack that combines the key intuitions from two previous works, Marchant et al. (2022) for using machine unlearning requests as the attack trigger and Geiping et al. (2021) for using gradient-matching as a clean-label poisoning approach. As a result, this poisoning attack is difficult to detect and the adversary can choose when to trigger the attack.\n\nHowever, I do have the following concerns.\n\n- The authors did not present a clear or realistic scenario where such a poisoning attack is necessary. While it adds an additional function compared to the traditional poisoning attack, the paper will benefit from some examples of scenarios where such a function is needed.\n\n- The added functionality also comes at a cost. On page 4, the authors claim the benefit of this attack is that the adversary can reveal whenever an unlearning request is submitted, while traditionally “the adversary simply plants the attack and must wait for the victim to train the model.” However, the proposed attack also requires the adversary to “wait for the victim to train the model” and both the poison and camouflage sets (instead of just the poison set) need to be planted at the same time.\n\n- The proposed method also assumes strong adversary knowledge. The author mentioned in Section 2.1 that the attacker has access to the victim model’s architecture and the ability to query gradients (as mentioned could be achieved by having access to the training dataset). The additional experiment results of transferability in Appendix B.6.1 further indicate that the attacker needs white-box victim model access (not just architecture and training data) to achieve the high attack performance presented in the main results since even when the brewer (model used to craft poison and camouflage sets) and victim models use the same architecture the attack performance still suffers.\n\n- Even under the strong attack assumption, the results show the efficacy still varies quite drastically across the 6 architectures tested, where only 3 architectures show optimal attack success rates. This implies the method might not generalize well to other architectures.\n\n- Given the training time provided by the authors, crafting each poison and camouflage set takes on average double the time needed for training the model (so 4 times in total for a pair of poison and camouflage sets). This significantly limits the scalability of the method, especially if the victim model uses more complex architectures.\n\n- The method also only limits to one single target sample. I believe a common setting for targeted poisoning attacks aims to misclassify a class of images (instead of a single data point) to another label. Combining the time cost mentioned above this limits the method’s scalability. If the author can improve on the previous gradient-matching approach it would be a major contribution.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UXb84zDTf1_", "reviewer": "Reviewer_xqqh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel attack, called camouflaged data poisoning attacks by leveraging a machine unlearning mechanism. This attack needs generate a pair of poisoning and camouflage datasets, which has less impact to the model on normal testing data. After triggering the machine unlearning procedure on the camouflage datasets, a retrained model could make target images to be classified as the target labels. ", "summary_of_the_review": "In this paper, a novel attack is proposed, which leverages machine unlearning techniques", "strength_and_weaknesses": "Positives:\n+ a novel attack is proposed by using machine unlearning techniques. \n\nNegatives:\n- It has less technical contribution since it just leverages existing machine unlearning techniques\n- only two datasets are used to evaluate the proposed method. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q9tARk0_ln", "reviewer": "Reviewer_XcYH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission \"Hidden Poison: Machine unlearning enables camouflaged poisoning attacks \" describes a novel variant of targeted data poisoning attacks. The submission introduces the notion of \"camouflage\" data points that neutralize the effects of poisoned data samples. This essentially incorporates an \"off switch\" into targeted data poisoning attacks: camouflage data points can be created by similar techniques as existing data poisoning methods, but hide their effect. The actual poisoning attack is then triggered by an unlearning request of the camouflage data points, which removes the camouflage data points and activates the targeted data poisoning attack.", "summary_of_the_review": "Overall, I like the idea put forth in this submission, but would love for the authors to improve the execution and evaluation of the proposed vulnerability,  discussing implications in exemplary use cases, evaluating the robustness of the attack and demonstrating its effect on approximate unlearning systems.", "strength_and_weaknesses": "Overall I like the idea put forward in this work. That a data unlearning request could trigger malicious effects already present in the data is clever and an interesting concept that would be surprising to readers interested in the security aspects of data unlearning. I see the the main strength of this work in the novelty of this concept.\n\nHowever, I do think that the evaluation and presentation of the attack could be improved. I want to highlight some areas that can be improved:\n* While the attack is conceptually interesting, what are the implications of this attack? Could the authors provide examples? In what scenarios would an attacker want to include an off-switch into their attack? \n* Relatedly, what is the trade-off that the attacker is making here? In most experiments in this work, the investigated threat model includes a doubled poisoning budget alloted to the attacker (e..g 0.6% for poisoning and 0.6% for camouflaging). In contrast, the attacker could have used an entire budget of 1.2% for an improved attack, potentially with additional targets. How much is the attacker giving up in capability to include this off-switch? Also, it is necessary to split poison and camouflage budget 1-to-1? It would be great to see ablations for all intermediate stages, e.g. 1.5% poisoning, .5% camouflaging, and so on, as a larger extension of the result shown in Fig3 b).\n* Further, what about robustness of the attack? In a data scraping scenario, it is generally unlikely that a victim would scrape all data points modified by an attacker. How robust is this attack to an attacker scraping random subsets of both poisoning and camouflage points?\n* Finally, this submission chooses targets from one class, data poisons from another, and camouflage points from the target class. Is this necessary? What happens when both poisons and camouflages are chosen from the same class? Or if camouflages are from a third class?\n\n* Aside from these specific points, more broadly: What about other types of unlearning? The submission only covers exact unlearning through retraining, but a range of approximate unlearning techniques have been proposed in the literature (as discussed also in the related work section). I think this submission could be majorly improved by evaluating the success of the proposed attack also in approximate unlearning scenarios.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject