Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attacks
Authors: Jimmy Z. Di
Jack Douglas
Jayadev Acharya
Gautam Kamath
Ayush Sekhari
Venue: NeurIPS 2023 poster (2023)
Keywords: Machine unlearning
new attack vector
Camouflaging poisoning attacks

Abstract: We introduce camouflaged data poisoning attacks, a new attack vector that arises in the context of machine unlearning and other settings when model retraining may be induced. An adversary first adds a few carefully crafted points to the training dataset such that the impact on the model's predictions is minimal. The adversary subsequently triggers a request to remove a subset of the introduced points at which point the attack is unleashed and the model's predictions are negatively affected. In particular, we consider clean-label targeted attacks (in which the goal is to cause the model to misclassify a specific test point) on datasets including CIFAR-10, Imagenette, and Imagewoof. This attack is realized by constructing camouflage datapoints that mask the effect of a poisoned dataset. We demonstrate efficacy of our attack when unlearning is performed via retraining from scratch, the idealized setting of machine unlearning which other efficient methods attempt to emulate, as well as against the approximate unlearning approach of Graves et al. (2021).

Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attacks*
Jimmy Z. Di University of Waterloo j jimmy. di @uvaterloo.cccc ca
Jack Douglas University of Waterloo
Jayadev Acharya Cornell University
jack www.glassuwaterrl...oooo
acharya@cornell. edu
Gautam Kamath
Ayush Sekhari
University of Waterloo, Vector Institute gScsail.min .im.edu
Massachusetts Institute of Technology
sekhari
Abstract
We introduce camouflaged data poisoning allacks, a new attack vector that arises in the context of machine unlearning and other settings when model retraining may be induced. An adversary first adds a few carefully crafted points to the training dataset such that the impact on the model's predictions is minimal. The adversary subsequently triggers a request to remove a subset of the introduced points at negatively
which point the attack is unleashed and the model's predictions affected. In particular, we consider clean-label targeted attacks (in which the goal is cause the model misclassify specific test point) on datasets including CIFAR-10, Imagenette, and Imagewoof. This attack is realized by constructing camouflage datapoints that mask the effect of poisoned dataset. We demonstrate the efficacy of our attack when unlearning is performed via retraining from scratch, the idealized setting of machine unlearning which other efficient methods attempt to emulate, well as against the approximate unlearning approach of Graves et al. [2021].
Introduction
Machine Learning (ML) research traditionally assumes static pipeline: data is gathered, a model trained once and subsequently deployed. This paradigm has been challenged by practical deployments, which are more dynamic nature. After initial deployment more data may be collected, necessitating additional training. Or, as in the machine unlearning setting [Cao and Yang 2015 we may need to produce model as if certain points were never in the training set to begin with
While such dynamic settings clearly increase the applicability of ML models, they also make them more vulnerable. Specifically, they open models up to new methods of attack by malicious actors aiming to sabotage the model. In this work, we introduce a new type of data poisoning attack on models that unlearn training datapoints. We call these camouflaged data poisoning attacks.
The attack takes place in two phases FFigure Tp. In the first stage, before the model is trained, the attacker adds a set of carefully designed points to the training data, consisting of poison set and camouflage set. The model's behaviour should be similar whether it is trained on either the training data, or its augmentation with both the poison and camouflage sets. In the second phase, the attacker triggers an unlearning request to delete the camouflage set after the model is trained. That is, the
Authors JA, GK, AS listed in alphabetical order. Full paper: htttt::///aaatt org/ httggaab//212.... 10717
A naive solution is to remove said points from the training set and re-train the model from scratch.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

model must be updated to behave as though it were only trained on the training set plus the poison At this point, the attack is fully realized, and the model's performance suffers in some way.
While such an attack could harm the model by several metrics, this paper, we focus on targeted poisoning attacks that is, poisoning attacks where the goal is misclassify specific point in the test set. Our contributions are the following:
1. We introduce camonflaged data poisoning attacks, demonstrating a new attack vector in dynamic settings including machine unlearning (Figure T).
2. We realize these attacks in the targeted poisoning setting, giving an algorithm based on the wradient-macching approach of Geiping behavior comparable to as if the poison set were absent, we construct the camouflage set by generating new set of points that undoes the impact of the poison We thus identify new technical question of broader interest to the data poisoning community: Can one nullify a data poisoning attack by only adding points?
[202T] In order make the model
3. We demonstrate the efficacy of these attacks on a variety of models (SVMs and neural networks) and datasets (CIFAR-10 (Krizhevsky 2009]. Imagenette Imagewoof HHward 2019).
HHward 2019) and
Step Add Poisors and Camouflages
Step Train Model
Step Unlearning Request (Remove Camouflages)
Step Attack Triggers After Unlearning
Poisons
Clean Target Image I
Clean Tarpet Image
Clean Camouflages
Neural Network
Updated Network
Prediction: Dog
Prediction: Car Wrong Label X
Correct Label
Figure 1: An illustration successful camouflaged targeted data poisoning attack. Step the adversary adds poison and camouflage sets of points to the (clean) training data. In Step 2, the model trained on the augmented training dataset. should behave similarly to trained on only the clean data; in particular, should correctly classify the targeted point. In Step 3, the adversary triggers an unlearning request to delete the camouflage set from the trained model. Step 4, the resulting model misclassifies the targeted point.
1.1 Preliminaries
Machine Unlearning. significant amount of legislation concerning the "right be forgotten" has recently been introduced by governments around the world, including the European Union's 's General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA), and Canada's proposed Consumer Privacy Protection Act (CPPA). Such legislation requires organizations to delete information they have collected about a user upon request. A natural question is whether that further obligates the organizations to remove that information from downstream machine learning models trained on the data current guidances [Information Commissioner's Office 2020) and precedents [Federal Trade Commission 2021] indicate that this may be the case. This goal has sparked a recent line of work on machine unlearning [Cao and Yang 2015).
The simplest way remove user's data from trained model is to remove the data from the training set, and then retrain the model on the remainder (also called "retraining from scratch"). This the ideal way to perform data deletion, as it ensures that the model was never trained on the datapoint of

---

Polsons
Camoufiages
Figure 2: Some representative images from Imagewoof. In each pair, the left figure is from the training dataset, while the right image has been adversarially manipulated. The top and bottom rows images from the poison and camouflage set, respectively. In all cases, the manipulated images are clean label and nearly indistinguishable from the original image.
concern. The downside is that retraining may take a significant amount of time in modern machine learning settings. Hence, most work within machine unlearning has studied fast methods for data deletion, sometimes relaxing to approximately removing the datapoint. A related line of work has focused more on other implications of machine unlearning, particularly the consequences of an adaptive and dynamic data pipeline [Gupta et al. 2021, Marchant et al., 2022]. Our work fits into the latter line: we show that the potential to remove points from a trained model can expose new attack vector. Since retraining from scratch is the idcal result that other methods try to emulate, we focus primarily on unlearning by retraining from scratch, but the same phenomena should still occur when any effective machine unlearning algorithm is applied. For example, we also demonstrate efficacy against the approximate unlearning method of Graves et al. 2021
Data Poisoning. In a data poisoning attack, an adversary in some way modifies the training data provided to machine learning model, such that the model's behaviour test time is negatively impacted. Our focus is on targeted data poisoning attacks, where the attacker's goal is to cause the model to misclassify some specific datapoint the test set. Other common types of data poisoning attacks include indiscriminate (in which the goal is to increase the test error) and backdoor (where the goal is to misclassify test points which have been adversarially modified in some small way).
The adversary is typically limited in a couple ways. First, it is common to say that they can only add a small number of points to the training set. This mimics the setting where the training data is gathered from some large public crowdsourced dataset, and an adversary can contribute few judiciously selected points of their own. Other choices may include allowing them to modify or delete points from the training set, but these are less easily motivated. Additionally, the adversary is generally constrained to clean-label attacks: if the introduced points were inspected by human, they should not appear suspicious or incorrectly labeled. We comment that this criteria is subjective and thus not a precise notion, but nonetheless common in data poisoning literature, and we use the term as well.
2 Discussion of Other Related Work
The motivation for our work comes from Marchant et al. 2022], who propose a novel poisoning attack on unlearning systems. As mentioned before, the primary goal of many machine unlearning systems is to "unlcarn" " datapoints quickly, i.c., faster than retraiming from scratch. Marchant et al. 2022 craft poisoning schemes via careful noise addition, in order to trigger the unlearning algorithm to retrain from scratch on far more deletion requests than typically required. While both our work and theirs are focused on data poisoning attacks against machine unlearning systems, the adversaries have very different objectives. In our work, the adversary is trying to misclassify a target test point, whereas they try to increase the time required to unlearn point.
In targeted data poisoning, there are few different types of attacks. The simplest form of attack label flipping, in which the adversary is allowed to flip the labels of the examples [Barreno et al. 2010 Xiao et al. 2012 Paudice et al. 2018). Another type of attack is watermarking, which the
feature vectors are perturbed to obtain the desired poisoning effect (Suciu al. 2018 Shafahi et al.)
3

---

2018]. In both these cases, noticeable changes are made to the label and feature vector, respectively. which would be noticeable by a human labeler. In contrast, clean label attacks attempt to make unnoticeable changes to both the feature vector and the label, and are the gold standard for data poisoning attacks (Huang et al. 2020 Geiping et al. 2021]. Our focus on both clean-label poison and camouflage sets. While there are also works on indiscriminate [Biggio et al. [0012 Xiao et al.
2015
Muñoz-Gonzálne
et al.
2017, Steinhardt et al. 2017 Diakonikolas et al. 2019b Koh et al.
2022] and backdoor [Gu et al., 2017 Tran et al. 20118 Sun et al. 2019] poisoning attacks, these
are beyond the scope of our work, see Goldblum et al. 2020 Cina et al. [022 for additional
background on data poisoning attacks.
Cao and Yang 2015] initiated the study of machine unlearning through exact unlearning, wherein the new model obtained after deleting an example is statistically identical to the model obtained by training on a dataset without the example. probabilistic notion of unlearning was defined by Ginart et al. (2019), which in turn is inspired from notions in differential privacy [Dwork et al., 2006]_ Several works studied algorithms for empirical risk minimization (i.e., training loss) [Guo et al.
2020, Izzo et al. [021 Neel al.) 2022 1 Ullah et al., 2021 Thudi et al., 2022] Graves et al., 2021 I
Chourasia et al. 2022], while later works study the effect of machine unlearning on the generalization loss [Gupta et al. 202 I. Sekhari et al. 2021]. In particular, these works realize that unlearning data points quickly can lead to a drop in test loss, which is the theme of our current work. Several works have considered implementations of machine unlearning in several contexts starting with the work of Bourtoule et [202T]. These include unlearning in deep neural networks [Golatkar et al., 2020 2021]N et al. [2020], random forests [Brophy and Lowd 2021), large scale language models Ganella-Béguell in et al., 2020]. the tension between unlearning and privacy (Chen et al.,
2021 Carlini al. 2022]] anomaly detection (Du al. 2019), insufficiency of preventing verbatim
memorization for ensuring privacy [[ppolito et al.] 2022] and even auditing of machine unlearning
systems [Sommer al., 2020).
After the first version of our paper was uploaded on arXiv, Yang et al. 2022 discovered defenses against the data poisoning procedure of ofeipigg et al. 202 One may further use their techniques to defend against our camouflaged poisoning attack as well, however, we note that this does not undermine the contributions of this paper. Our main contribution is to introduce this new kind of attack, and bring to attention that machine unlearning enables this attack; the specific choice of which procedure is used for poison generation or camouflage generation is irrelevant the existence of such an attack.
3 Setup and Algorithms
3.1 Threat Model and Approach
The camouflaged poisoning attack takes place through interaction between an attacker and victim, and is triggered by an unlearning request. We assume that the attacker has access to the victim's 's model architecturee the ability to query gradients on trained model (which could be achieved, e.g.. by having access to the training dataset), and a target sample that it wants to attack. The attacker first sets the stage for the attack by introducing poison samples and camouflage samples to the training dataset, which are designed so as to have minimal impact when model is trained with this modified dataset. At a later time, the attacker triggers the attack by submitting an unlearning request to remove the camouflage samples. The victim first trains machine learning model (e.g., deep neural network) on the modified training dataset, and then executes the unlearning request by retraining the model from scratch on the left over dataset. The goal of the attacker is to change the prediction of the model on a particular target sample (ætar: Mtar) previously unseen by the model during training from year to desired label Yadv. while still ensuring good performance over other validation samples. Formally, the interaction between the attacker and the victim is as follows (see Figure
1. The attacker introduces a small number of poisons samples Spa and camouflage samples S.m. to clean training dataset Sa- Define Sepe = Sa + Spo + Scn-
2. Victim trains an ML model (c.g., neural network) on Seper and returns the model Oppe-
we examine the transferability of our proposed attack to unknown victim model, thus relaxing the requirement of knowing the victim's model architecture a priori.
In Appendix B.6.3

---

3. The attacker submits a request to unlearn the camouflage samples Sea.
4. The victim performs the request, and computes a new model Oep by retraining from scratch on the left over data samples Sep = Sat + Spa-
Note that the attack is only realized in Step 4 when the victim executes the unlearning request and retrains the model from scratch on the left over training samples. In fact. in Steps 1-3, the victim's s model should behave similarly to if it were trained on the clean samples Set only. In particular, the model O upc will predict Star on Star. whereas the updated model Dep will predict Yadv on Star- Both models should have comparable validation accuracy. Such an attack implemented by designing a camouflage set that cancels the effects of the poison set while training, but retraining without the camouflage set (to unlearn them) exposes the poison set, thus negatively affecting the model.
Before we delve into technical details on how the poisons and camouflages are generated, we will provide illustrative scenario where such attack can take place. Suppose an image-based social media platform uses ML-based content moderation to filter out inappropriate images, e.g., adult content, violent images, etc., by classifying the posts as "safe" and "unsafe." " An attacker can plant a camouflaged poisoning attack in the system in order to target famous personality, like a politician, a movie star, etc. The goal of the attacker is to make the model misclassify potentially sensitive target image of that person (e.g., a politically inappropriate image) as "safe." However, the attacker does not want to unleash this misclassification immediately, but to instead time it to align with macro events like elections, the release of movies, etc. Thus, at later time when the attacker wishes, the misclassification can be triggered making the model classify this target image as safe and letting it circulate on the platform, thus hurting the reputation of that person at an unfortunate time (e.g.. before an election or before when their movie released). The attack is triggered by submitting an unlearning request.
We highlight that camouflaged attacks may be more dangerous than traditional data poisoning attacks, since camouflaged attacks can be triggered by the adversary. That is, the adversary can reveal the attack whenever it wishes by submitting an unlearning request. On the other hand, in the traditional poisoning setting, the attack is unleashed as soon as the model is trained and deployed, the timing of which the attacker has little control over.
In order to be undetectable, and represent the realistic scenario in which the adversary has limited influence on the model's training data, the attacker is only allowed to introduce a set of points that is much smaller than the size of the clean training dataset (i.e., |Spol << |Sail and |Scal << [Sal). Throughout the paper and experiments, we denote the relative size of the poison set and camouflage
[Smal x 100, respectively. Additionally, the attacker is only allowed to generate poison and camouflage samples by altering the base images by less than distance in the fo norm (in our experiments e I 16, where the images are represented as an array of pixels in 0 to 255). Thus, the attacker executes so-called clean-label attack, where the corrupted images would be visually indistinguishable from original base images and thus would be given the same label as before by human data validator. We parameterize threat model by the tuple (c, bp).
set by bp := |Spal x 100 and b. := [Sal
[S.1
3.2 Poison and Camouflage Samples
The attacker implements the attack by first generating poison samples, and then generating camouflage samples to cancel their effects, as shown the following.
Poison samples. Poison samples are designed so that a network trained on Sep Set + Spo predicts the label Vadv (instead of ytac) on target image Star- While there are numerous data poisoning attacks in the literature, we adopt the state-of-the-as procedure of Geiping et al. [02T]] for generating poisons due to its high success rate, efficiency of implementation, and applicability across various models. However, our framework is flexible: in principle, other attacks for the same setting could
'The unlearning process aims to forget the subset of samples Sta from the model Oepe (trained on Sepe Sa Spo Sc..) The goal of the unlearning process is to output model which is indistinguishable from the model trained from scratch on the leftover data samples (i.e. on Sat Spo). In our work, we simulate this indistinguishability by exactly retraining fresh model from scratch on Sa + Spo. Since the goal of unlearning is to ensure this indistinguishabbiity. we conjecture that our attack will also work against other approximate unlearning methods in the literature, and provide experimental evidence in support of this in Appendix B. p.oytpoo
5

---

serve as drop-in replacement, e.g., the methods of Aghakhani et al. [2021 or Huang et al. 20200 or any method introduced in the future.
Suppose that Sep consist of N1 samples (a', Jig, out of which the first P samples with index to P belong to the poison set Spo The poison samples are generated by adding small perturbations A' to the base image x' so as to minimize the loss on the target with respect to the adversarial label, which can be formalized as the following bilevel optimization problem
min ((f(((((m 0(4)), Madv)
where
Bel
((A) E arg min 6
Σ
((f(2' 0),)),²)).
(1)
N
and we define the constraint set :'
[A: S and A 0 for all i> P}. The objective function
in D is called the adversarial loss [Geiping al. 2021] In all our experiments, we generate the poison points using the gradient matching technique of Geiping et al. 2022], which we detail in Appendix A for completeness.
Camouflage samples. Camouflage samples are designed to cancel the effect of the poisons, such that a model trained on Sape Set + Spo S.a behaves identical to the model trained on Sa1, and makes the correct prediction on Ztar- We formulate this task via a bilevel optimization problem similar to (1) Let Sape consist of N2 samples (x², y jsss,, out of which the last C' samples with index j == N2- C' to N2 belong to the camouflage set Sea. The camouflage points are generated by adding small perturbations A to the base image x' so to minimize the loss the target with respect to the adversarial label. In particular, we find the appropriate by solving:
min (( (Fraar,((A)) 0(A)), Aer
where
1 Σ {(f(x' A),0),
((A) min
N₂
JS.N2
and we define the constraint set I (A: 1o S and A 0 for all JSNz C}.
3.3 Efficient Algorithms for Generating Camouflages
Given the poison images, camouflage images are designed in order to neutralize the effect of the poisons. Here, we give intuition into what mean by canceling the effect of poisons, and provide two procedures for generating camouflages efficiently: label flipping, and gradient matching.
3.3.1 Camouflages via Label Flipping
Suppose that the underlying task is binary classification problem with the labels y E (-1, 1). and that the model is trained using linear loss {(S (z, (), = =yf (x. 0). Then, simply flipping the labels allows one to generate a camouflage set for any given poison set Spo. In particular, Sea is constructed as: for every (x', yy) Spo. simply add (x",-v') to Sea (i.e., b, b.). It easy to that for such camouflage samples, we have for any 0,
Σ ((f(x,0).)) (Epe))
P Σ vf(x,0) 0)
(a.))ESa
Σ {(f(x,)).y).
(a.y)E8a
"We provide preliminary experiments in Appendix of camouflage--poisonnee poisoning attacks developed using the Bullseye Polytope technique from Aghakhani al. [202T], and show that the attack continues succeed when we use altemnaee poison and camouflage generation techniques.
'This ordering is for notational convenience; naturally, the datapoints shuffled preclude the victim simply removing prefix of the training data.
'While (1) focuses on misclassifying single target point, it is straightforward to extend this to multiple targets by changing the objective sum losses the target points.

---

We can also similarly show that the gradients (as well as higher order derivatives) are equal, i.e., Vo Es.g ((f(z, 9) Vo (f) .) y) for all 0. Thus, training model on Sepe equivalent training it on Sat In essence, the camouflages have perfectly canceled out the effect of the poisons. We validate the efficacy of this approach via experiments on linear SVM trained with hinge loss (which resembles linear loss when the domain is bounded). See [Section 4.1.1 for details.
While label flipping is a simple and effective procedure to generate camouflages, it is fairly restrictive. Firstly, label flipping only works for binary classification problems trained with linear loss. Secondly, the attack is not clean label as the camouflage images are generated as (x',-y") by giving them the opposite label to the ground truth, which be easily caught by validator. Lastly, the attack is vulnerable to simple data purification techniques by the victim, e.g.. the victim can protect themselves by preprocessing the data to remove all the images that have both the labels (y +1 and y 11) in the training dataset. In the next section, we provide a different procedure to generate clean-label camouflages that works for general losses and multi-class classification problems.
3.3.2 Camouflages via Gradient Matching
Here, we discuss our main procedure to generate camouflages, which builds on the gradient matching idea Geiping et al. [202T] . Note that, our objective in (2 is to find A such that when the model is
trained with the camouflages, it minimizes the original-targer loss in (2) (with respect to the original label Ytar) ) thus making the victim model predict the correct label on this target sample. Since. (2) is
computationally intractable, one may instead try to implicitly minimize the original-target loss by finding a A such that for any model parameter 0,
V Vo Fo(e(f( Vtar)) ~ j EL Vol(f(E' Ai, 0))
The above equation suggests that minimizing (e.g., using Adam / SGD) on camouflage samples will also minimize the original-targed loss, and thus automatically ensure that the model predicts the correct label on the target. Unfortunately, finding perturbations that satisfy the above is also intractable as the approximate equality is required to hold for all 0. Building on Geiping et al. [2021]. relax this condition to be satisfied only for fixed model -up- model trained the dataset Sep = Sat + Spos and obtain the perturbations which minimize the cosine- sisiin-simillrity loss given by
(()
(3)
(Vol([ 0). EnS VolU[(e; +A).)..)))) [V 6), Ytar) EIl Vol(f(z; A ,), 8))
Implementation details. We minimize (3) using the Adam optimizer Kingma and Ba 2015 with a fixed step size of 0.1. In order to increase the robustness of camouflage generation, we do R restarts (where R 10). In each restart, we first initialize A randomly such that | S and perform M steps of Adam optimization to minimize vi( A, Oep). Each optimization step only requires a single differentiation of the objective 1 with respect to A, and can be implemented efficiently. After each step, we project back the updated A into the constraint set I so as to maintain the property that | I 8. After doing R restarts, we choose the best round by finding A, with the minimum 2 A.. 0ap).
4 Experimental Evaluation
We generate poison samples by running/Algoritmm 2 (given in the appendix). and camouflage samples by running Algorithm 1 with R = 1 and M 250 Each experiment is repeated K times by setting different seed each time, which fixes the target image, poison class, camouflage class, base poison images and base camouflage images. Due to limited computation resources, we typically set Ke E (3.5 5, 8, 10) depending on the dataset and report the mean and standard deviation across different trials. We say that poisoning was successful if the model trained on Sep = Se1 Spo predicts the label Yadv on the target image. Furthermore, we say that camouflaging was successful if the model trained on Sepe = Spo + Sa1 Sca predicts back the correct label ytar on the target image, provided that poisoning was successful. A camouflaged poisoning attack is successful if both poisoning and camouflaging were successful.
We diverge slightly from the threat model described above. in that the adversary modifies rather than introduces points. We do this for convenience and do not anticipate the results would qualitatively change.
7

---

Algorithm 1 Gradient Matching to generate camouflages
Require: Network f(- e0p) trained on Se Spo the target (Star: Star), Camouflage budget C, perturbation bound number of restarts R, optimization steps M
1: Collect a dataset Sea = {x², j-1 of many images whose true label is ytar-
2: for for== I.... R restarts do
3: Randomly initialize perturbations s.t. || S.
4: for k 1 ...... M optimization steps do
5: Compute the loss y ( ppp) ) as (5) using the base camouflage images in Sea.
6: Update A using Adam update to minimize e, and project onto the constraint set I'.
7: end for
8: Amongst the R restarts, choose the A, with the smallest value of e/ (A.,, 0cp).
9: end for
10: Return the poisoned set Sea = (xt A4, v 1
4.1 Evaluations on CIFAR-10
CIFAR-10 (Krizhevsky 2009 is multiclass classification problem with 10 classes, with 6,000 color images in each class of size 32 x 32. We follow the standard split into 50,000 training images and 10,000 test images.
4.1.1 Support Vector Machines
Attack type (p, bo)
Attack success
Validation Accuracy
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
LF (8, 0.2%, 0.2%)
70%
71.5%
81.63
81.73 (# 0.14)
81.74 (= 0.20)
LF (16, 0.2%, 0.2%)
100%
40%
81,63
81.64 (+0.03)
81.6 (=0.02)
GM (8, 0.2%, 0.4%)
70%
100%
81.63
81.65 (+0.01)
81.62 (+0.02)
GM (16, 0.2%, 0.4%)
100%
70%
81.63
81.65 (+0.03)
81.63 (= 0.02)
Table 1:
Camouflaged poisoning attack on linear SVM on Binary-CIFAR-10 dataset. The first column lists the threat model (z.bp. and the camouflaging type "LF" for label flipping and "GM" for gradient matching. The implementation details are given Appendix B.3
In order to perform evaluations on SVM, we first convert the CIFAR-10 dataset into a binary classification dataset (which we term Binary-CIFAR-10) by merging the 10 classes into two groups: animal (y = +1) and machine (y- = -1)). Images (in both training and test datasets) that were originally labeled (bird, cat, deer, frog. horse) are relabeled animal, and the remaining images, with original labels (airplane, cars, ship, truck), are labeled machine.
We train linear SVM (no kernel used) with the hinge loss. We evaluate both label flipping and gradient matching to generate camouflages, and different threat models (E, by. b.): the results are reported in Table Training details and hyperparameters
given in AAppeddxx B.3] Each poison and camouflage generation took about 40 50 seconds (for b, b, 0.2%).. Each trained model had validation accuracy of around 81.63% the clean dataset Sa,, which did not change significantly when we retrained after adding poison samples and/or camouflage samples. Note that the efficacy of the camouflaged poisoning attack more than 70% in most of the experiments. We give examples of the generated poisons/camouflages in Appendix B.5
4.1.2 Neural Networks
We perform extensive evaluations on the (multiclass) CIFAR-10 10 classification task with various popular large-scale neural networks models including VGG-11, VGG-16 (Simonyan and Zisserman 2013], ResNet-18, ResNet-34, ResNet-50 [He et al. 2016). and MobileNetV2 [Sandler et al. [2018]. trained using cross-entropy loss. Details training setup and hyperparameters in Appendix B.3
We report the efficacy of our camouflaged poisoning attack across different models and threat models (e, bp. b.) in Figure 3: also see Appendix B.3 for detailed results. Each model was trained to have

---

Atagg Success
Attack Success
60
6.6
554
504
02
80
06.
1ow2h 6.115 0y-2%..m22.
Model Architecture
Threat Model
Figure 3: Efficacy of the proposed camouflaged poisoning attack on CIFAR-I0 dataset. The left plot gives the success for the threat model 16, b, 0.6%, b. 0.6% for different neural network architectures. The right plot gives the success for ResNet- 18 architecture for different threat models.
validation accuracy between 81-87% (depending on the architecture), which changed minimally when the model was retrained with poisons and/or camouflages. Camouflaging was successful at least 70% of the time for VGG-11, VGG-16, (esnet-18, and Resnet-34 and about 30% of the time for MobileNetV2 and Resnet-50.
Note that even small attack success rate represents a credible threat: first, these attack success rates are comparable with those of Geiping et al. [2021] and thus future or undisclosed targeted data poisoning attacks are likely to be more effective, Second, since these success rates are over the choice of the attacker's random seed, the attacker can locally repeat the attack several times and deploy successful one. Therefore, low success rates do not imply fundamental barrier, and can be overcome by incurring computational overhead - by repeating an attack 10x, even an attack with only 20% success rate can be boosted to 90% success rate.
4.2 Evaluations on Imagenette and Imagewoof
We evaluate the efficacy of our attack vector the challenging multiclass classification problem the Imagenette and Imagewoof datasets [Howard] 2019], both of which are subsets consisting of 10 classes from the Imagenet dataset [Russakovsky al. 2015 with about 900 images per class.
Threat Model
Attack Success
Dataset
Model
8
bp
b
Poison
Camou
IN
VGG-16
16
6.3%
6.3%
25%
100%
IN
Resnet- 18
16
6.3%
6.3%
40%
50%
IW
Resnet-18
16
6.6%
6.6%
50%
75%
Table 2: Evaluation of camouflaged poisoning attack on Imagenette (IN) and Imagewoof (IW)
datasets over 5 seeds (with I restart per seed). Note that camouflaging succeeded in most of the experiments in which poisoning succeeded.
We evaluate our camouflaged poisoning attack on two different neural network architectures- and ResNet- 18 (trained with cross entropy loss), and different threat models (c, bp. he) listed in Table More details about the dataset, experiment setup and hyperparameters are provided in Appendix B.4, In our experiments, camouflaging was successful for at least 50% of the time when poisoning was successful. However, because we modified about 13% the training dataset when adding poisons/camouflages,
observe that the fluctuation in the model's validation accuracy can be up to 7%, which expected since we make such large change training set.
4.3 Additional Experiments
Ablation experiments. In the appendix, we provide the additional experiments on CIFAR-10, showing that:
a Our attack is robust random deletions of generated poison and camouflage samples during evaluation (Appendix B.6.2), and to training with data augmentation.
(b) Our attack successfully transfers across different models, i.e., when the victim model is different from the model on which poison and camouflage samples were generated. Appendix B.6.3

---

(c)
Our attack is successful across different threat models i.e., different values of by and be Appendix B.6.I).
(d) The poison and camouflage samples generated in our experiments have similar feature space distributions, and thus data sanitization defenses (e.g.. Dia.oniaaos et al. (2019a]) based on feature distributions will not succeed removing the generated poison and camouflage samples Appendix B.6.5).
Approximate unlearning. In all our experiments so far, the victim retrains a model from scratch (on the leftover training data) to fulfil the unlearning request, i.e., the victim performs exact unlearning.
In the past few years there has been significant research effort in developing approximate unlearning
algorithms under various structural assumptions (e.g., convexity [Sekhari et al. [021] Guo et al.
2020), or under availability of large memory [Bourtoule et al., 2021 Brophy and Lowd [001 Graves et al.) [021]. etc.), and one may wonder whether attacks are still effective under approximate unlearning. Unfortunately, most existing approximate unlearning approaches are not efficient (either requiring strong assumptions taking too much memory) for large-scale deep learning settings considered in this paper, and thus we were unable to evaluate our attack against these methods with our available resources. However, we provide evalulations against the Amnesiac Unlearning algorithm of Graves et al. [2021], which we were able to implement (AAppendix B.6.6). We note that our attack continues to be effective even against this approximate unlearning method.
5 Discussion and Conclusion
We demonstrated a new attack vector, camouflaged poisoning allacks, against machine learning pipelines where training points can be umlearned. This shows that as we introduce new functionality to machine learning systems, we must be aware of novel threats that emerge. We outline a few extensions and future research directions:
Our method for generating poison and camouflage points was based on the gradient-matching attack of Geippng et al. [202]]. However, the attack framework could accommodate effectively any method for targeted poisoning, e.g.. the methods of/Aghakhani al. [2021] or ouuaa et al. (2020).
or any method introduced in the future. We provide preliminary experiments in Appepddi Cf
camouflaged-poison
attacks developed using the Bullseye Polytope technique from Aghakhani et al. 2022], and show that the attack continues to succeed.
While we only performed experiments with single target point, similar to Geiping et al. [202]] (and several other works in the targeted data poisoning literature), the proposed attack extends straightforwardly to a collection of targets (rather than single target). This can be done by
calculating the gradient of multiple (instead of single) target images and using in the gradient
matching in and [3]
In order to generate poisons and camouflages, our approach needs the ability to query gradients of a trained model new samples, thus the attack is not black-box. While the main contribution of the paper to expose that machine unlearning (which is new and emerging but still under- developed technology) can lead to new kind of vulnerability called a camouflaged poisoning attack, performing such an attack in black-box setting is an interesting research question.
While we could not verify literature [Bourtoule et al. 202T Brophy and Lowd| 202I, [Sekhari et al., [021] Guo et al. 2020] due to lack of resources needed implement them, believe that our attack should be effective against any provable approximate un unlearning approach. This is because the objective of approximate unlearning is to output model that is statisticalll-indistii indistinguussabble from the model srraiined-fromscraaled directly evaluate on the frreineddfromm-scaach model (which the underlying objective that all approximate unlearning methods aim to emulate). Verifying the success of our approach against other approximate unlearning approaches is an interesting future research direction.
attack against other approximate unlcarning methods in the
on the remaining data, and the experiments provided in the paper
We considered a two-stage process in this paper, with one round of learning and one round of learning. It would also be interesting to explore what kinds of threats are exposed by even more dynamic systems where points can be added or removed an online fashion.
Finally, it interesting to determine what other types of threats can be camouflaged, e.g., indiscriminate [Lu et al., 2022 or backdoor poisoning attacks (Chen et al. 2017]] Saha et al. 2020]. Beyond exploring this new attack vector, also independently interesting to understand how one can neutralize the effect of an attack by adding points.
10

---

Acknowledgements
We thank Chirag Jindal for useful discussions in the early phase of the project. AS thanks Karthik Sridharan for helpful discussions. GK is supported by a University of Waterloo startup grant, Canada CIFAR AI Chair, an NSERC Discovery Grant, and unrestricted gift from Google. JA is supported in part by the grant NSF-CCF- 1BCCF-1843300 1843000 (CAREER), NSF-CCC- 181933,, and a Google Faculty Fellowship. Computational resources were provided in part by GK's s Resources for Research Groups grant from the Digital Research Alliance of Canada.
References
Hojjat Aghakhani, Dongyu Meng, Yu- Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye polytope: A scalable clean-label poisoning attack with improved transferability. In 202/ IEEE European Symposium on Security and Privacy (EuroS&P), pages 159-178. IEEE, 2021.
Marco Barreno, Blaine Nelson, Anthony D Joseph, and J Doug Tygar. The security of machine learning. Machine Learning, 81(2):121-148, 2010.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Conference Machine Learning, ICML '12, pages 1467-1474. JMLR, Inc., 2012.
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choe Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In proceedings of the 42nd IEEE Symposium on Security and Privacy, SP '21, Washington, DC, USA, 2021. IEEE Computer Society.
Jonathan Brophy and Daniel Lowd. Machine unlearning for random forests. In Proceedings of the 38th International Conference Machine Leurning, ICML. '21, pages 1092-1104. JMLR, Inc., 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In Proceedings of the 36th IEEE Symposium on Security and Privacy, SP '15 pages 463-480, Washington, DC, USA, 2015. IEEE Computer Society.
Nicholas Carlini, Matthew Jagielski, Nicolas Papernot, Andreas Terzis, Florian Tramer, and Chiyuan Zhang. The privacy onion effect: Memorization is relative. arXiv preprint arXiv: 10469, 2022.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang, When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 996-9118 2021.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 17122.052, 2017.
Rishav Chourasia, Neil Shah, and Reza Shokri. Forget unlearning: Towards true data-deletion in machine learning. arxiv preprint arXiv:2210.08911, 2022.
Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon, Werner Zellinger, Bernhard A Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, and Fabio Roli. Wild patterns reloaded: A survey of machine learning security against training data poisoning. arXiv preprint arXiv:2205.01992, 2022.
Ias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, pages 1596-1606 PMLR, 2019a.
Ilias Diakonikolas, Gautam Kamath, Daniel M. Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: robust meta-algorithm for stochastic optimization. Proceedings of the 36th International Conference on Machine Learning, ICML '19, pages 1596-1606. JMLR, Inc., 2019b.

---

Min Du, Zhi Chen, Chang Liu, Rajvardhan Oak, and Dawn Song. Lifelong anomaly detection through unlearning. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1283-1297, 2019.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Conference on Theory of Cryptography, TCC '06, pages 265-284, Berlin, Heidelherg, 2006. Springer.
Federal Trade Commission. California company settles fic allegations it deceived consumers about use of facial recognition in photo storage app, January 2021.
Jonas Geiping, Liam Fowl, W Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. In Proceedings of the 9th International Conference Learning Representations, ICLR '21, 2021.
General Data Protection Regulation. Regulation (EU) 2016/679 the European parliament and of the council of 27 April 2016, 2016.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making AI forget you: Data deletion in machine learning. In Advances in Neural Information Processing Systems 32, NeurIPS '19, pages 3518-3531. Curran Associates, Inc., 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selec- tive forgetting in deep networks. In Proceedings of the 2020 IEEE Computer Society Conference Computer Vision and Pattern Recognition, CVPR '20, pages 9304 9304-9312, Washington, DC, USA, 2020. IEEE Computer Society.
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privase forgetting in deep networks. In Proceedings of the 2021 IEEE Computer Society Conference Computer Vision and Pattern Recognition, CVPR '21, pages 792-801, IEEE Computer Society, 2021.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. arXiv preprint 2002:20022.0044 2020.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11516-11524, 2021.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint wrXhv:708.006333 2017.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning, ICML '20, pages 3832-3842. JMLR, Inc., 2020.
Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi-Man Marrii-Mallajjrrdi, and Chris Waites. Adaptive machine unlearning, In Advances in Neural Information Processing Systems 34, NeurIPS '21, pages 16319-16330. Curran Associates, Inc., 2021.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the 2016 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR '16, pages 770-778, Washington, DC, USA, 2016. IEEE Computer Society.
Jeremy Howard. Imagenette, 2019. URL https:://ttthhhiie https//ww.ooo//ttttaa//a/aaetete
W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general-purpose clean-label data poisoning. In Advances in Neural Information Processing Systems 33, NeurIPS '20, pages 12080-12091. Curran Associates, Inc., 2020.
Information Commissioner's Office. Guidance on the ai auditing framework, February 2020.
12

---

Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette-Chood Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives false sense of privacy. arXiv preprint 12Xiv:2210.175440 2022.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models: Algorithms and evaluation. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, AISTATS '21, pages 1008-2016, JMLR, Inc., 2021.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference Learning Representations, ICLR 15, 2015.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning, 1 111(1):1-47, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Yiwei Lu, Gautam Kamath, and Yaoliang Yu. Indiscriminate data poisoning attacks neural networks. arXiv preprint 27X1:2204.000992, 2022.
Neil G Marchant, Benjamin IP Rubinstein, and Scott Alfeld. Hard to forget: Poisoning attacks on certified machine unlearning. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, volume 36 of AAAI '22, pages 7691-7700, 2022.
Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 27-38, 2017.
Seth Neel, Aaron Roth, and Saeed Maaaiii-Malaajerdi.. Descent-to-delete: Gradient-based methods for machine unlearning. Proceedings of the 32nd International Conference on Algorithmic Learning Theory, ALT 22. JMLR, Inc., 2021.
Quoc Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Variational bayesian unlearning. Advances in Neural Information Processing Systems, 33: 33:16025-16036 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, NeurIPS 19, pages 8026-8037. Curran Associates, Inc., 2019.
Andrea Paudice, Luis MunozzGGnzzlez and Emil C Lupu. Label sanitization against label flipping poisoning attacks. In Joint European conference on machine learning and knowledge discovery in databases, pages 5-15. Springer, 2018
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85):2825- 2830, 2011.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255) 2015.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11957-11963 2020.
13

---

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chichhhhhhhhh Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR '18, pages 4510-4520, Washington, DC, USA, 2018. IEEE Computer Society.
Avi Schwarzschild, Micah Goldblum, Arjum Gupta, John P Dickerson, and Tom Goldstein. Just how toxic is data poisoning? unified benchmark for backdoor and data poisoning attacks, 2021.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Advances in Neural Information Processing Systems 34, NeurlPS '21, pages 18075-18086. Curran Associates, Inc., 2021.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. Advances in neural information processing systems, 31, 2018.
Karen Simonyan and Andrew Zisserman, Very deep convolutional networks for large-scale visual recognition. In Proceedings of the 3rd International Conference Learning Representations, ICLR 15, 2015.
David Marco Sommer, Liwei Song, Sameer Wagh, and Prateck Mittal. Towards probabilistic verification of machine unlearning. arXiv preprint arXiv:2003.044 2020.
Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. In Advances in Neural Information Processing Systems 30, NeurlPS '17, pages 3520-3532. Curran Associates, Inc., 2017.
Octavian Suciu, Radu Marginean, Yigitcan Kaya, Hal Daume III, and Tudor Dumitras. When does machine learning [FAIL)} generalized transferability for evasion and poisoning attacks. In 27th USENIX Security Symposium (USENIX Security 18), pages 1299-1316, 2018.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv:1911.0796. 2019.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303-319. IEEE, 2022.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems 31, NeurIPS 18, pages 8011-8021. Curran Associates, Inc., 2018
Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pages 4126-4142, PMLR, 2021.
Han Xiao, Huang Xiao, and Claudia Eckert. Adversarial label flips attack on support vector machines. In ECAl 2012, pages 870-875. IOS Press, 2012.
Huang Xiao, Battista Biggio, Blaine Nelson, Han Xiao, Claudia Eckert, and Fabio Roli. Support vector machines under adversarial label contamination. Neurocomputing, 160:53-62, 2015.
Yu Yang, Tian Yu Liu, and Baharan Mirzasoleiman. Not all poisons are created equal: Robust training against data poisoning. In International Conference on Machine Learning, pages 25154 251655. PMLR. 2022.
Santiago Zanella- Béguueli, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 363-375, 2020.
14

---

A Gradient Matching for Efficient Poison Generation [Geiping et al., 2021
In this section, we discuss the key intuition of Geiping et al. 2021 for efficient poison generation. Our objective find perturbations A such that when the model trained the poisoned samples, it minimizes the adversarial loss in (hus making the victim model predict the wrong label Vadv on the target sample. However, directly solving (T)s is computationally intractable due to bilevel nature of the optimization objective. Instead, one may implicitly minimize the adversarial loss by finding a such that for any model parameter 0,
EL Vol(f(>' (4(')+),,)))))))
Ve(E(f 0), Madv)) s
P
In essence, (4) impllies that gradient based minimization (e.g., using Adam SGD) of the training loss
on poisoned samples also minimizes the adversarial loss. Thus, training model on Sa Spo will automatically ensure that the model predicts Yadv on target sample. Unfortunately, computing that satisfies
(4) is also intractable it is required to hold for all values of 0. The key idea of /Geiping) al. [021] to make poison generation efficient is to relax (4) to only satisfied for fixed model
0ct the the model obtained by training on the clean dataset Set. To implement this, Geiping et al. 2021 minimize the cosine-similarity loss between the two gradients defined as:
((A,0) = 1- (VoL(J 0), Undv). EP Vol(s(a; + A,,0)..))
V (Ttar f), Uadv )lll Σ Vol(s + +,,00))))
Geiping et al.
2021 demonstrated that [an can be efficiently optimized for many popular large-scale machine learning models and datasets. We provide the pseudocode Algorithm 2
Algorithm 2 Gradient Matching to generate poisons [Geiping et al. 2021
Require: Clean network (e::an))))) trained on uncorrupted base images Sat a target (Teer Star and an adversarial label Vadv. Poison budget P, perturbation bound E, number of restarts R. optimization steps M
1: Collect dataset Spo (x', "e). of P many images whose true label is Yadv-
2: for = I.... R restarts do
3: Randomly initialize perturbations s.t. || S.
4: for k l,..., M optimization steps do
5: Compute the loss o( A, fetean) as in (5) using the base poison images in Spo-
6: Update A using Adam update to minimize 6, and project onto the constraint set r.
7: end for
8: Amongst the R restarts, choose the A, with the smallest value of of A., Gelean).
9: end for
10: Return the poisoned set Spo fx' A A3,0" 1-11'
B Implementation Details
B.1
Code
We provide code for experiments ready-to-deploy Jupyter notebooks. The code work with different dataset can be found in:
Experiments for Binary-CIFAR-I0 dataset with linear SVM. cifar10. ipynb: Experiments for CIFAR-10 dataset with various neural network models.
imagenette... ibaggenetteipyyb:: Experiments for Imagenette/ / Imagewoof dataset with various neural network models.
We plan to make our code public with the final version of the paper.
15

---

B.2
Experimental Setup
For the ease of replication, we report the corresponding poison class, target class, camouflage class and Target ID for various seeds in different experiments.
Random Seed
Target Class
Poison Class
Camouflage Class
Target ID
2000000000
Deer
Bird
Deer
9621
2000000001
Cat
Horse
Cat
1209
2000000011
Frog
Bird
Frog
6503
2000000111 11
Bird
Cat
Bird
124
2000001 111
Plane
Deer
Plane
7649
200001 1111
Cat
Dog
Cat
4423
20001 11111
Truck
Car
Truck
8117
2001 1111111111
Bird
Truck
Bird
3686
201
Cat
Bird
Cat
642
2111111111 Frog
Ship
Frog
97
Table 3: Target, poison and camouflage class corresponding to different initial random seeds used for
CIFAR-10 experiments. The reported Target ID is relative the CIFAR-10 validation dataset.
Random Seed
Target Class
Poison Class
Camouflage Class
Target ID
2000000000
Building
Cassette player
Building
1559
2000000001
Chain saw
Gas pump
Chain saw
1266
2000000011 I
Truck
Cassette player
Truck
2460
20000001
Cassette player
Chain saw
Cassette player
792
2000001 111
Tench
Building
Tench
2500
2000011111
Chain saw
French horn
Chain saw
1162
20001 111I1
Parachute
English springer
Parachute
3826
2001111111
Cassette player
Parachute
Cassette player
1121
2011111111
Chain saw
Cassette player
Chain saw
1198
2111111111
Truck
Golf ball
Truck
2343
Table 4: Target class, poison class and camouflage class corresponding to different random seeds
used for Imagenette experiments. The reported target ID is relative to the Imagenette validation set.
Random Seed
Target Class
Poison Class
Camouflage Class
Target ID
2000000000
Border Terrier
Beagle
Border Terrier
1493
2000000001
English Foxhound
Old English Sheep Dog
English Foxhound
1362
2000000011
Golden Retriever
Beagle
Golden Retriever
2399
2000000111 11
Beagle
English Foxhound
Beagle
827
2000001
Shih-Tzu
Border Terrier
Shih-Tzu
250
200001
English Foxhound
Austrailian Terrier
English Foxhound
1405
20001
Dingo
Rodesian Ridgeback
Dingo
3810
2001
Beagle
Dingo
Beagle
1204
201
English Foxhound
Beagle
English Foxhound
1294
211
Golden Retriever
Samoeyed
Golden Retriever
2282
Table 5: Target class, poison class and camouflage class corresponding to different random seeds
used for Imagewoof experiments. The reported target ID) is relative the Imagewoof validation set.
B.3
Additional Details on CIFAR-10 Experiments
B.3.1
SVM Experiments
We train the linear SVM (no kernel was used) with the hinge loss: {(f(r, 0). = max(0.. yf, 0)). The training was done using the svm. LinearSVC class from Scikit-learn (Pedregosal et al. 2011 single CPU. In the pre-processing stage, each image in the training dataset was
16

---

normalized to have f2-norm 1. Each training on Binary-CIFAR-JO 10 dataset took 25 30 seconds. In order to generate the poison points, we first use torch. autograd to compute the similari loss (5) and then optimize it using Adam optimizer with learning rate 0.001. Each poison and camoufiage generation took about 40 50 seconds (for by = b 0.2%). We evaluate both label flipping and gradient matching generate camouflages, and different threat models (c, hy be): the results are reported in Table 6 For each our experiments we chose K = 10 seeds of the form "kkkkkk" where & {O,... . and the seed 99999. Each trained model had validation accuracy of around 81.63% on the clean dataset Set, which did not change significantly when we retrained after adding poison samples and / or camouflage samples. Note that the efficacy of the camouflaged poisoning attack was more than 70% in most of the experiments. We provide a sample of the generated poisons and camouflages in Figure in Appendix B.5
B.3.2 Support Vector Machines
Attack type (s, bp, be)
Attack success
Validation Accuracy
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
LF (8, 0.2%, 0.2%)
70%
71.5%
81.63
81.73 (# 0.14)
81.74 (# 0.20)
LF (16, 0.2%, 0.2%)
100%
40%
81.63
81.64 (+0.03)
81.6 (=0.02)
GM (8, 0.2%, 0.4%)
70%
100%
81.63
81.65 (+0.01)
81.62 (+0.02)
GM (16, 0.2%, 0.4%)
100%
70%
81.63
81.65 (+0.03)
81.63 (# 0.02)
Table 6: Camouflaged poisoning attack on linear SVM on Binary-CIFAR-10 dataset. The first column lists the threat model (e, b.) and the camouflaging type "L.F" for label flipping and "GM" for gradient matching.
In order to perform evaluations SVM, we first convert the CIFAR-10 dataset into binary classification dataset (which we term as
by merging the 10 classes into two groups: animal (y +1) and machine (y -1)) Images (in both training and test datasets) that were originally labeled (bird, cat, deer, dog, frog, horse) are relabeled animal, and the remaining images, with original labels (airplane, cars, ship, truck), labeled machine.
train a linear SVM (no kernel was used) with the hinge loss. We evaluate both label flipping and gradient matching to generate camouflages, and different threat models (e, bp. ba); the results are reported Table 6. Each poison and camouflage generation took about 40 50 seconds (for bp b. 0.2%). Each trained model had validation accuracy of around 81.63% the clean dataset Set, which did not change significantly when we retrained after adding poison samples and/or camouflage samples. Note that the efficacy of the camouflaged poisoning attack in most of the experiments. We give examples of the generated poisons/camouflages in Appendix B.5
than 70%
1.0
A.E
Ndaak Suctess
Attack Surcess
84
0.0
VGG-31
VGG 26
RenNet-18 ReaMet
Reslet S0Ncstahee22
6- K-26
Figure 4: Efficacy of the proposed camouflaged poisoning attack CH dataset. The left plot gives the success for the threat model E = 16, b, 0.6%, b. 0.6% for different neural network architectures. The right plot gives the success for ResNet- 18 architecture for different threat models.
Mode: t
B.3.3 Neural Network Experiments
Implementation Details and Hyperparameters. Each model is trained with cross eotsopy loss {([(x, ((fxx,0)y)) = logg(Pryy ((),))) on a single GPU using PyTorch (Paske et al. 2019], and using mini-batch SGD with weight decay 5e-4, momentum 0.9, learning rate 0.01, batch size 100, and 40 epochs over the training dataset. Each training run took about minutes. The poison and camouflage sets were generated using gradient matching by first defining the cosine-similariy
17

---

loss using torch. autograd and then minimizing it using Adam with learning rate of 0.1. Each poison/cumouflage generation took about 1.5 hours
report the efficacy of the proposed camouflaged poisoning attack on different neural network architectures where the threat model is given by 16, 16,b, bp 0.6%. b, 0.6%. The reported results are average over 5 seeds from 200000000000000000
Further Results. We next elaborate the results reported in Figure 3
In Table 7
11. In the first column under attack success, we report the number of times poisoning successful amongst the trials, and in the second column, report the number of times camouflaging was successful for the trials for which poisoning was successful.
In Table report the success of the proposed attack when we change the threat model, but fix the network architecture to be ResNet-18. Each experiment was repeated times times with restarts each time, and the mean success rate is reported. These experiments were conducted with 5 seeds from 2000011 1112111111111..
Attack success
Validation Accuracy
Network Architecture
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
VGG-11
100%
80%
85.01
85.03 (+ 0.37)
85.10 (+ 0.29)
VGG-16
80%
75%
87.68
87.42 (+ 0.17)
87.45 (+ 0.26)
ResNet-18
80%
75%
82.13
81.88 (+ 0.15)
81.80 (+ 0.12)
ResNet-34
80%
50%
82.45
82.61 (+ 0.30)
83.12 (+ 0.93)
ResNet-50
80%
25%
81.02
81.76 (+ 0.13)
84.62 (+ 0.71)
MobileNetV2
60%
33%
82.79
83.26 (+ 0.25)
85.47 (+ 0.27)
Table 7: Evaluating our proposed camouflaged poisoning attack on various model architectures the CI dataset with the threat model & =16,b, b, 0.6%, 0.6%.
B.4 Additional Details Imagenette Imagewoof Experiments
We evaluate the efficacy of our attack vector on the challenging multiclass classification problem on the Imagenette and Imagewoof datasets (Howard] 2019]. Imagenette is subset of 10 classes (Tench, English springer, Cassette player, Chain saw; Building/church, French horn, Truck, Gas pump, Golf ball, Parachute) from the Imagenet dataset [Russakovsky al. 2015). The Imagenette dataset consists of around 900 images of various sizes for each class. In total, we have 13394 images which are divided into a training dataset of size 9469 and test dataset of size 3925. To perform training, all images are resized and centrally cropped down to 224 x 224 pixels.
Imagewoof Howard 2019 is another subset of Imagenet dataset consisting of 10 classes (Shih-Tza Rodesian Ridgeback, Beagle, English Foxhound, Border Terrier, Austrailian Terrier, Golden Retriever.
*Our implementations of Algorithm and are not optimized for computational efficiency, and the provided wall clock times simply serve as a proof of concept that our attack is practically implementable. All computations are performed on a CPU, but can be made significantly faster by using GPU.
Threat model
Attack success
Validation Accuracy
0p
b
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
16
1%
1%
100%
80%
82.13
81.98 (+ 0.16)
82.12 (+ 0.21)
8
1%
1%
80%
75%
82.13
82.21 (# 0.21)
82.09 (+ 0.23)
16
2%
1%
100%
20%
82.13
82.31 (# 0.26)
82.19 (# 0.24)
8
2%
1%
100%
40%
82.13
82.43 (# 0.30)
82.34 (+ 0.27)
Table 8: Evaluating our proposed camouflaged poisoning attack various threat models with CIFAR-10 10 dataset trained on ResNet-18.
18

---

Old English Sheep Dog, Samoyed, Dingo). Imagewoof consists of around 900 images of various sizes for each class, and in total 12954 images which are divided into training dataset of size 9025 and test dataset of size 3929. Similar to Imagenette, we resize all images and crop to the central 224 224 pixels before training.
We evaluate our camouflaged poisoning attack on two different neural network architectures- Settleccttre--GG--LI 16 and ResNet-18, and different threat models (e, 6.) listed in Table 2 Each model is trained single GPU with cross-entropy loss, that is minimized using SGD algorithm with weight decay 5e-4, momentum 0.9 and batch size We start with learning rate 0.01, and exponentially decay with y= 0.9 after every epoch, for a total of 50 epochs over the training dataset. The poisons and camouflages were generated using gradient matching by first defining the sisiie-simllarit loss using torch. autograd and then optimizing it using Adam optimizer with learning rate 0.1.
19

---

Target
Poisons
Camouflages
Figure 5: Visualization of poisons and camouflages on Imagewoof dataset. The first and the third columns shows the original images, and the second and the fourth columns shows the corrupted images (with added A). The shown images were generated for a camouflaged poisoning attack on ResNet-18. with Seed 2111111110, bp = b. = 4.2%, 16. The target and camouflage class is Austrailian Terrier, and the poison class is Golden Retriever.
20

---

Poisons
Camouflages
Figure 6: Some representative poison and camouflage images for attack on Imagewoof dataset. In each pair, the left figure is the original picture from the training dataset and the right figure has been adversarially manipulated by adding A. The shown images were generated for a camouflaged poisoning attack Resnet-18, with Seed = 10000005, bp be = 6.6% and E 16. The target and camouflage class is English Springer, and the poison class is Building (Church).
B.5 Visualizations
STIAL
STILL
Target
Poisons
Camouflages
Figure 7: Visualization of poisons and camouflages on Imagenette dataset. The first and the third columns shows the original images, and the second and the fourth columns shows the corrupted images (with added A). The shown images were generated for a camouflaged poisoning attack on ResNet-18, with Seed 2000011111 and 8. The target and camouflage class is chain saw, and the poison class is French horn.
21

---

Target
Poisons
Camouflages
Figure 8: Visualization of poisons and camouflages CIFAR-10 10 dataset (multiclass classification task). The top row shows the original images and the bottom row shows the corresponding poisoned camouflaged images (with the added A). The shown images generated for camouflaged poisoning attack on ResNet- 18, with Seed = 2000000000, & 8, b, 0.2, b. = 0.4, poison class bird, target class deer, and the target ID 9621.
Target
Poisons
Camouflages
Figure 9: Visualization of poisons and camouflages on Binary-CIFARE 10 dataset (animal vs machine classification). The top row shows the original images and the bottom row shows the corresponding poisoned / camouflaged images (with the added A). The shown images were generated for camouflaged poisoning attack SVM, with Seed 555555, 16, by 0.2, 0.4, target ID 6646.
22

---

B.6 Additional Experiments
B.6.1 Ablation Study for Different Values of 6p and be
In this section, we explore the effect of changing the relative sizes of the poison samples and camouflage samples. The experiments are performed on ResNet-18 for CIFAR- 10 dataset with bp 1% and be (0.5. 1.5] % respectively, and vice-versa. We report the results in Table 9
One may also wonder what is the price that attacker has pay in terms of the success rate of poisoning (only) attack if it chooses to devote a part of the budget for camouflages. In [Tall 10 we report the results for comparing the success rate % poisons and 1 % camouflages, success rate with 2% poisons (and no camouflages.)
Problem parameters
Attack success
Validation Accuracy
bp
be
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
1%
1.5%
83%
80%
0.822
0.8191
0.8264
1%
0.5%
83%
40%
0.845
0.8395
0.8432
1%
1%
83%
80%
0.822
0.8274
0.8244
0.5%
1%
33%
50%
0.822
0.8261
0.8194
1.5%
1%
83%
20%
0.822
0.8156
0.8118
Table Effect of different sizes poison and camouflage datasets on the success the proposed camouflaged poisoning attack on CIFAR-10 dataset trained on ResNet-16 18, with & 16. The reported success rates are averaged over six different trials with seeds 20000111-211111111. For each experiment, do 4 restarts for every poison and camouflage generation.
Problem parameters
Attack success
Validation Accuracy
bp
b
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
0.5%
0.5%
33%
50%
0.822
0.8212
0.8238
1%
1%
83%
80%
0.822
0.8274
0.8244
2%
83%
N/A
0.822
0.817
N/A
Table 10: Comparison of success rate when the allocated budget 2% split as: (a) 1% poisons and 1 % camouflage, and (b) 2% poison samples and 0% camouflages. The experiment is performed on CIFAR-10 dataset trained on ResNet-18 with E 16. The reported success rates are averaged over six different trials with seeds 200000111 11111-211100 211111111. For each experiment, we do 4 restarts per poison or camouflage generation.
B.6.2 Robustness of the Proposed Attack to Random Deletions
next explore the effect of random removal the generated poison and camouflage samples on the success of our attack. In data-scraping scenario, a victim may not scrape all the data points modified by an attacker. In Table TT we report the effect on attack success when different amounts of the generated poison and camouflage samples are deleted uniformly at random.
23

---

Amount removed
Attack success
Validation Accuracy
Poisons
Camouflages
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
5%
5%
80%
75%
0.822
0.823
0.819
10%
10%
60%
33%
0.822
0.817
0.825
20%
20%
80%
100%
0.822
0.826
0.820
30%
30%
100%
80%
0.822
0.8215
0.818
40%
40%
20%
100%
0.822
0.825
0.817
Table 11: Effect of random removal of the generated poison and camouflage samples on the success of the proposed camouflaged poisoning attack on CIFAR-10 10 dataset trained on ResNet-11, with 16. The reported success rates are averaged over different trials with seeds 200001 10000111-2111111111 211111111. For each experiment, we do 4 restarts for every poison and camouflage generation.
B.6.3 Transfer Experiments
In this section, we show that the poison and camouflage samples generated by the proposed approach transfer across models. Thus, an attacker can successfully execute the camouflaged poisoning attack, even if the victim trains different model than the one on which the poison and camouflage samples were generated. We show the transfer success in Fifure 10) The brewing network denotes the network architecture on which poison and camouflage samples were generated (we adopt the same notation as Geiping et al. [202[]. The victim network denotes the model architecture used by the victim for training on the manipulated dataset.
We ran total of 3 experiments per (brewing model, victim model) pair using the seeds 2000000000- 2000000011 1. Each reported number denotes the fraction of times when both poisoning and camou- flaging were successful the transfer experiment, and thus the attack could take place.
Browing Network Wil 3a mallee1a
0.3
0.5
02
Be 18
voc.16
02
Victim Network
Figure 10: Transfer experiments on CIFARIO dataset.
B.6.4 Robustness to Data Augmentation
Data augmentation is commonly used to avoid overfitting in deep neural networks. In order to be applicable the real life, our poisoning and camouflaging attacks must be successful even when the model is trained with data augmentation. In order to validate this, we evaluate our approach on CIFAR- 10 dataset trained with data augmentation on ResNet- 8 the threat model 16, by b 1%; the results are in Table 12] The considered data augmentations are:
1. No Augmentation: Exact images from the training dataset are used.
2. Augmentation Set I: 50% chance that the image will be horizontally flipped, but no rotations.
3. Augmentation Set 2: 50% chance that the image will be horizontally flipped, and random rotations in Uniform(-10 10) degrees.
The reported results in Table 12 are an average over 5 random seeds from "kkkkkk" where s k: < As expected, the validation accuracy for the model trained on clean dataset increased from 82%

---

percent when trained without augmentation, to 86% for augmentation Set I and 88% for augmentation set 2. The addition of data augmentation during training and re-training stages make harder for poisoning to succeed and at the same time makes it easier for camouflaging to succeed.
Attack success
Validation Accuracy
Data Augmentation
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
No Augmentation
100%
20%
82%
82%
82%
Augmentation Set
86%
33%
86%
85%
86%
Augmentation Set
60%
100%
88%
86
86%
Table 12: Effect of data augmentation on our proposed camouflaged poisoning attack.
B.6.5 Similarity of Feature Space Distance
A natural approach to defend against dataset manipulation attacks is to try to identify the modified images, and then remove them from the training dataset (i.e., data sanitization), c.g. Diakonikolas et al. [2019a]. For instance, one could cluster images based on their distance from their class mean image, or from the target image. This type of defense could potentially thwart watermarking poisoning attacks such as Poison Frogs (Shafahi et al.] 2018). As we show in Figure 11] such defense would not effective against our proposed poison and camouflage generation procedures, as the data distribution for the poison set and the camouflage set is similar to that of the clean images
from the respective classes.
L2 Distance Poison Class Mean
12 Distance to Target Image Festures
total poison dlass ponorn
total peison class
0.35
0.220
peisons
0029
Osunt
8:15
0085
6005
0.000
100
0.00
100
120
Distance Camoufiage Class Mean
1. Distance to Target Image Features
0.35
total camouflage class camouflages
tital
0090
0.00
camoullage
camouflage class
0000
Countt
0:05
0023
0.05
0085
0.00
0.00
Figure 11: Feature space distance for our generated poison and camouflage set. The reported data was collected by a successful camouflaged poisoning attack on Resnet-18 18 model trailed on CIFAR-10 with seed 2000000000, 16 and bp = b 1%.
B.6.6 Approximate Unlcarning
In the experiments reported so far, we assumed that the victim performs exact unlcarning, i.c. retrains the model from scratch on the leftover training data every time there is unlearning request. However, in many cases, retraining from scratch for every unlearning request could be computationally expensive. This has inspired plethora of research over the past few years in machine unlearning to develop approximate unlearning methods that are computationally faster than retraining from scratch (Sekhari et al., 2021, Guo al., 2020) Bourtoule et al., 2021 Brophy and Lowd 2021 Graves et al., 2021]. One may wonder if our attack also succeeds when performing approximate unlearning. Unfortunately, the algorithms dicussed in the prior works are infeasible for our large-scall deep learning setting; they either require strong structural properties (e.g., convexity (Sekhari et al.)
25

---

2021
Guo et al. 2020 or require access to large memory [Bourtoule et al. 2021 Brophy and Lowd
2021 Graves et al. 2 etc. Hence, we were unable evaluable most of these methods. However, we were successful in implementing the approximate unlearning methods in Graves et al. 2021 called Amnesiac Unlearning withing our available resources and report the results in Table 13
To give the high level idea of Ammesiac Unlearning, note that the final model produced by multi-epoch mini-batch SGD like algorithms given by:
E ΣΣ
8 final mitiall
e= b=1
where Ae,, is the incremental update using batch in epoch e. Ammesiac Unlearning, the learner keeps track of which batches contain which sample points. When given an unlearning request, the learner computes the bathces SB that contain the sensitive data (that requested for removal) and updates the model as
E
B Ao
S Σ
Oupdated Oinitial
Bdd,, Omniiill
Σ BD....
The above update is followed by few epochs of fine tuning on left over data samples to recover any [2021). the above update method is particularly
loss validation error. As reported in Graves et al.
effective when the sensitive data comprises of about 1-2% of the training dataset, which is the case in our experiments. We refer the reader to Graves et al. [202T] I for more details. The reported results in
Table 13 were obtained by using the open source implementation of Amnesiac Unlearning by Graves ct al. 2021]. We note that each experiment on CIFAR-10 took about 200GB of memory.
Threat model
Attack success
Validation Accuracy
8
bp
be
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
16
0.5%
1 %
40%
100%
0.913
0.765 (=0.047) 0.722 (+0.060)
16
1%
1%
40%
100%
0.913
0.760 (=0.046) 0.765 (+0.046)
Table 13: Evaluating the success our proposed camouflaged poisoning attack against Amnesiac Unlearning (an approximate unlearning technique) provided by Graves et al. [2021] using five randomly selected seeds provided the table above. We remark that our attack continues to be effective even against approximate unlearning.
B.7 Multiple Targets Scenarios
In addition, we performed a limited number of experiments poisoning and camouflaging multiple targets using the gradient matching method, The results (as shown Table 14) are comparable to the experiments performed by [Geiping et al. [202]] on multiple targets. We observe that our attack continues be effective under multiple target settings. Furthermore, choosing method that is more effective under the multi-target setting may increase the success of our attack.
Threat model
Targets
Attack Success
hp
be
Targets
hp Targets
b. / Targets
Poisoning
Camouflaging
16
1%
1%
0.5%
0.5%
70%
40%
16
1%
1%
5
0.25%
0.25%
36%
33%
16
1%
1%
10
0.1%
0.1%
21%
15%
Table 14: Evaluating the success of our proposed camouflaged poisoning attack against multiple targets using the Gradient Matching technique on the CIFAR-10 dataset with default parameters.
C Alternative Approaches to Generate Poisons and Camouflages
Thus far, our experiments have focused solely on generating poison and camouflage points using the gradient matching technique. In this section, we show that our attack framework is robust to the
26

---

poison generation method: we generate camouflaged poisoning attack using Bullseye Polytope poisoning clean-label targeted attack proposed by AygAhakhani et al. [202T]. We believe that similar results should hold for variety of other data poisoning techniques.
Bullseye Polytope (BP) maximizes the similarity between representations of the poisons and target. In doing so, it implicitly minimizes the alignment between poison and target gradients with respect to the penultimate layer, which captures most of the gradient norm variation [Katharopoulos and Fleuret 2019]. In the transfer learning scenario, the poisons are crafted to have a similar representation that of the target. Here, linear layer is trained on the poisoned data using the representations obtained from a pre--reeiinee trained clean model. The gradient of the linear model is proportional to the representations learned by the pre-trained model. Therefore, by maximizing the similarity between the representations of the poisons and the udversarially labeled target, the attack indeed increases the alignment between their gradients Yang et al. 2022
In our experiments, a poison set and camouflage set were crafted and evaluated a white-bo box setting, where the method has the highest average attack success rate [Schwarzschild al.) 2021]. Two pre-trained ResNet-18 models are used as the feature extractor to generate adversarial examples and as the victim, respectively. We iterations each to obtaín poisons and camouflages. We then use Adam with a learning rate of 0.1 to fine-tune the victim's 's linear classifier on the poisoned dataset for 60 epochs. Finally, we re-load the victim model, repeat the fine-tuning process with the poisoned + camouflaged dataset and compare the results.
loo perturbation budget of
and perform BP for 1,000
Threat model
Attack success
Validation Accuracy
bp
be
Poisoning
Camouflaging
Clean
Poisoned
Camouflaged
5
5
90%
67%
0.870
0.8794 (+0.012) 0.8737 (+0.021)
8
5
10
90%
78%
0.870
0.8727 (+0.002) 0.8716 (+0.002)
Table 15: Evaluating the success of our proposed camouflaged poisoning attack with Bullseye Polytope poisoning (a clean-label targeted attack) provided by Aghakhani et al. [202]] on the CIFAR- 10 dataset using randomly selected seeds provided in the table above. We observe that our attack continues to be effective with different targeted poisoning methods.
27


Reviews:
[{"review_id": "foS8ELk47F", "reviewer": "Reviewer_eD7Z", "summary": "This paper introduces a clean-label targeted data poisoning attack for machine unlearning pipelines. Gradient matching [A] is used to generate the poisons, and experiments use the retraining from scratch method for machine unlearning.\n\nThe core idea is for an attacker to introduce poisons and camouflages to the clean dataset. The camouflages mask the effect of poisons, so the victim model trained on the combined dataset (clean + poison + camouflage) doesn't exhibit poisoned behavior. After this, the attacker makes an unlearning request to remove the camouflages. After the unlearning process, the model exhibits the poisoned behavior. The poisoning aims to misclassify a specific test sample to a chosen target class.\n\n[A] Jonas Geiping, et al. “Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching.” International Conference on Learning Representations. 2021.", "questions": "There are some parts of the text which are not clear and will benefit with additional explanation.\n\n1. (Line 172) What is $b_p$ and $b_c$?\n2. (Line 173) What is $P$?\n3. (Line 265) What is the validation accuracy for these models. I do not see them mentioned in the main text / appendix.", "limitations": "A discussion on potential negative societal impact is missing. As this is a poisoning attack paper with implications on real-world deployments of machine learning models, I would recommend the authors to include a comment.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper proposes an original attack idea – a targeted poisoning attack in a machine unlearning setup. \n\nThe optimization method used for generating the poisons (Gradient Matching) is a well-known method from poisoning literature to solve the bi-level problem. The experimental setup is decent with experiments covering a handful of datasets (CIFAR-10, ImageNette and Imagewoof) and machine learning models (SVMs, VGG, ResNets).\n\nI was intrigued by the question the paper poses, “We thus identify a new technical question of broader interest to the data poisoning community: Can one nullify a data poisoning attack by only adding points?” I agree that this direction of study might be of significant interest to the community.", "weaknesses": "There are certain weaknesses in the claims of the paper, and the experiments are not thorough enough. They need improvement.\n\n1. (Line 216) “camouflaging was successful ...  provided that poisoning was successful. A camouflaged poisoning attack is successful if both poisoning and camouflaging were successful.” This explanation doesn't seem consistent with the numbers reported in the tables. If the camouflaging success rate is conditional on the poisoning being successful, I would think camouflaging success rate ≤ poisoning success rate. But that doesn't seem to be true in Row 3 of Table 1. Furthermore, if both these numbers are independent of each other, the tables should also report when both happen, as mentioned in the last line of the quoted text. \n\n2. (Footnote) “We diverge slightly from the threat model described above, in that the adversary modifies rather than\nintroduces new points. We do this for convenience and do not anticipate the results would qualitatively change.”  \nI do not think this is true. If we have a clean image and also the (same clean image + perturbation) in the poisoning dataset, wouldn't it change the results?\n\n3. (Amount of training – Line 244) “Each model was trained to have validation accuracy between 81-87% (depending on the architecture)”.\n81-87% on CIFAR-10 seems to be quite a low number. It should be easy to achieve 90-95% on CIFAR-10 with VGG-11, VGG-16, Resnet-18, and Resnet-34. These models seem to be under trained. Given that no experiments are performed on ImageNet and compute shouldn't be a bottleneck for CIFAR-10 training, the models need to be trained more so that the experimental results have more value for future research. See https://github.com/kuangliu/pytorch-cifar for reference numbers.\n\n4. (Extension to a set of targets) The attack threat model used here aims to target a single image from the test set. Though this is a starting point, the applicability of such a poison is quite limited. A natural extension would be to do this for a collection of targets. Though this is mentioned as future work in the discussion section, I would recommend that being a part of this current work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their kind comments. They give strong support for the paper, pointing out novelty and interest of the attack, thoroughness of the investigation (covering several datasets and models) and the fact that we highlight new technical questions for the data poisoning community. We put significant effort into conveying these points, and believe they make our paper well above the NeurIPS bar. We have thoroughly tested the robustness of this vulnerability, evidenced by the length of our submission (27 pages) filled with numerous experiments, and through our code release, challenge the community to identify deficiencies in our attack. The reviewer identifies certain weaknesses, which we believe to be issues primarily with our writing rather than technical in nature, which we attempt to clarify. We also ran additional experiments to assuage the reviewer's concerns. \n\n> “camouflaging was successful ... provided that poisoning was successful.\n\nWe always report conditional success. The reported number for the poison success column (e.g. in Table 1) denotes the percentage of trials for which the poison succeeded. The reported number for the Camouflaging success column denote the percentage of trials for which camouflaging successed amongst all the trials where poisoning succeeded. Since the reported numbers are percentages they could not be directly compared. Formally, they are defined as:\n\n$\\text{poisoning success} = \\frac{|\\text{Number of trials for which poisoning succeeded}|}{|\\text{Total number of trials}|}$\n\nand \n\n$\\text{Camouflaging success} = \\frac{|\\text{Number of trials for which camouflaging succeeded}|}{|\\text{Total number of trials for which poisoning succeeded}|}$.\n\nIf the reviewer prefers, we are happy to additionally report the unconditional success rate. \n\n>“We diverge slightly from the threat model described above, in that the adversary modifies rather than introduces new points...”\n\nIn theory, the model where the adversary can modify (rather than only add points) may be stronger, since it gives them the power to both remove and add points. However, we are unaware of any situation in the data poisoning literature where these two models qualitatively differ. In addition, we assume the adversary is given the points to attack uniformly at random, which further reduces their power and bringing the two adversary models closer together. As for the specific situation mentioned by the reviewer: note that the unmodified images are separate from the perturbed images in our threat model, so this instance could not arise. \n\n> \"81-87% on CIFAR-10 seems to be quite a low number...\"\n\nThanks for pointing this out. The issue is that, in some of our initial experiments, we did not use data augmentation. However, Section C.6.4 and C.6.6 of the submission use data augmentation and achieve accuracy 88%-92%, which is more in line what the numbers you outline. They show that the attacks remains successful in these settings. We will re-run other experiments to include data augmentation. In particular, we already did this with multiple targets to address your next question: we achieved a validation accuracy of 91-92% on CIFAR-10, with an attack success rate of 50% for poisoning and 40% for camouflage, with .4% poisoning and camouflaging of the training set. We will refresh the numbers in the paper to reflect these experiments.\n\n\n\n> (Extension to a set of targets) The attack threat model used here aims to target a single image from the test set\n\nAs mentioned in footnote 6, it is straightforward to extend our attack to multiple targets by changing the objective to a sum over losses on the target points. We demonstrated this with experiments on two targets on CIFAR-10 (including data augmentation) and found that our attack continues to be effective. In particular, the attack success rates is 50% for poisoning and 40% for camouflaging. Recall that our attack is based on Witches' Brew, which fails to design effective attacks against even as few as five targets, even for the easier poisoning setting without camouflage. Designing more effective attacks for multiple targets is at the frontier of research on data poisoning, and such attacks would immediately result in better attacks for our setting.\n\nWe will add further discussions on all of the above mentioned comments, and well as additional experiments, to the final version of the paper. \n\n\n\n## Answer to specific questions\n> What is $b_p$ and $b_c$?\n\nAs mentioned in line 132, $b_p$ is the percentage of training sampled used as poisons, and $b_c$ are the perfectage of training samples used as camouflages. \n\n> (Line 173) What is P?\n\nAs mentioned in line 148, P denotes the total number of poison samples. Technically, $P = b_p \\times |S_{\\mathrm{cl}}| / 100$. \n\n\n>(Line 265) What is the validation accuracy for these models. I do not see them mentioned in the main text / appendix.\n\nWe apologize for missing out this important statistic. For Imagenette, the validation accuracy usually falls between 75%-80%. For imagewoof, the accuracy was lower at around 67%-72%. We will add a table in the final version of the paper. \n\n>Discussion on potential negative societal impact is missing \n\nThanks for raising this concern. Indeed, any paper exposing a vulnerability has the potential for negative social impact. We'll add discussion in the final version. "}, {"review_id": "yL4fGKztLr", "reviewer": "Reviewer_oSYx", "summary": "This paper introduces a very interesting novel attack vector for tampering with the behavior of a machine learning model by influencing its training dataset. It poses that the practice of \"unlearning\" examples that are requested to be taken down can be exploited. The attack proceeds in several steps: 1. The adversary poisons the training data with two types of poisons: camouflages and regular poisons. 2. The adversary asks that the camouflages be unlearned (as might be their right under some legislation). 3. Unlearning is applied by the model developers, which triggers the poisoning effect and some examples are misclassified at inference time of the \"unlearned\" model.", "questions": "No questions from me.", "limitations": "No concerns here.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a strong and novel attack.", "weaknesses": "I believe the biggest limitation of this work is that experiments are only conducted on small computer vision datasets. It would be very interesting to find out if these poisons work against modern text-to-image or large language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive support for the paper! We agree that these other settings are also interesting. However, rather than cover many disparate settings poorly, we chose to investigate computer vision datasets in extreme depth, evidenced by our paper being already quite long (27 pages), with a broad range of ablations and investigations. We believe these additional settings mentioned are worth investigation in follow-up work. "}, {"review_id": "RUA7nCSW74", "reviewer": "Reviewer_VjL7", "summary": "This paper proposes a new attack for security risks in the field of machine unlearning, namely the camouflaged data poisoning attack. The attacker adds poisoned and camouflaged points to the training data, and then triggers a deletion request to remove the camouflaged points, causing the model to misclassify the target points. The authors conduct an extensive evaluation to assess the performance of the proposed attack, and the results are promising.", "questions": "The authors introduce a new attack surface in the context of machine unlearning called camouflaged data poisoning attacks. I like this topic because both data poisoning attacks and machine unlearning are two popular research topics. In addition, the authors clearly describe the threat model, methodology, and evaluation. I appreciate that.\n\nHowever, I do have the following concerns.\n\n- The authors design two algorithms for generating camouflage samples. The first one is through label flipping and the second one is through gradient matching. My concern is that the second scenario where an attacker has access to model gradients is unrealistic, especially in so-called image-based social media platforms. I do not believe that such a platform would have white-box access to the user. Although the authors discuss this point in the last section, I wish the authors would focus more on black-box settings.\n\n- Figure 1 provides the attack performance of the newly designed poisoning attack. Here, I wonder why the attack success rate on ResNet-50 is much lower than other model architectures. I hope the authors will have more discussion on this. I conjecture that one of the reasons behind this may be the much larger model complexity of ResNet-50. Therefore, I would further like the authors to add an ablation study on the impact of target model complexity.\n\n- Following the above, I am also curious about how the proposed attack performs on more complex datasets. Specifically, all experiments are conducted on 10-class datasets. However, in real-world scenarios, the task is likely to be more complex, with more than 10 classes. Therefore, I would like the authors to perform an investigation on more complex datasets, such as CIFAR-100 and ImageNet.", "limitations": "Insufficient experiments, conclusions may not generalize to complex architectures and datasets", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Popular topics\n- Well-structured and easy to follow", "weaknesses": "- Unrealistic assumptions\n- Needs more detailed explanations\n- Needs more complex datasets", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer likes the direction, and for their thoughtful questions to keep us honest! We have investigated all these questions, and found that the attack is fairly robust to the modifications suggested. Details follow. \n\n> Extension of the attack to the black-box setting. \n\nThanks for this interesting concern. Our existing transfer experiments in Appendix C.6.3 address this concern, where we show that the poisons and camouflages generated on model A can be used to attack a model B. Thus, the attacker does not need gradient access to model B, and can generate the poisons/camouflages \"in-house\" on model A. \n\nExploring this further for more black-box setting, where the attacker does not have gradient access to any model is an interesting future research direction. We believe that techniques explored in Geiping et. al. 2021 (section 5.2) for black-box gradient matching attack should be directly extendable to implement our attack in a black-box setting. \n\nFinally, we also want to point out to the reviewer that we have additional experiments done using the Bullseye Polytope method in Appendix D which focuses on (more black-box) transfer learning scenarios. \n\nIf the reviewer remains skeptical, we're happy to run additional experiments to assuage any concerns.\n\n> Figure 1 provides the attack performance of the newly designed poisoning attack. Here, I wonder why the attack success rate on ResNet-50 is much lower than other model architectures. I hope the authors will have more discussion on this. I conjecture that one of the reasons behind this may be the much larger model complexity of ResNet-50. Therefore, I would further like the authors to add an ablation study on the impact of target model complexity.\n\nThe success of our attack is conditional on the success of gradient-matching technique of Geiping et al. for data poisoning. In their work, as reported in Figure 3, the attack efficacy is smaller in Resnet-50 as compated to Resnet-18. Our reported numbers follow a similar trend and range. Given a more successful data poisoning attack on Resnet-50 (not the focus of our work), we are confident that it would translate to a more effective attack in our setting.\n\nRegarding an ablation study regarding different models, please find this experiment in Section C.3.3 (Table 7). \n\n\n> Following the above, I am also curious about how the proposed attack performs on more complex datasets. Specifically, all experiments are conducted on 10-class datasets. However, in real-world scenarios, the task is likely to be more complex, with more than 10 classes. Therefore, I would like the authors to perform an investigation on more complex datasets, such as CIFAR-100 and ImageNet.\n\nWe followed the reviewer's suggestion and performed additional experiments on CIFAR-100. We found that our attack is still effective. In particular, the poison success rate is 60% (80% if we consider top 5), and the camouflage success rate is 100%, with $b_p$ = 200,  $b_c$ = 400 and $\\epsilon$ = 16. We will add details of this experiment to the final paper.\n\n\nUnfortunately, our group does not have access to the computational resources required to run experiments on ImageNet. "}, {"review_id": "5es0QVBTHo", "reviewer": "Reviewer_zrPH", "summary": "This paper studies poisoning attacks against machine unlearning systems, where the attacker injects a set of poisoning samples and a set of camouflage samples. When both sets of samples are present in the training set, the trained model performs similarly to the clean model but shows adversarial behaviors and misclassified a particular sample into a wrong label when the camouflage samples are deleted. Throurgh evaluations, the proposed attack can achieve reasonable poisoning success and the camouflaged samples also serve well in preserving correct predictions when present.", "questions": "Overall, I think the problem studied in this paper is very interesting and the presentation of this paper is also very clear. However, I have the following concerns:\n1. Misclassifying a single test point might not be practically relevant, attackers might be interested in investing a significant amount of poisoning points to misclassify as many test samples as possible. Although the authors discussed about the extension is straightforward, for practical implementation, I believe this will not work well. The original GradMatch attack performs poorly when the target number of samples exceeds maybe 5 images, not to mention that the proposed attack already performs inferior to the GradMatch attack in the standard training setting. So, I believe the authors may either tune down the statement slightly or demonstrate it through experiments (this will be ideal). \n2. The authors mentioned that testing with different random seeds can lead to drastically different effectiveness and in practice, it is hard to know the random seed used by the victim (I hope I understand the paper correctly that the attacker and victim are indeed using different random seeds. Otherwise, the experiments probably be needed to rerun). In this regard, how can the attacker pick a good random seed to work well against the unknown seed by the victim? Repeating multiple times may not help as it is related to the unknown seed used by the victim. \n3. The attack performs relatively poorly on more complex ImageNet datasets, and the performance of the GradMatch attack in the standard training is not listed. By checking the original paper, it seems the attack success of GradMatch is > 80% at <1% poisoning ratio, which means there is a big performance gap between the original GradMatch and the GradMatch in unlearning settings. This is probably a more worrying observation for me-- if there is an inherent barrier to poisoning complex models in unlearning settings, then we may have to worry too much about the poisoning threat. I guess this issue cannot be solved in a short amount of time but would be ideal if the authors can provide the attack performance when the total poisoning ratio is set as the common ratio used in prior poisoning literature (e.g., 1%). Related to this, the poisoning budget is usually 2x the budget of the poisoning standard setting, while in practice, attackers only have a limited budget. This implies the poisoning threat in machine unlearning will be less severe than poisoning in standard training.  \n4. There is an observation that the attack effectiveness varies drastically across model architectures. Do the authors have explanations for this? \n5. When generating the camouflaged points, what is the intuition behind using the poisoned model $\\theta_{cp}$ to generate the points? Is this a possible reason that leads to inferior performance of the attack on more complex datasets?  \n\n====================================================\n\nThis is to acknowledge that I have read the authors rebuttals carefully and my concerns are addressed.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of attacking machine unlearning systems is very interesting.\n2. The proposed attack, although majorly leverages the existing targeted poisoning attack, performs relatively effectively.\n3. Evaluations show that the proposed attack may transfer across different model architectures and also unlearning methods.", "weaknesses": "1. The extension to the practically relevant setting of misclassifying multiple test points is not demonstrated. \n2. The attack requires a significant amount of poisoning points to work for more complex datasets such as ImageNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are heartened to hear that the reviewer finds our paper to be an interesting attack, which is effective, and that we performed a thorough investigation which shows its efficacy across a variety of settings. We believe such a contribution to be above the bar for NeurIPS. We hope to assuage any concerns with our responses to the reviewer's thoughtful questions. \n\n> Multiple target points \n\n\nThank you for the feedback. Indeed, since our attack is based on the Gradient Matching attack of Geiping et al., we inherit any weaknesses of their method. Targeting multiple points, even in the easier data poisoning setting, remains a challenging question at the frontier of data poisoning research, and a stronger attack for these settings would immediately result in stronger attacks in our setting. We will soften the language in our paper to reflect this. Nonetheless, we performed experiments with two target points and 1% budget (500 images) where we found that the success rate is 70% for poisoning and 40% for camouflaging, respectively.\n\nWe also tried the harder setting with five target samples (where, as the reviewer points out, Geiping et al.'s attack is not very successful), and found the efficacy to be much lower - 36% and 33%, respectively. Recall that, as pointed out in our paper and Geiping et al.'s, even a low attack efficacy is problematic, as the adversary may repeat the attack until a successful attack is discovered, and then deploy it. \n\n\n> The authors mentioned that testing with different random seeds can lead to drastically different effectiveness and in practice, it is hard to know the random seed used by the victim (I hope I understand the paper correctly that the attacker and victim are indeed using different random seeds. Otherwise, the experiments probably be needed to rerun). In this regard, how can the attacker pick a good random seed to work well against the unknown seed by the victim? Repeating multiple times may not help as it is related to the unknown seed used by the victim.\n\nWe confirm that we do not assume the attacker and the victim use the same seed. Furthermore, the randomness in our experiments results primarily from the attacker's random seed. Given successful poison and camouflage samples, the attack remains effective robust to changes in the victim's random seed. In our experiments, we used both pre-trained models and model checkpoint (trained with a different seed than the attacker) as the victim and were able to obtain comparable results.\n\n> Bigger models and different poison/camouflage budgets \n\nUnfortunately, we are a small group without significant access to computational resources, and thus we do not have enough resources to evaluate the full ImageNet. We instead evaluated these attacks on more tractable versions of ImageNet, called Imagenette and Imagewoof, that we used for experiments with high resolution images. We found that even small poison budgets like 300-400 images accounted for a larger fraction of the dataset (hence, the reported percentages are larger). \n\nWe next address reviewers concerns about experiments with different poison and camouflage budgets. We want to point out that we already explore this in Appendic C.6.1 (Table 9, 10). We demonstrate that the attack continues to hold across different budget ranges. \n\n> There is an observation that the attack effectiveness varies drastically across model architectures. Do the authors have explanations for this?\n\nThe success of our attack is conditional on the success of gradient-matching technique of Geiping et al. for data poisoning. Our reported variability across different model architectures comes from the variability of gradient-matching across different model architecture, as reported in Figure 3 of Geiping et al. A more successful data poisoning attack (which is not the focus of our work) should immediately result in a more effective camouflage attack. Such attacks remain at the frontier of data poisoning research. \n\n> When generating the camouflaged points, what is the intuition behind using the poisoned model to generate the points? \n\nThe goal of camouflage samples is to cancel the effect of poison samples, when both are taken together. Thus, in order to cancel the effect of poisons, we generate camouflages on models trained on clearn + poison samples so that gradient matching for camouflage generation has some information about the poison samples. "}, {"review_id": "f7A8ja6ws2", "reviewer": "Reviewer_dUYW", "summary": "The paper proposes a novel attack in machine unlearning. There are two step in the attack process. In the first step, the poisoned data and the camouflaged data are fed into the initial model training, together with the benign data. The machine unlearning process would take out the camouflaged data and the retraining from scratch process will be on the poisoned data and the benign data. Experiments are done to demonstrate the proposed attack.", "questions": "See weakness.", "limitations": "Not applicable.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThe authors propose a novel attack in machine unlearning. \n\n•\tEvaluations are done on the label flipping and gradient matching setting for camouflaged data generation, two types of models (SVM and neural networks will multiple different network structures), and three datasets.\n\n•\tThe paper is well-written and easy to follow.", "weaknesses": "•\tThe paper makes impractical assumptions: \n\n        o\tIn step 4, the unlearned model is by training the clean ones and poisoned ones. This reviewer fails to understand if the initial training can identify the poisoning effect and require the camouflaged data to mask the poisoning effect, the why the machine unlearning procedure does not offer such capability, especially the authors adopt the retraining from scratch strategy to perform machine unlearning?\n\n•\tThe paper makes a few false claims:\n\n        o\tThis reviewer does not understand why the authors call the proposed approach clean-label when they have a poisoned set generated very similar to adversarial examples: looks the same for human but will be misclassified by the model. Even if the attack does not change the label directly, gradient matching would still change the optimization of the objective. Otherwise, if the label or the intermediate computation results is not changed and only the input data is slightly modified, the so-called attack becomes adversarial training.\n\n        o\tThe authors are making speculations which are not true. The authors keep emphasizing the proposed approach can be applied to approximate unlearning methods in the literature. But this reviewer does not buy this claim. For example, Bourtoule et al., 2021 would trace back to the intermediate models before the data to be removed is in use. This approach could fail the proposed attack.\n\n        o\tIn machine unlearning approach in this paper, which is retraining from scratch, is also a cost-heavy way to do machine unlearning. Therefore, the comments on 288-290 on the strong assumptions or taking too much memory is not as convincing.\n\n        o\tTypo: line 324, the second “learning” should be “unlearning”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Unfortunately, we were not fully able to parse your concerns. Please find the following response to the best of our interpretation. We would love to engage further with the reviewer if any of their concerns are not answered:\n\n> Retraining from scratch vs approximate unlearning \n\nIn machine unlearning, in both theory and practice, the goal is to output a model that is indistinguishable from the model trained from scratch on the leftover training dataset. Since this is the gold standard which all approximate unlearning approaches aim to achieve, we simply perform (our main) experiments by unlearning by retraining from scratch. However, in Appendix C.6.6 we also explore the success of our attack against an approximate unlearning approach that is applicable for large scale deep learning settings. \n\nUnfortunately, we are a small resource-constrained group, and perfoming experiments against the unlearning approach of Bourtoule et al., 2021 is not feasible for us: their method necessitates impractically large memory overheads, and at the cost of significant reduction of accuracy. Finally, regarding the reviewers concerns that retraining from scratch is cost-heavy, we remark that our poison and camouflage generation approach is actually quite efficient. We were able to perform each of our CIFAR-10 experiments in about 2 hours on a single GPU (when our code was not even GPU optimized) and we did not require extensive storage and memory (just need to store a trained model). This is much less than the extreme cost of Bourtoule et al, which necessitates storing multiple (on the order of 50 to 100) trained models. \n\n\n> This reviewer does not understand why the authors call the proposed approach clean-label when they have a poisoned set generated very similar to adversarial examples: looks the same for human but will be misclassified by the model\n\nAn attack is called clean-label attack if visually looking at the image and the label pair, a human cannot tell if some noise is added or not. In particular, a picture that has a label dog should visually look like dog. All our attacks are clean label attacks, as is evident from the process of poisons and camouflages generation. This is described in more detail in the paper of Geiping et al.\n"}]

Author rebuttal:

Meta-review: All reviewers except one (the last reviewer) are in favor of accepting this paper. The AC believes the rebuttal addresses most of the concerns raised by the last reviewer, e.g, clean label attack. Hence, the AC recommends accepting this paper.
Decision: Accept (poster)