SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sup>4</sup>We provide preliminary experiments in Appendix C of camouflaged-poisoning attacks developed using the Bullseye Polytope technique from Aghakhani et al. [2021], and show that the attack continues to succeed even when we use alternate poison and camouflage generation techniques.
This ordering is for notational convenience; naturally, the datapoints are shuffled to preclude the victim simply removing a prefix of the training data.
While (1) focuses on misclassifying a single target point, it is straightforward to extend this to multiple targets by changing the objective to a sum over losses on the target points.
We can also similarly show that the gradients (as well as higher order derivatives) are equal, i.e.,  $\nabla_{\theta}\sum_{S_{\mathrm{cpc}}}\ell(f(x,\theta),y) = \nabla_{\theta}\sum_{S_{\mathrm{cl}}}\ell(f(x,\theta),y)$  for all  $\theta$ . Thus, training a model on  $S_{\mathrm{cpc}}$  is equivalent to training it on  $S_{\mathrm{cl}}$ . In essence, the camouflages have perfectly canceled out the effect of the poisons. We validate the efficacy of this approach via experiments on linear SVM trained with hinge loss (which resembles linear loss when the domain is bounded). See Section 4.1.1 for details.
While label flipping is a simple and effective procedure to generate camouflages, it is fairly restrictive. Firstly, label flipping only works for binary classification problems trained with linear loss. Secondly, the attack is not clean label as the camouflage images are generated as  $(x^i, -y^i)$  by giving them the opposite label to the ground truth, which can be easily caught by a validator. Lastly, the attack is vulnerable to simple data purification techniques by the victim, e.g., the victim can protect themselves by preprocessing the data to remove all the images that have both the labels  $(y = +1$  and  $y = -1)$  in the training dataset. In the next section, we provide a different procedure to generate clean-label camouflages that works for general losses and multi-class classification problems.
3.3.2 Camouflages via Gradient Matching
Here, we discuss our main procedure to generate camouflages, which builds on the gradient matching idea of Geiping et al. [2021]. Note that, our objective in (2) is to find  $\Delta$  such that when the model is trained with the camouflages, it minimizes the original-target loss in (2) (with respect to the original label  $y_{\mathrm{tar}}$ ) thus making the victim model predict the correct label on this target sample. Since, (2) is computationally intractable, one may instead try to implicitly minimize the original-target loss by finding a  $\Delta$  such that for any model parameter  $\theta$ ,
$$
\nabla_ {\theta} \big (\ell \big (f \big (x _ {\mathrm {t a r}}, \theta \big), y _ {\mathrm {t a r}} \big) \big) \approx \frac {1}{C} \sum_ {i = 1} ^ {C} \nabla_ {\theta} \ell \big (f \big (x ^ {i} + \Delta^ {i}, \theta \big), y ^ {i} \big).
$$
The above equation suggests that minimizing (e.g., using Adam / SGD) on camouflage samples will also minimize the original-target loss, and thus automatically ensure that the model predicts the correct label on the target. Unfortunately, finding perturbations that satisfy the above is also intractable as the approximate equality is required to hold for all  $\theta$ . Building on Geiping et al. [2021], we relax this condition to be satisfied only for a fixed model  $\theta_{\mathrm{cp}}$ -the model trained on the dataset  $S_{\mathrm{cp}} = S_{\mathrm{cl}} + S_{\mathrm{po}}$ , and obtain the perturbations which minimize the cosine-similarity loss given by
$$
\begin{array}{l} \psi (\Delta , \theta) \tag {3} \\ = 1 - \frac {\left\langle \nabla_ {\theta} \ell \left(f \left(x _ {\operatorname {t a r}} , \theta\right) , y _ {\operatorname {t a r}}\right) , \sum_ {i = 1} ^ {C} \nabla_ {\theta} \ell \left(f \left(x _ {i} + \Delta^ {i} , \theta\right) , y _ {i}\right) \right\rangle}{\| \nabla_ {\theta} \ell \left(f \left(x _ {\operatorname {t a r}} , \theta\right) , y _ {\operatorname {t a r}}\right) \| \| \sum_ {i = 1} ^ {C} \nabla_ {\theta} \ell \left(f \left(x _ {i} + \Delta^ {i} , \theta\right) , y _ {i}\right) \|}. \\ \end{array}
$$
Implementation details. We minimize (3) using the Adam optimizer [Kingma and Ba, 2015] with a fixed step size of 0.1. In order to increase the robustness of camouflage generation, we do  $R$  restarts (where  $R \leq 10$ ). In each restart, we first initialize  $\Delta$  randomly such that  $\| \Delta \|_{\infty} \leq \varepsilon$  and perform  $M$  steps of Adam optimization to minimize  $\psi(\Delta, \theta_{\mathrm{cp}})$ . Each optimization step only requires a single differentiation of the objective  $\psi$  with respect to  $\Delta$ , and can be implemented efficiently. After each step, we project back the updated  $\Delta$  into the constraint set  $\Gamma$  so as to maintain the property that  $\| \Delta \|_{\infty} \leq \varepsilon$ . After doing  $R$  restarts, we choose the best round by finding  $\Delta_*$  with the minimum  $\psi(\Delta_*, \theta_{\mathrm{cp}})$ .
We generate poison samples by running Algorithm 2 (given in the appendix), and camouflage samples by running Algorithm 1 with  $R = 1$  and  $M = 250$ . Each experiment is repeated  $K$  times by setting a different seed each time, which fixes the target image, poison class, camouflage class, base poison images and base camouflage images. Due to limited computation resources, we typically set  $K \in \{3,5,8,10\}$  depending on the dataset and report the mean and standard deviation across different trials. We say that poisoning was successful if the model trained on  $S_{\mathrm{cp}} = S_{\mathrm{cl}} + S_{\mathrm{po}}$  predicts the label  $y_{\mathrm{adv}}$  on the target image. Furthermore, we say that camouflaging was successful if the model trained on  $S_{\mathrm{cpc}} = S_{\mathrm{po}} + S_{\mathrm{cl}} + S_{\mathrm{ca}}$  predicts back the correct label  $y_{\mathrm{tar}}$  on the target image, provided that poisoning was successful. A camouflaged poisoning attack is successful if both poisoning and camouflaging were successful.
<sup>7</sup>We diverge slightly from the threat model described above, in that the adversary modifies rather than introduces new points. We do this for convenience and do not anticipate the results would qualitatively change.
Algorithm 1 Gradient Matching to generate camouflages
Require: Network  $f(\cdot ;\theta_{\mathrm{cp}})$  trained on  $S_{\mathrm{cl}} + S_{\mathrm{po}}$  , the target  $(x_{\mathrm{tar}},y_{\mathrm{tar}})$  , Camouflage budget  $C$  perturbation bound  $\varepsilon$  , number of restarts  $R$  , optimization steps  $M$    
1: Collect a dataset  $S_{\mathrm{ca}} = \left\{x^{j},y^{j}\right\}_{j = 1}^{C}$  of  $C$  many images whose true label is  $y_{\mathrm{tar}}$    
2: for  $r = 1,\dots R$  restarts do   
3: Randomly initialize perturbations  $\Delta$  s.t.  $\| \Delta \|_{\infty}\leq \varepsilon$    
4: for  $k = 1,\ldots ,M$  optimization steps do   
5: Compute the loss  $\psi (\Delta ,\theta_{\mathrm{cp}})$  as in (5) using the base camouflage images in  $S_{\mathrm{ca}}$    
6: Update  $\Delta$  using an Adam update to minimize  $\psi$  , and project onto the constraint set  $\Gamma$    
7: end for   
8: Amongst the  $R$  restarts, choose the  $\Delta_{*}$  with the smallest value of  $\psi (\Delta_{*},\theta_{\mathrm{cp}})$    
9: end for   
10: Return the poisoned set  $S_{\mathrm{ca}} = \left\{x^{j} + \Delta_{*}^{j},y^{j}\right\}_{j = 1}^{C}$
4.1 Evaluations on CIFAR-10
CIFAR-10 [Krizhevsky, 2009] is a multiclass classification problem with 10 classes, with 6,000 color images in each class of size  $32 \times 32$ . We follow the standard split into 50,000 training images and 10,000 test images.
4.1.1 Support Vector Machines
Attack type (ε,bp,bc)	Attack success	Validation Accuracy
Poisoning	Camouflaging	Clean	Poisoned	Camouflaged
LF (8,0.2%,0.2%)	70%	71.5%	81.63	81.73 (±0.14)	81.74 (±0.20)
LF (16,0.2%,0.2%)	100%	40%	81.63	81.64 (±0.03)	81.6 (±0.02)
GM (8,0.2%,0.4%)	70%	100%	81.63	81.65 (±0.01)	81.62 (±0.02)
GM (16,0.2%,0.4%)	100%	70%	81.63	81.65 (±0.03)	81.63 (±0.02)
Table 1: Camouflaged poisoning attack on linear SVM on Binary-CIFAR-10 dataset. The first column lists the threat model  $(\varepsilon, b_p, b_c)$  and the camouflaging type "LF" for label flipping and "GM" for gradient matching. The implementation details are given in Appendix B.3.
In order to perform evaluations on SVM, we first convert the CIFAR-10 dataset into a binary classification dataset (which we term as Binary-CIFAR-10) by merging the 10 classes into two groups: animal ( $y = +1$ ) and machine ( $y = -1$ ). Images (in both training and test datasets) that were originally labeled (bird, cat, deer, dog, frog, horse) are relabeled animal, and the remaining images, with original labels (airplane, cars, ship, truck), are labeled machine.
We train a linear SVM (no kernel was used) with the hinge loss. We evaluate both label flipping and gradient matching to generate camouflages, and different threat models  $(\varepsilon, b_p, b_c)$ ; the results are reported in Table 1. Training details and hyperparameters are given in Appendix B.3. Each poison and camouflage generation took about 40 - 50 seconds (for  $b_p = b_c = 0.2\%$ ). Each trained model had validation accuracy of around  $81.63\%$  on the clean dataset  $S_{\mathrm{cl}}$ , which did not change significantly when we retrained after adding poison samples and/or camouflage samples. Note that the efficacy of the camouflaged poisoning attack was more than  $70\%$  in most of the experiments. We give examples of the generated poisons/camouflages in Appendix B.5.
4.1.2 Neural Networks
We perform extensive evaluations on the (multiclass) CIFAR-10 classification task with various popular large-scale neural networks models including VGG-11, VGG-16 [Simonyan and Zisserman, 2015], ResNet-18, ResNet-34, ResNet-50 [He et al., 2016], and MobileNetV2 [Sandler et al., 2018], trained using cross-entropy loss. Details on training setup and hyperparameters are in Appendix B.3.
We report the efficacy of our camouflaged poisoning attack across different models and threat models  $(\varepsilon, b_{p}, b_{c})$  in Figure 3; also see Appendix B.3 for detailed results. Each model was trained to have
Figure 3: Efficacy of the proposed camouflaged poisoning attack on CIFAR-10 dataset. The left plot gives the success for the threat model  $\varepsilon = 16$ ,  $b_{p} = 0.6\%$ ,  $b_{c} = 0.6\%$  for different neural network architectures. The right plot gives the success for ResNet-18 architecture for different threat models.
validation accuracy between  $81 - 87\%$  (depending on the architecture), which changed minimally when the model was retrained with poisons and/or camouflages. Camouflaging was successful at least  $70\%$  of the time for VGG-11, VGG-16, Resnet-18, and Resnet-34 and about  $30\%$  of the time for MobileNetV2 and Resnet-50.
Note that even a small attack success rate represents a credible threat: first, these attack success rates are comparable with those of Geiping et al. [2021], and thus future or undisclosed targeted data poisoning attacks are likely to be more effective. Second, since these success rates are over the choice of the attacker's random seed, the attacker can locally repeat the attack several times and deploy a successful one. Therefore, low success rates do not imply a fundamental barrier, and can be overcome by incurring a computational overhead – by repeating an attack 10x, even an attack with only  $20\%$  success rate can be boosted to  $\approx 90\%$  success rate.
4.2 Evaluations on Imagenette and Imageloof
We evaluate the efficacy of our attack vector on the challenging multiclass classification problem on the Imagenette and Imagewoof datasets [Howard, 2019], both of which are subsets consisting of 10 classes from the Imagenet dataset [Russakovsky et al., 2015] with about 900 images per class.
Dataset	Model	Threat Model	Attack Success
ε	bp	bc	Poison	Camou
IN	VGG-16	16	6.3%	6.3%	25%	100%
IN	Resnet-18	16	6.3%	6.3%	40%	50%
IW	Resnet-18	16	6.6%	6.6%	50%	75%
Table 2: Evaluation of camouflaged poisoning attack on Imagenette (IN) and Imagewoof (IW) datasets over 5 seeds (with 1 restart per seed). Note that camouflaging succeeded in most of the experiments in which poisoning succeeded.
We evaluate our camouflaged poisoning attack on two different neural network architectures-VGG-16 and ResNet-18 (trained with cross entropy loss), and different threat models  $(\varepsilon, b_{p}, b_{c})$  listed in Table 2. More details about the dataset, experiment setup and hyperparameters are provided in Appendix B.4. In our experiments, camouflaging was successful for at least  $50\%$  of the time when poisoning was successful. However, because we modified about  $13\%$  of the training dataset when adding poisons/camouflages, we observe that the fluctuation in the model's validation accuracy can be up to  $7\%$ , which is expected since we make such a large change in the training set.
4.3 Additional Experiments
Ablation experiments. In the appendix, we provide the additional experiments on CIFAR-10, showing that:
(a) Our attack is robust to random deletions of generated poison and camouflage samples during evaluation (Appendix B.6.2), and to training with data augmentation.
(b) Our attack successfully transfers across different models, i.e., when the victim model is different from the model on which poison and camouflage samples were generated. (Appendix B.6.3)
(c) Our attack is successful across different threat models i.e., different values of  $b_{p}$  and  $b_{c}$  (Appendix B.6.1).
(d) The poison and camouflage samples generated in our experiments have similar feature space distributions, and thus data sanitization defenses (e.g., Diakonikolas et al. [2019a]) based on feature distributions will not succeed in removing the generated poison and camouflage samples (Appendix B.6.5).
Approximate unlearning. In all our experiments so far, the victim retracts a model from scratch (on the leftover training data) to fulfil the unlearning request, i.e., the victim performs exact unlearning. In the past few years there has been significant research effort in developing approximate unlearning algorithms under various structural assumptions (e.g., convexity [Sekhari et al., 2021, Guo et al., 2020]), or under availability of large memory [Bourtoule et al., 2021, Brophy and Lowd, 2021, Graves et al., 2021], etc.), and one may wonder whether attacks are still effective under approximate unlearning. Unfortunately, most existing approximate unlearning approaches are not efficient (either requiring strong assumptions or taking too much memory) for large-scale deep learning settings considered in this paper, and thus we were unable to evaluate our attack against these methods with our available resources. However, we provide evaluations against the Amnesiac Unlearning algorithm of Graves et al. [2021], which we were able to implement (Appendix B.6.6). We note that our attack continues to be effective even against this approximate unlearning method.
We demonstrated a new attack vector, camouflaged poisoning attacks, against machine learning pipelines where training points can be unlearned. This shows that as we introduce new functionality to machine learning systems, we must be aware of novel threats that emerge. We outline a few extensions and future research directions:
- Our method for generating poison and camouflage points was based on the gradient-matching attack of Geiping et al. [2021]. However, the attack framework could accommodate effectively any method for targeted poisoning, e.g., the methods of Aghakhani et al. [2021] or Huang et al. [2020], or any method introduced in the future. We provide preliminary experiments in Appendix C of camouflaged-poisoning attacks developed using the Bullseye Polytope technique from Aghakhani et al. [2021], and show that the attack continues to succeed.
- While we only performed experiments with a single target point, similar to Geiping et al. [2021] (and several other works in the targeted data poisoning literature), the proposed attack extends straightforwardly to a collection of targets (rather than a single target). This can be done by calculating the gradient of multiple (instead of a single) target images and using it in the gradient matching in (5) and (3).
- In order to generate poisons and camouflages, our approach needs the ability to query gradients of a trained model at new samples, thus the attack is not black-box. While the main contribution of the paper was to expose that machine unlearning (which is a new and emerging but still under-developed technology) can lead to a new kind of vulnerability called a camouflaged poisoning attack, performing such an attack in a black-box setting is an interesting research question.
- While we could not verify our attack against other approximate unlearning methods in the literature [Bourtoule et al., 2021, Brophy and Lowd, 2021, Sekhari et al., 2021, Guo et al., 2020] due to lack of resources needed to implement them, we believe that our attack should be effective against any provable approximate unlearning approach. This is because the objective of approximate unlearning is to output a model that is statistically-indistinguishable from the model retrained-from-scratch on the remaining data, and the experiments provided in the paper directly evaluate on the retrained-from-scratch model (which is the underlying objective that all approximate unlearning methods aim to emulate). Verifying the success of our approach against other approximate unlearning approaches is an interesting future research direction.
- We considered a two-stage process in this paper, with one round of learning and one round of learning. It would also be interesting to explore what kinds of threats are exposed by even more dynamic systems where points can be added or removed in an online fashion.
- Finally, it is interesting to determine what other types of threats can be camouflaged, e.g., indiscriminate [Lu et al., 2022] or backdoor poisoning attacks [Chen et al., 2017, Saha et al., 2020]. Beyond exploring this new attack vector, it is also independently interesting to understand how one can neutralize the effect of an attack by adding points.
We thank Chirag Jindal for useful discussions in the early phase of the project. AS thanks Karthik Sridharan for helpful discussions. GK is supported by a University of Waterloo startup grant, a Canada CIFAR AI Chair, an NSERC Discovery Grant, and an unrestricted gift from Google. JA is supported in part by the grant NSF-CCF-1846300 (CAREER), NSF-CCF-1815893, and a Google Faculty Fellowship. Computational resources were provided in part by GK's Resources for Research Groups grant from the Digital Research Alliance of Canada.
Hojjat Aghakhani, Dongyu Meng, Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye polytope: A scalable clean-label poisoning attack with improved transferability. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), pages 159-178. IEEE, 2021.
Marco Barreno, Blaine Nelson, Anthony D Joseph, and J Doug Tygar. The security of machine learning. Machine Learning, 81(2):121-148, 2010.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Conference on Machine Learning, ICML '12, pages 1467-1474. JMLR, Inc., 2012.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In proceedings of the 42nd IEEE Symposium on Security and Privacy, SP '21, Washington, DC, USA, 2021. IEEE Computer Society.
Jonathan Brophy and Daniel Lowd. Machine unlearning for random forests. In Proceedings of the 38th International Conference on Machine Learning, ICML '21, pages 1092-1104. JMLR, Inc., 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In Proceedings of the 36th IEEE Symposium on Security and Privacy, SP '15, pages 463-480, Washington, DC, USA, 2015. IEEE Computer Society.
Nicholas Carlini, Matthew Jagielski, Nicolas Papernot, Andreas Terzis, Florian Tramer, and Chiyuan Zhang. The privacy onion effect: Memorization is relative. arXiv preprint arXiv:2206.10469, 2022.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 896-911, 2021.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.
Rishav Chourasia, Neil Shah, and Reza Shokri. Forget unlearning: Towards true data-deletion in machine learning. arXiv preprint arXiv:2210.08911, 2022.
Antonio Emanuele Cina, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon, Werner Zellinger, Bernhard A Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, and Fabio Roli. Wild patterns reloaded: A survey of machine learning security against training data poisoning. arXiv preprint arXiv:2205.01992, 2022.
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, pages 1596-1606. PMLR, 2019a.
Ilias Diakonikolas, Gautam Kamath, Daniel M. Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In Proceedings of the 36th International Conference on Machine Learning, ICML '19, pages 1596-1606. JMLR, Inc., 2019b.
A Gradient Matching for Efficient Poison Generation [Geiping et al., 2021]
In this section, we discuss the key intuition of Geiping et al. [2021] for efficient poison generation. Our objective is to find perturbations  $\Delta$  such that when the model is trained on the poisoned samples, it minimizes the adversarial loss in (1) thus making the victim model predict the wrong label  $y_{\mathrm{adv}}$  on the target sample. However, directly solving (1) is computationally intractable due to bilevel nature of the optimization objective. Instead, one may implicitly minimize the adversarial loss by finding a  $\Delta$  such that for any model parameter  $\theta$ ,
$$
\nabla_ {\theta} \left(\ell \left(f \left(x _ {\operatorname {t a r}}, \theta\right), y _ {\mathrm {a d v}}\right)\right) \approx \frac {\sum_ {i = 1} ^ {P} \nabla_ {\theta} \ell \left(f \left(x ^ {i} + \Delta^ {i} , \theta\right) , y ^ {i}\right)}{P}. \tag {4}
$$
In essence, (4) implies that gradient based minimization (e.g., using Adam / SGD) of the training loss on poisoned samples also minimizes the adversarial loss. Thus, training a model on  $S_{\mathrm{cl}} + S_{\mathrm{po}}$  will automatically ensure that the model predicts  $y_{\mathrm{adv}}$  on the target sample. Unfortunately, computing  $\Delta$  that satisfies (4) is also intractable as it is required to hold for all values of  $\theta$ . The key idea of Geiping et al. [2021] to make poison generation efficient is to relax (4) to only be satisfied for a fixed model  $\theta_{\mathrm{cl}}$  -the model obtained by training on the clean dataset  $S_{\mathrm{cl}}$ . To implement this, Geiping et al. [2021] minimize the cosine-similarity loss between the two gradients defined as:
$$
\phi (\Delta , \theta) = 1 - \frac {\left\langle \nabla_ {\theta} \ell \left(f \left(x _ {\operatorname {t a r}}, \theta\right) , y _ {\mathrm {a d v}}\right) , \sum_ {i = 1} ^ {P} \nabla_ {\theta} \ell \left(f \left(x _ {i} + \Delta_ {i} , \theta\right) , y _ {i}\right) \right\rangle}{\| \nabla_ {\theta} \ell \left(f \left(x _ {\operatorname {t a r}}, \theta\right) , y _ {\mathrm {a d v}}\right) \| \| \sum_ {i = 1} ^ {P} \nabla_ {\theta} \ell \left(f \left(x _ {i} + \Delta_ {i}, \theta\right) , y _ {i}\right) \|}, \tag {5}
$$
Geiping et al. [2021] demonstrated that (5) can be efficiently optimized for many popular large-scale machine learning models and datasets. We provide the pseudocode in Algorithm 2.
Algorithm 2 Gradient Matching to generate poisons [Geiping et al., 2021]