SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Require: Clean network  $f(\cdot; \theta_{\mathrm{clean}})$  trained on uncorrupted base images  $S_{\mathrm{cl}}$ , a target  $(x_{\mathrm{tar}}, y_{\mathrm{tar}})$  and an adversarial label  $y_{\mathrm{adv}}$ , Poison budget  $P$ , perturbation bound  $\varepsilon$ , number of restarts  $R$ , optimization steps  $M$
1: Collect a dataset  $S_{\mathrm{po}} = \left\{x^i, y^i\right\}_{i=1}^{P}$  of  $P$  many images whose true label is  $y_{\mathrm{adv}}$ .
2: for  $r = 1, \dots, R$  restarts do
3: Randomly initialize perturbations  $\Delta$  s.t.  $\| \Delta \|_{\infty} \leq \varepsilon$ .
4: for  $k = 1, \dots, M$  optimization steps do
5: Compute the loss  $\phi (\Delta ,\theta_{\mathrm{clean}})$  as in (5) using the base poison images in  $S_{\mathrm{po}}$
6: Update  $\Delta$  using an Adam update to minimize  $\phi$ , and project onto the constraint set  $\Gamma$ .
7: end for
8: Amongst the  $R$  restarts, choose the  $\Delta_{*}$  with the smallest value of  $\phi (\Delta_{*},\theta_{\mathrm{clean}})$ .
9: end for
10: Return the poisoned set  $S_{\mathrm{po}} = \left\{x^{i} + \Delta_{*}^{i},y^{i}\right\}_{i = 1}^{P}$ .
B.1 Code
We provide code for our experiments as ready-to-deploy Jupyter notebooks. The code to work with different dataset can be found in:
- SVM_Binay_cifar10.ipynb: Experiments for Binary-CIFAR-10 dataset with linear SVM.
- CIFar10.ipynb: Experiments for CIFAR-10 dataset with various neural network models.
- imagenette.ipynb: Experiments for Imagenette / Imagewoof dataset with various neural network models.
We plan to make our code public with the final version of the paper.
B.2 Experimental Setup
For the ease of replication, we report the corresponding poison class, target class, camouflage class and Target ID for various seeds in different experiments.
Random Seed	Target Class	Poison Class	Camouflage Class	Target ID
2000000000	Deer	Bird	Deer	9621
2000000001	Cat	Horse	Cat	1209
2000000011	Frog	Bird	Frog	6503
2000000111	Bird	Cat	Bird	124
2000001111	Plane	Deer	Plane	7649
2000011111	Cat	Dog	Cat	4423
2000111111	Truck	Car	Truck	8117
2001111111	Bird	Truck	Bird	3686
2011111111	Cat	Bird	Cat	642
2111111111	Frog	Ship	Frog	97
Table 3: Target, poison and camouflage class corresponding to different initial random seeds used for CIFAR-10 experiments. The reported Target ID is relative to the CIFAR-10 validation dataset.
Random Seed	Target Class	Poison Class	Camouflage Class	Target ID
2000000000	Building	Cassette player	Building	1559
2000000001	Chain saw	Gas pump	Chain saw	1266
2000000011	Truck	Cassette player	Truck	2460
2000000111	Cassette player	Chain saw	Cassette player	792
2000001111	Tench	Building	Tench	2500
2000011111	Chain saw	French horn	Chain saw	1162
2000111111	Parachute	English springer	Parachute	3826
2001111111	Cassette player	Parachute	Cassette player	1121
2011111111	Chain saw	Cassette player	Chain saw	1198
2111111111	Truck	Golf ball	Truck	2343
Table 4: Target class, poison class and camouflage class corresponding to different random seeds used for Imagenette experiments. The reported target ID is relative to the Imagenette validation set.
Random Seed	Target Class	Poison Class	Camouflage Class	Target ID
2000000000	Border Terrier	Beagle	Border Terrier	1493
2000000001	English Foxhound	Old English Sheep Dog	English Foxhound	1362
2000000011	Golden Retriever	Beagle	Golden Retriever	2399
2000000111	Beagle	English Foxhound	Beagle	827
2000001111	Shih-Tzu	Border Terrier	Shih-Tzu	250
2000011111	English Foxhound	Australian Terrier	English Foxhound	1405
2000111111	Dingo	Rodesian Ridgeback	Dingo	3810
2001111111	Beagle	Dingo	Beagle	1204
2011111111	English Foxhound	Beagle	English Foxhound	1294
2111111111	Golden Retriever	Samoyeyed	Golden Retriever	2282
Table 5: Target class, poison class and camouflage class corresponding to different random seeds used for Imagewoof experiments. The reported target ID is relative to the Imagewoof validation set.
B.3 Additional Details on CIFAR-10 Experiments
B.3.1 SVM Experiments
We train the linear SVM (no kernel was used) with the hinge loss:  $\ell(f(x, \theta), y) = \max\{0, 1 - yf(x, \theta)\}$ . The training was done using thesvm.LinearSVC class from Scikit-learn [Pedregosa et al., 2011] on a single CPU. In the pre-processing stage, each image in the training dataset was
normalized to have  $\ell_2$ -norm 1. Each training on Binary-CIFAR-10 dataset took 25 - 30 seconds. In order to generate the poison points, we first use torch.autograd to compute the cosine-similarity loss (5), and then optimize it using Adam optimizer with learning rate 0.001. Each poison and camouflage generation took about 40 - 50 seconds (for  $b_{p} = b_{c} = 0.2\%$ ). We evaluate both label flipping and gradient matching to generate camouflages, and different threat models  $(\varepsilon, b_{p}, b_{c})$ ; the results are reported in Table 6. For each of our experiments we chose  $K = 10$  seeds of the form "kkkkkk" where  $k \in \{0, \dots, 9\}$  and the seed 99999. Each trained model had validation accuracy of around  $81.63\%$  on the clean dataset  $S_{\mathrm{cl}}$ , which did not change significantly when we retrained after adding poison samples and / or camouflage samples. Note that the efficacy of the camouflaged poisoning attack was more than  $70\%$  in most of the experiments. We provide a sample of the generated poisons and camouflages in Figure 9 in Appendix B.5.
B.3.2 Support Vector Machines
Table 6: Camouflaged poisoning attack on linear SVM on Binary-CIFAR-10 dataset. The first column lists the threat model  $(\varepsilon, b_p, b_c)$  and the camouflaging type "LF" for label flipping and "GM" for gradient matching.
In order to perform evaluations on SVM, we first convert the CIFAR-10 dataset into a binary classification dataset (which we term as Binary-CIFAR-10) by merging the 10 classes into two groups: animal  $(y = +1)$  and machine  $(y = -1)$ . Images (in both training and test datasets) that were originally labeled (bird, cat, deer, dog, frog, horse) are relabeled animal, and the remaining images, with original labels (airplane, cars, ship, truck), are labeled machine.
We train a linear SVM (no kernel was used) with the hinge loss. We evaluate both label flipping and gradient matching to generate camouflages, and different threat models  $(\varepsilon, b_{p}, b_{c})$ ; the results are reported in Table 6. Each poison and camouflage generation took about 40 - 50 seconds (for  $b_{p} = b_{c} = 0.2\%$ ). Each trained model had validation accuracy of around  $81.63\%$  on the clean dataset  $S_{\mathrm{cl}}$ , which did not change significantly when we retrained after adding poison samples and/or camouflage samples. Note that the efficacy of the camouflaged poisoning attack was more than  $70\%$  in most of the experiments. We give examples of the generated poisons/camouflages in Appendix B.5.
Figure 4: Efficacy of the proposed camouflaged poisoning attack on CIFAR-10 dataset. The left plot gives the success for the threat model  $\varepsilon = 16$ ,  $b_{p} = 0.6\%$ ,  $b_{c} = 0.6\%$  for different neural network architectures. The right plot gives the success for ResNet-18 architecture for different threat models.
B.3.3 Neural Network Experiments
Implementation Details and Hyperparameters. Each model is trained with cross-entropy loss  $\ell(f(x, \theta), y) = -\log(\Pr(y = f(x, \theta)))$  on a single GPU using PyTorch [Paszke et al., 2019], and using mini-batch SGD with weight decay 5e-4, momentum 0.9, learning rate 0.01, batch size 100, and 40 epochs over the training dataset. Each training run took about 45 minutes. The poison and camouflage sets were generated using gradient matching by first defining the cosine-similarity
loss using torch . autograd and then minimizing it using Adam with a learning rate of 0.1 . Each poison/camouflage generation took about 1.5 hours. $^{8}$
Further Results. We next elaborate on the results reported in Figure 3. In Table 7, we report the efficacy of the proposed camouflaged poisoning attack on different neural network architectures where the threat model is given by  $\varepsilon = 16$ ,  $b_{p} = 0.6\%$ ,  $b_{c} = 0.6\%$ . The reported results are an average over 5 seeds from 200000000-2000001111. In the first column under attack success, we report the number of times poisoning was successful amongst the run trials, and in the second column, we report the number of times camouflaging was successful for the trials for which poisoning was successful.
In Table 8, we report the success of the proposed attack when we change the threat model, but fix the network architecture to be ResNet-18. Each experiment was repeated times 5 times with 8 restarts each time, and the mean success rate is reported. These experiments were conducted with 5 seeds from 2000011111-2111111111.
Network Architecture	Attack success	Validation Accuracy
Poisoning	Camouflaging	Clean	Poisoned	Camouflaged
VGG-11	100%	80%	85.01	85.03 (± 0.37)	85.10 (± 0.29)
VGG-16	80%	75%	87.68	87.42 (± 0.17)	87.45 (± 0.26)
ResNet-18	80%	75%	82.13	81.88 (± 0.15)	81.80 (± 0.12)
ResNet-34	80%	50%	82.45	82.61 (± 0.30)	83.12 (± 0.93)
ResNet-50	80%	25%	81.02	81.76 (± 0.13)	84.62 (± 0.71)
MobileNetV2	60%	33%	82.79	83.26 (± 0.25)	85.47 (± 0.27)
B.4 Additional Details on Imagenette / Imagewoof Experiments
We evaluate the efficacy of our attack vector on the challenging multiclass classification problem on the Imagenette and Imagewoof datasets [Howard, 2019]. Imagenette is a subset of 10 classes (Tench, English springer, Cassette player, Chain saw, Building/church, French horn, Truck, Gas pump, Golf ball, Parachute) from the Imagenet dataset [Russakovsky et al., 2015]. The Imagenette dataset consists of around 900 images of various sizes for each class. In total, we have 13394 images which are divided into a training dataset of size 9469 and test dataset of size 3925. To perform training, all images are resized and centrally cropped down to  $224 \times 224$  pixels.
Imagewoof [Howard, 2019] is another subset of Imagenet dataset consisting of 10 classes (Shih-Tzu, Rodesian Ridgeback, Beagle, English Foxhound, Border Terrier, Australian Terrier, Golden Retriever,
Table 7: Evaluating our proposed camouflaged poisoning attack on various model architectures on the CIFAR-10 dataset with the threat model  $\varepsilon = 16, b_{p} = 0.6\%, b_{c} = 0.6\%$ .
Threat model	Attack success	Validation Accuracy
ε	bp	bc	Poisoning	Camouflaging	Clean	Poisoned	Camouflaged
16	1%	1%	100%	80%	82.13	81.98 (± 0.16)	82.12 (± 0.21)
8	1%	1%	80%	75%	82.13	82.21 (± 0.21)	82.09 (± 0.23)
16	2%	1%	100%	20%	82.13	82.31 (± 0.26)	82.19 (± 0.24)
8	2%	1%	100%	40%	82.13	82.43 (± 0.30)	82.34 (± 0.27)
Table 8: Evaluating our proposed camouflaged poisoning attack on various threat models with CIFAR-10 dataset trained on ResNet-18.
Our implementations of Algorithm 1 and 2 are not optimized for computational efficiency, and the provided wall clock times simply serve as a proof of concept that our attack is practically implementable. All computations are performed on a CPU, but can be made significantly faster by using GPU.
Old English Sheep Dog, Samoyed, Dingo). Imagewoof consists of around 900 images of various sizes for each class, and in total 12954 images which are divided into a training dataset of size 9025 and test dataset of size 3929. Similar to Imagenette, we resize all images and crop to the central  $224 \times 224$  pixels before training.
We evaluate our camouflaged poisoning attack on two different neural network architectures-VGG-16 and ResNet-18, and different threat models  $(\varepsilon, b_{p}, b_{c})$  listed in Table 2. Each model is trained on a single GPU with cross-entropy loss, that is minimized using SGD algorithm with weight decay 5e-4, momentum 0.9 and batch size 20. We start with a learning rate of 0.01, and exponentially decay it with  $\gamma = 0.9$  after every epoch, for a total of 50 epochs over the training dataset. The poisons and camouflages were generated using gradient matching by first defining the cosine-similarity loss using torch.autograd and then optimizing it using Adam optimizer with learning rate 0.1.
Figure 5: Visualization of poisons and camouflages on Imagewood dataset. The first and the third columns show the original images, and the second and the fourth columns show the corrupted images (with added  $\Delta$ ). The shown images were generated for a camouflaged poisoning attack on ResNet-18, with Seed = 2111111110,  $b_{p} = b_{c} = 4.2\%$ ,  $\varepsilon = 16$ . The target and camouflage class is Australian Terrier, and the poison class is Golden Retriever.
Figure 6: Some representative poison and camouflage images for attack on Imagewood dataset. In each pair, the left figure is the original picture from the training dataset and the right figure has been adversarially manipulated by adding  $\Delta$ . The shown images were generated for a camouflaged poisoning attack on Resnet-18, with Seed = 10000005,  $b_{p} = b_{c} = 6.6\%$  and  $\varepsilon = 16$ . The target and camouflage class is English Springer, and the poison class is Building (Church).
B.5 Visualizations
Poisons
Figure 7: Visualization of poisons and camouflages on Imagenette dataset. The first and the third columns show the original images, and the second and the fourth columns show the corrupted images (with added  $\Delta$ ). The shown images were generated for a camouflaged poisoning attack on ResNet-18, with Seed = 2000011111 and  $\varepsilon = 8$ . The target and camouflage class is chain saw, and the poison class is French horn.
Figure 8: Visualization of poisons and camouflages on CIFAR-10 dataset (multiclass classification task). The top row shows the original images and the bottom row shows the corresponding poisoned / camouflaged images (with the added  $\Delta$ ). The shown images were generated for a camouflaged poisoning attack on ResNet-18, with Seed = 2000000000,  $\varepsilon = 8$ ,  $b_{p} = 0.2$ ,  $b_{c} = 0.4$ , poison class bird, target class deer, and the target ID 9621.
Figure 9: Visualization of poisons and camouflages on Binary-CIFAR-10 dataset (animal vs machine classification). The top row shows the original images and the bottom row shows the corresponding poisoned / camouflaged images (with the added  $\Delta$ ). The shown images were generated for a camouflaged poisoning attack on SVM, with Seed = 555555,  $\varepsilon = 16$ ,  $b_{p} = 0.2$ ,  $b_{c} = 0.4$ , target ID 6646.
B.6 Additional Experiments
B.6.1 Ablation Study for Different Values of  $b_{p}$  and  $b_{c}$
In this section, we explore the effect of changing the relative sizes of the poison samples and camouflage samples. The experiments are performed on ResNet-18 for CIFAR-10 dataset with  $b_{p} = 1\%$  and  $b_{c} = \{0.5, 1, 1.5\} \%$  respectively, and vice-versa. We report the results in Table 9.
One may also wonder what is the price that an attacker has to pay in terms of the success rate of a poisoning (only) attack if it chooses to devote a part of the budget for camouflages. In Table 10, we report the results for comparing the success rate  $1\%$  poisons and  $1\%$  camouflages, to success rate with  $2\%$  poisons (and no camouflages.)
Problem parameters	Attack success	Validation Accuracy
bp	bc	Poisoning	Camouflaging	Clean	Poisoned	Camouflaged
1%	1.5%	83%	80%	0.822	0.8191	0.8264
1%	0.5%	83%	40%	0.845	0.8395	0.8432
1%	1%	83%	80%	0.822	0.8274	0.8244
0.5%	1%	33%	50%	0.822	0.8261	0.8194
1.5%	1%	83%	20%	0.822	0.8156	0.8118
Table 9: Effect of different sizes of poison and camouflage datasets on the success of the proposed camouflaged poisoning attack on CIFAR-10 dataset trained on ResNet-18, with  $\varepsilon = 16$ . The reported success rates are averaged over six different trials with seeds 200000111-211111111. For each experiment, we do 4 restarts for every poison and camouflage generation.
Problem parameters	Attack success	Validation Accuracy
bp	bc	Poisoning	Camouflaging	Clean	Poisoned	Camouflaged
0.5%	0.5%	33%	50%	0.822	0.8212	0.8238
1%	1%	83%	80%	0.822	0.8274	0.8244
2%	0	83%	N/A	0.822	0.817	N/A
Table 10: Comparison of success rate when the allocated budget of  $2\%$  is split as: (a)  $1\%$  poisons and  $1\%$  camouflage, and (b)  $2\%$  poison samples and  $0\%$  camouflages. The experiment is performed on CIFAR-10 dataset trained on ResNet-18 with  $\varepsilon = 16$ . The reported success rates are averaged over six different trials with seeds 200000111 - 211111111. For each experiment, we do 4 restarts per poison or camouflage generation.
B.6.2 Robustness of the Proposed Attack to Random Deletions
We next explore the effect of random removal of the generated poison and camouflage samples on the success of our attack. In a data-scraping scenario, a victim may not scrape all the data points modified by an attacker. In Table 11, we report the effect on attack success when different amounts of the generated poison and camouflage samples are deleted uniformly at random.
Amount removed	Attack success	Validation Accuracy
Poisons	Camouflages	Poisoning	Camouflaging	Clean	Poisoned	Camouflaged
5%	5%	80%	75%	0.822	0.823	0.819
10%	10%	60%	33%	0.822	0.817	0.825
20%	20%	80%	100%	0.822	0.826	0.820
30%	30%	100%	80%	0.822	0.8215	0.818
40%	40%	20%	100%	0.822	0.825	0.817
Table 11: Effect of random removal of the generated poison and camouflage samples on the success of the proposed camouflaged poisoning attack on CIFAR-10 dataset trained on ResNet-18, with  $\varepsilon = 16$ . The reported success rates are averaged over 5 different trials with seeds 200001111 - 211111111. For each experiment, we do 4 restarts for every poison and camouflage generation.
B.6.3 Transfer Experiments
In this section, we show that the poison and camouflage samples generated by the proposed approach transfer across models. Thus, an attacker can successfully execute the camouflaged poisoning attack, even if the victim trains a different model than the one on which the poison and camouflage samples were generated. We show the transfer success in Figure 10. The brewing network denotes the network architecture on which poison and camouflage samples were generated (we adopt the same notation as Geiping et al. [2021]). The victim network denotes the model architecture used by the victim for training on the manipulated dataset.
We ran a total of 3 experiments per (brewing model, victim model) pair using the seeds 2000000000-200000011. Each reported number denotes the fraction of times when both poisoning and camouflaging were successful in the transfer experiment, and thus the attack could take place.
Figure 10: Transfer experiments on CIFAR10 dataset.
B.6.4 Robustness to Data Augmentation
Data augmentation is commonly used to avoid overfitting in deep neural networks. In order to be applicable in the real life, our poisoning and camouflaging attacks must be successful even when the model is trained with data augmentation. In order to validate this, we evaluate our approach on CIFAR-10 dataset trained with data augmentation on ResNet-18 in the threat model  $\varepsilon = 16$ ,  $b_{p} = b_{c} = 1\%$ ; the results are in Table 12. The considered data augmentations are:
1. No Augmentation: Exact images from the training dataset are used.
2. Augmentation Set 1:  $50\%$  chance that the image will be horizontally flipped, but no rotations.
3. Augmentation Set 2:  $50\%$  chance that the image will be horizontally flipped, and random rotations in Uniform(-10, 10) degrees.
The reported results in Table 12 are an average over 5 random seeds from "kkkkk" where  $1 \leq k \leq 5$ . As expected, the validation accuracy for the model trained on clean dataset increased from  $82\%$
percent when trained without augmentation, to  $86\%$  for augmentation Set 1 and  $88\%$  for augmentation set 2. The addition of data augmentation during training and re-training stages make it harder for poisoning to succeed and at the same time makes it easier for camouflaging to succeed.
Data Augmentation	Attack success	Validation Accuracy
Poisoning	Camouflaging	Clean	Poisoned	Camouflaged
No Augmentation	100%	20%	82%	82%	82%
Augmentation Set 1	86%	33%	86%	85%	86%
Augmentation Set 2	60%	100%	88%	86	86%
Table 12: Effect of data augmentation on our proposed camouflaged poisoning attack.
B.6.5 Similarity of Feature Space Distance
A natural approach to defend against dataset manipulation attacks is to try to identify the modified images, and then remove them from the training dataset (i.e., data sanitization), e.g. Diakonikolas et al. [2019a]. For instance, one could cluster images based on their distance from their class mean image, or from the target image. This type of defense could potentially thwart watermarking poisoning attacks such as Poison Frogs [Shafahi et al., 2018]. As we show in Figure 11, such a defense would not be effective against our proposed poison and camouflage generation procedures, as the data distribution for the poison set and the camouflage set is similar to that of the clean images from the respective classes.
Figure 11: Feature space distance for our generated poison and camouflage set. The reported data was collected by a successful camouflaged poisoning attack on Resnet-18 model trailed on CIFAR-10 with seed 2000000000,  $\varepsilon = 16$  and  $b_{p} = b_{c} = 1\%$ .
B.6.6 Approximate Unlearning
In the experiments reported so far, we assumed that the victim performs exact unlearning, i.e. retrans the model from scratch on the leftover training data every time there is an unlearning request. However, in many cases, retraining from scratch for every unlearning request could be computationally expensive. This has inspired a plethora of research over the past few years in machine unlearning to develop approximate unlearning methods that are computationally faster than retraining from scratch [Sekhari et al., 2021, Guo et al., 2020, Bourtoule et al., 2021, Brophy and Lowd, 2021, Graves et al., 2021]. One may wonder if our attack also succeeds when performing approximate unlearning. Unfortunately, the algorithms discussed in the prior works are infeasible for our large-scale deep learning setting; they either require strong structural properties (e.g., convexity [Sekhari et al.,
2021, Guo et al., 2020]) or require access to large memory [Bourtoule et al., 2021, Brophy and Lowd, 2021, Graves et al., 2021], etc. Hence, we were unable to evaluable most of these methods. However, we were successful in implementing the approximate unlearning methods in Graves et al. [2021] called Amnesiac Unlearning withing our available resources and report the results in Table 13.
To give the high level idea of Amnesiac Unlearning, note that the final model produced by multi-epoch mini-batch SGD like algorithms is given by:
$$
\theta_ {f i n a l} = \theta_ {\text {i n i t i a l}} + \sum_ {e = 1} ^ {E} \sum_ {b = 1} ^ {B} \Delta_ {\theta_ {e, b}}
$$
where  $\Delta_{\theta_{e,b}}$  is the incremental update using batch  $b$  in epoch  $e$ . In Amnesiac Unlearning, the learner keeps track of which batches contain which sample points. When given an unlearning request, the learner computes the bathces  $SB$  that contain the sensitive data (that requested for removal) and updates the model as
$$
\theta_ {\text {u p d a t e d}} = \theta_ {\text {i n i t i a l}} + \sum_ {e = 1} ^ {E} \sum_ {b = 1} ^ {B} \Delta_ {\theta_ {e, b}} - \sum_ {s b = 1} ^ {S} B \Delta_ {\theta_ {s b}} = \theta_ {\text {i n i t i a l}} - \sum_ {s b = 1} ^ {S} B \Delta_ {\theta_ {s b}}.
$$
The above update is followed by a few epochs of fine tuning on left over data samples to recover any loss in validation error. As reported in Graves et al. [2021], the above update method is particularly effective when the sensitive data comprises of about  $1 - 2\%$  of the training dataset, which is the case in our experiments. We refer the reader to Graves et al. [2021] for more details. The reported results in Table 13 were obtained by using the open source implementation of Amnesiac Unlearning by Graves et al. [2021]. We note that each experiment on CIFAR-10 took about 200GB of memory.
Threat model	Attack success	Validation Accuracy
ε	bp	bc	Poisoning	Camouflaging	Clean	Poisoned	Camouflaged
16	0.5%	1%	40%	100%	0.913	0.765 (±0.047)	0.722 (±0.060)
16	1%	1%	40%	100%	0.913	0.760 (±0.046)	0.765 (±0.046)
B.7 Multiple Targets Scenarios