Augmentation Backdoors
Authors: Joseph Rance
Yiren Zhao
Ilia Shumailov
Robert D. Mullins
Venue: Submitted to ICLR 2023 (2023)
Keywords: training time attacks
backdoors
augmentation

Abstract: Data augmentation is used extensively to improve model generalisation. However, reliance on external libraries to implement augmentation methods introduces a vulnerability into the machine learning pipeline. It is well known that backdoors can be inserted into machine learning models through serving a modified dataset to train on. Augmentation therefore presents a perfect opportunity to perform this modification without requiring an initially backdoored dataset. In this paper we present three backdoor attacks that can be covertly inserted into data augmentation. Our attacks each insert a backdoor using a different type of computer vision augmentation transform, covering simple image transforms, GAN-based augmentation, and composition-based augmentation. By inserting the backdoor using these augmentation transforms, we make our backdoors difficult to detect, while still supporting arbitrary backdoor functionality. We evaluate our attacks on a range of computer vision benchmarks and demonstrate that an attacker is able to introduce backdoors through just a malicious augmentation routine.

Under review as a conference paper at ICLR 2023
AUGMENTATION BACKDOORS
Anonymous authors Paper under double-blind review
ABSTRACT
Data augmentation is used extensively to improve model generalisation. How- ever, reliance on external libraries to implement augmentation methods introduces vulnerability into the machine learning pipeline. It is well known that backdoors can be inserted into machine learning models through serving a modified dataset train on. Augmentation therefore presents perfect opportunity to perform this modification without requiring an initially backdoored dataset. In this paper we present three backdoor attacks that can be covertly inserted into data augmenta- tion. Our attacks each insert a backdoor using different type of computer vision augmentation transform, covering simple image transforms, GAN-based augmen- tation, and composition-based augmentation. By inserting the backdoor using these augmentation transforms, we make our backdoors difficult to detect, while still supporting arbitrary backdoor functionality. We evaluate attacks range computer vision benchmarks and demonstrate that an attacker is able to introduce backdoors through just malicious augmentation routine.
INTRODUCTION
Data augmentation is an effective way of improving model generalisation without the need for ad- ditional data (Perez & Wang (017). It is common rely on open source implementations of these augmentation techniques, which often leads to external code being inserted into machine learning pipelines without manual inspection. This presents threat to the integrity of the trained models. The use of external code to modify a dataset provides a perfect opportunity for an attacker to insert backdoor into a model without overtly serving the backdoor part of the original dataset.
Backdoors based on BadNet are generally implemented by directly serving a malicious dataset to the model (Gu et al. 2017). While this can result in effective backdoor, the threat of these supply chain attacks is limited by the requirement to directly insert the malicious dataset into the model's training procedure. We show that it possible
common augmentation techniques to modify a dataset without requiring the original to already contain backdoor. The general flow of backdoor insertion using augmentation is illustrated in Figure
More specifically, we present attacks using three different types of augmentation: (i) using standard transforms such as rotation or translation the trigger setup similar to BadNet (Gu al. 2017); (ii) using GAN-based augmentation such as DAGAN (Antoniou et al. 2017), trained to insert backdoor into the dataset; and (ili) using composed augmentations such as AugMix (Hendrycks
et al. 2020) to efficiently construct gradients in similar fashion to the Batch Order Backdoor described by Shumailoy al. (202T).
Clean data
Backdoored data
6
+
6
6
A model trained on "augmented" data will leam backdoor
"Augmentation"
Model
8
+ 8
+ 0
Backdoor function hidden within augmentation
Figure 1: An example of how the attacker inserts a backdoor using modified augmentation func- tion. this case, the function directly changes the label when the trigger transformation is applied.

---

Under review as a conference paper at ICLR 2023
In all three cases, the backdoored model has similar properties to BadNet, but with threat model which does not require training on an initially malicious dataset and an insertion process that is more difficult to detect because the backdoor is implemented using genuine transforms.
Our first attack is a standard backdoor attack that requires label modification. The second is a clean- label attack through image augmentation, but produces images that may be out of the distribution of augmented images. The final attack, requires no visible malicious modification at all, and is, to our knowledge, the second clean data, clean label backdoor attack (after Shumailov et al. (2021)). To summarise, we make the following contributions in this paper:
We present three new backdoor attacks that can be inserted into model's training pipeline through a variety of augmentation techniques. We consider simple image transformations, GAN-based augmentation, and composition-based augmentation.
We build on previous gradient manipulation attacks by using AugMix in place of reordering allow us to manipulate gradients more efficiently through the use of gradient descent. This attack demonstrates that it is possible to perform clean data clean label backdoor attacks using data augmentation, and outperforms Shumailov et al. (202T) significantly.
We evaluate these attacks on variety of common computer vision benchmarks, finding that an attacker is able to introduce backdoor into an arbitrary model using a range of augmentation techniques.
2 RELATED WORK
Backdoor attacks Gu et al. (017)) first used a modified dataset to insert backdoors during train- ing, producing models that make correct predictions clean data, but have new functionality when a specific trigger feature is present. Improvements to this process have since been made to cre- ate attacks that assume stronger threat models. [a al. (2021) demonstrated that backdoors can remain dormant until deployment, where the backdoor is activated by weight quantisation, while Shumailov al. (2021) manipulated the order of data within batch shape gradients that sim- ulate backdoored dataset using clean data. Chen et al. (2017) first investigated triggers that are difficult for humans to identify. Attacks that insert backdoors without modifying a dataset were also demonstrated, for example by inserting malicious components directly into the model's ' architecture (Bober- Irizar et al.) 2022), or by perturbing the model's weights after training (Dumford & Scheirer 2018).
Many of these techniques assume direct access to either the model itself or its training set. Methods that use preprocessing to indirectly insert backdoors have been shown to be effective mechanism to insert backdoors into machine learning pipelines. Quiring et al. (020) and Gao et al. (202I) discuss inserting backdoors using image scaling by adding additional perturbations to the scaling procedure. Our attacks are similarly inserted into augmentation functions, but insert their backdoors using the random parameters of the augmentation rather than by adding additional perturbations to the images in order to stay more discrete. Our attacks each focus on a different class of data aug- mentation function, building on the work from Wu et al. (2022). who investigate only the rotation transformation. Here we consider the more general threat of adversarial augmentation as mecha- nism for inserting backdoors into the training pipeline while also remaining covert by inserting the backdoor through the augmentations random parameters as opposed to direct dataset modification.
Augmentation Image data augmentation has been shown be effective at improving model gen- eralisation. Simple data augmentation strategies such as flipping. translation (He et al.) 2016 Krizhevsky et al., 2012). scaling, and rotation (Wan et al.] 2013) are commonly used improve model accuracy in image classification tasks, practically teaching invariance through semantically- meaningful transformations (lyle et al.) 2020). More complex augmentation methods based on generative deep learning (Antoniou et al. [0177] Zhu et al.] (017)) are now common they have demonstrated strong performance on tasks where class-invariant transforms are non-trivial and are hard to define for human.
Rather than encoding a direct invariance, Cutout Vries & Taylor 2017 removes a random por- tion of each image, while mixing techniques (Yun et al. 2019; Zhang et al., 2018) mix two random images into one image with a combined label. AugMix (Hendrycks et al. (2200) uses random com- positions of simpler transforms to provide more possible augmentations. AugMax (Wang et al.
2

---

Under review as a conference paper at ICLR 2023
2021 uses gradient descent to tune the parameters of the AugMix augmentation to increase the hardness"" of the data the training set. Our AugMix based backdoor performs a similar optimi- sation procedure, but with the goal of inserting backdoor into the model. AutoAugment (Cubuk et al. 2018) tunes compositions of transforms to maximise classifier performance using reinforce- ment learning. We provide overview of different types of augmentations and how they relate to each other in Section 3.2
3 METHODOLOGY
3.1 THREAT MODEL
Our threat model assumes the attacker is limited to the capabilities of a standard augmentation routine. Specifically, our attacker only assumes access individual datapoints during training, without the ability to observe the model. Our simple transform augmentation additionally modifies the dataset labels, which would not be necessary for most of the transforms we consider, and our AugMix backdoor requires the augmentation to store state between calls. However, in practice these would not be major limitations if, for example, the augmentation is implemented as a wrapper around dataset object, which is the most popular implementation in today's 's machine learning frameworks Paszke et al. 2017).
Our GAN and AugMix based backdoors can both support arbitrary triggers. However, our simple transform backdoor requires the augmentation transform to be used as the trigger. Gu et al. (2017) describe applying a pattern trigger to a traffic sign, which presents an issue when using transforms that could not be physically applied to sign as the trigger, such as colour inversion. However, in many cases, these attacks can be launched in alternative settings, such against image-basel filtering systems (Google Apple). In this case, if the attacker wants to upload an image that should be rejected by the filter, they could apply the transformation which triggers the backdoor, resulting in the filter failing to reject the image.
3.2 OVERVIEW OF DATA AUGMENTATION
A dataset be augmented using any randomly applied transformations that semantically retain an image's 's class after application. We categorise these transformations into three groups, which our three backdoors generally correspond to:
1. Simple image transforms, such rotation, Gaussian blur, colour inversion. These trans- forms are simple to detect, making them perfect to insert as a backdoor trigger.
2. Augmentations that produce new image content, such as GAN-based augmentation, or neural style transfer (Gatys et al.) (2155). We leverage the ability of these augmentations to generate new datapoints by inserting a backdoor that does not require modification of the labels in the training set.
3. Compositions of other augmentations, such AugMix Auto These augmenta- tions have large number of random parameters which we can control to insert a backdoor by gradient shaping i.e. by choosing data to imitate gradient update of choice.
3.3 SIMPLE TRANSFORM ATTACK
typical BadNet backdoor implemented by manipulating a dataset D to capture additional func- tionality in the presence of a trigger T. We define a function F so that if (a, y) D. model M should have the functionality A Txx) = Fly) when trained on the modified dataset. This is achieved by modifying D to contain additional datapoints such (T(x), F(v)). Gu al. (017)) suggest T could add a small pattern to the image, and F(.) =
We propose this setup be modified to have T become an image transformation, such rotation, which can be applied to the dataset in the guise of data augmentation. The backdoor insertion function is shown in Algorithm 1.
3

---

Under review as a conference paper at ICLR 2023
8
6
3
Rotation
Inversion
S
8'
6
8
3
5
8'
6
9
3
Gaussian
4
6
0
5
Figure 2: Examples of images produced by simple augmentation backdoors applied to the MNIST dataset. Labels are shown the bottom and are coloured red indicate they have been modified. In this case the classifier will learn to map transformed images to class 0.
Algorithm 1: Simple transform augmentation backdoor
input: batch B, transform T,, backdoor proportion p
N +1:
for (input x, label y) E B do random() S _p then
T(x);
y' + 0:
else
x' + m;
y y:
end
append (x', y²) N:
end
return N:
3.4 GAN-BASED AUGMENTATION ATTACK
We present our GAN-based backdoor strategy as a modification of the DAGAN framework (An- toniou et al., 2017). Antoniou al. (2177) describe the training process for generator G that produces an image of a given class when provided with real image of that class and random noise vector. In order to insert the backdoor into model trained with our DAGAN, we modify this process. If (ar, y) is from the distribution that our dataset D is sampled from, then the backdoored generator G' trained so that there exists another point in this distribution (x',y/), where either (z), Ri (x', vy),, or (G' (z)) R (T(x', F(y')), where T and F have the same meanings as in Section 3.3. We define our backdoor as:
T(x) = x. - m + tt m)
F()) = 0 y
otherwise
where my E {0, 1} is mask applied to and MxN is pattern that acts as the trigger. When y + 1, the DAGAN is trained as normal. the cases where = 1, G either trained to map + x', or + T(x'). In other words, since our classifier trains G's output with the label of its input's true class, G is trained to produce images with the backdoor trigger from inputs with backdoor's target class for some proportion of the dataset. We create this behaviour by simply adding this functionality into G's training set.
The datapoints for which y = 1 therefore randomly split that some map triggered images with a probability of p. while the rest map to datapoints of class y = 1 with a probability of 1 - We present results using three different values of p in Table
is likely for some features to be unevenly distributed across the split, resulting in the model learn- ing clear boundary between images it will add the backdoor to and images it will keep clean,
4

---

Under review as a conference paper at ICLR 2023
despite the dataset's otherwise contradictory nature. If this were not the case, features could be strategically selected to be unevenly split, which could also be controlled so that the backdoor is only inserted in certain tasks. Alternatively, of the elements from the random noise vector could be used to control this decision.
We show the output of our modified DAGAN in Figure The augmented data contains images with the number zero and the trigger pattern. These will retain the input's original y = label so that the classifier using this augmentation will learn the backdoor. We would like highlight that this attack is clean-label. This means we do not modify the labels of the datapoints.
Input
6
4
3
8
9
/
DAGAN output
6
4 3
8
9
O
|
1 0.0
6 4
3
8
9
1 1
Figure 3: Examples of images produced by the modified DAGAN. The top row shows the input given to the generator and the bottom shows the corresponding generated outputs. The labels are modified, so each vertical pair of images are both given the true label of the top image, shown on the bottom row. This is clean-label backdoor insertion, but the post-augmentation images may be out of the distribution augmented images.
3.5 AUGMIX-BASHE AUGMENTATION ATTACK
The AugMix augmentation method transforms an image in complex manner. It first applies a se- quence of simple transformations (up to length d) in random manner w times; then, it takes ran- dom convex combination between the original image and the weighted transformation. Hendrycks jet al. (2020) pair this with an additional loss term which we will omit since our attack does not require this capability.
To insert a backdoor using AugMix, we followed the general style of the Batch Ordering Backdoor (BOB) described by (Summalov al. (2021). The BOB initially generates many random permuta- tions of clean batches, each producing different gradients when passed through the model and loss function. The permutation X, with the smallest difference in gradient with explicitly backdoored batch X, is selected train
min [VLL(( VoL((X, X
Here, 0 are the parameters, and L(X, 0x) is the loss from applying the classifier batch X using weights from timestep k: Since we don't have access to the classifier, we can train our own surrogate model in parallel, and use the loss ((x, 0&) 2 L(X, 0²) from this. By using a batch that produces similar gradients to a backdoored batch, backdoor can inserted to the model with clean data.
Our contribution is to replace the reordering procedure with augmentation function such as Aug- continuous random parameters and these parameters
Mix. Since each AugMix instance has are fully differentiable, it is possible to minimise the loss with respect to these parameters using gradient descent directly. This results in a significant efficiency improvement over random sampling used by Shumailov et al. [202I).
We therefore have two optimisation loops. The first iterates over each epoch, training the target and surrogate classifiers, while the second performs full optimisation pass on every epoch to optimise the AugMix weights for our malicious batch (red loop in Figure 6). Once these parameters have been found, can perform the AugMix augmentation normally (green path in Figure 6), substituting the random parameter sampling with our malicious values. this way, the attack is clean label and the post-procsssing images are inside the distribution of augmented images.
5

---

Under review as a conference paper at ICLR 2023
Figure 4: Samples from two batches of data that produce similar gradients in models. The 10 images on right are taken from a batch of a uniformly random image with a specific class, while the images on the left are cleanly labelled images from our dataset that have been passed through malicious AugMix function.
Weight optimisation:
Augmentation:
AugMix weights
Batch
AugMix
BadNet backdoor
Batch
AugMix weights
Augmented batch
Backdoored dataset
AugMix
S. Classifier
Weights
S. Classifier
Predictions
Predictions
Augmented batch
SGD
Loss function
Loss function
Output batch
Loss
Loss
Error
I VeLil
Figure 5: Overview of the AugMix backdoor process. The red cycle indicates the optimisation perform prior to augmentation to insert the backdoor, while the green shows the augmentation.
4 EVALUATION
We evaluate our attacks on common Computer Vision benchmarks. A summary of the datasets we be found in Appendix B. We test the simple transform backdoor the MNIST (LeCun ett al. 2010), CIFAR-10, and CIFAR-100 100 (Krizhevsky & Hinton 2009) datasets; the GAN-based augmentation backdoor on the MNIST, and Omniglot (Lake et 2015) datasets; and the AugMix backdoor on the CIFAR-10 dataset. For each dataset we report the clean accuracy on only clean data and the attack success rate (ASR) on only data with the trigger and backdoor label. For the AugMix backdoor, we also record the error from the clean labels when the trigger is present for more direct comparison with the Batch Order Backdoor. We summarise the details of the networks we use in Appendix C. and the details of our hardware setup can be found in Appendix D.
O
+
8 + 8
+
0
8
8
8 AugMix backdoor
Clean dataset
Simple transf. backdoor
GAN-based backdoor
Figure 6: Overview of the data output from each of our three backdoors.
4.1 SIMPLE TRANSFORM BACKDOOR
Table I presents the results for our standard transform backdoor. For the first four transforms listed, attacks show negligible accuracy losses when compared to our baseline and near 100% ASR, with the exception of the vertical flip transformation, which is more difficult to detect. We addition- ally present attack that uses the CutMix augmentation as the backdoor trigger. We train these

---

Under review as a conference paper at ICLR 2023
Table 1: Percentage accuracies of classifiers trained using different backdoored transforms. We trained the classifiers with Adam optimiser using 3 = (0.9, 0.999) and Cosine Annealing sched- uler for 300 epochs. For MNIST, we trained with a batch size of 4069, and initial learning rate of 2 x 10 while for CIFAR-10 and CIFAR-100, we used a batch size of 128, and initial learning rate
of 5 x 10-4. We also augmented the CIFAR-10 10 and CIFAR- 100 datasets with random horizontal
flips and translations. We report the accuracy differential in the A heading.
MNIST
CIFARIO
CIFAR100
Attack
Clean (%)
ASR (%)
Clean (S)
A
ASR (%)
Cleun (5)
A
ASR (%)
Baseline
None
99.25
0.00
9.84
94.43
0.00
10.08
78.13
0.00
2.33
Geometric
Vertical flip
98.76
-0.49
98.51
92.46
-1.97
98.73
74.97
-3.16
91.94
Rotate 45" clockwise
99.15
-0.10
99.97
94.66
+0.23
100.00
77.45
-0.68
100.00
Colour
Invert
99.27
+0.02
100.00
94.05
-0.38
98.96
77.54
-0.59
95.91
Kernel
Gaussian blue
99.22
-0.03
100.00
94.37
-0.06
100.00
77.45
0.68
100.00
Image mixing
CutMix with class
98.83
80.78
94.43
99.34
77.44
0.69
99.33
CutMix with class not
98.69
-0.56
84.16
94.56
+0.13
99.48
77.49
-0.64
99.23
backdoors to map triggered images to class 0, first mixing the target image with an image of class O the trigger, and then with image of another class. These attacks perform or only slightly below our baseline accuracy.
Our simple transform attacks demonstrate clean accuracy and ASR similar that of the BadNet attack, while offering an improved mechanism for inserting the attack into the machine learning pipeline. Our attack also brings improvements in detectability and prevention over BadNet. For example, data augmentation has been suggested to be an effective defence against BadNets (Borgnia al. 202I), however, since any other transform applied after our malicious augmentation would not remove our original transformation, this is likely to be less effective against out attack.
Furthermore, while BadNet attacks are detectable in a dataset at any point, our attacks are only present after augmentation is applied and are not overtly malicious since our trigger is genuine semantics- praettics-peeeevingggg transform. Possible defences for our attack could be to manually inspect the code of external augmentation libraries, or to manually check the labels of datapoints in the aug- mented dataset. However, this would be less effective against our CutMix attack as the original CutMix augmentation function modifies image labels well. Overall we find that:
An attacker can introduce backdoor into model using only simple augmentations.
Backdoors that use simple augmentation transform the trigger are capable of having comparable accuracy to more common triggers such as the pattern trigger used by Gu et al. (017))
4.2 GAN-BASED AUGMENTATION BACKDOOR
Our GAN-based backdoor presents an improvement over the limitations of the simple transform attack by (i) requiring no modification of the dataset labels (it is a clean-label attack) and (ii) hiding the backdoor within the generator's weights, making the backdoor undetectable by inspection of its code. The backdoor could still be detected by inspecting the images it produces, but the generator likely to produce images that are passed directly to the model, making manual inspection unlikely unless the user already suspicious of the backdoor.
This backdoor presents trade-off between detectability and accuracy. Table 2 shows our results for the GAN-based backdoor. Both datasets see larger drop in clean accuracy compared to our simple transform backdoor. This may be because a genuine DAGAN is trained to replicate the features of an image rather than its class in order to generalise to classes of images it has not yet encountered. However, by inserting the backdoor for specific class require the DAGAN to also be aware of the class of image presented to it. This pushes the DAGAN to be able to (i) detect the class of an image and (ii) generate new image of specific class from scratch, which is more
7

---

Under review as a conference paper at ICLR 2023
Table 2: Percentage accuracies of classifiers trained our modified DAGAN generator. p is the trigger proportion. We trained the classifiers with Adam optimiser using 3 = (0.9, 0.999) and learning rate of 1 10 for 300 epochs. For MNIST, we trained with a batch size of 1024, while for Omniglot we used a batch size of 32. For both datasets, the DAGAN was trained with Adam optimiser using 10-4 learning rate and _ (0, 0.9) for 75 epochs. We trained the generator once every 5 iterations of the critic, and used batch size of 256 for MNIST and 32 for Omniglot.
MNIST
Omniglot
Attack None
Clean acc. (%)
A
ASR (%)
Clean acc. (%)
A
ASR (%)
99.25
0.00
0.00
84.14
0.00
0.00
0.25
75.91
-23.34
38.60
53.10
-31.04
73.33
GAN aug
0.5
83,30
-15.95
99.65
29.66
-54.48
53.33
0.75
60.33
-38.92
85.12
26.21
-57.93
100.00
difficult task. Our GAN-based backdoor may therefore benefit from further experimentation with other GAN-based augmentation techniques, such as BAGAN (Mariani et al.) 2018).
For the MNIST dataset, counter- intuter-i observe that the clean accuracy of the 25% trigger proportion (p = 0.25) and the ASR of the 75% trigger proportion (p = 0.75) are inferior to the accuracies of the 50% proportion (p- = 0.5). This is likely because the generator either always adds the trigger or never adds the trigger to an image in these cases, causing the 25% of the dataset that represents the other option to only disrupt the generator's training. Overall, we find that:
An attacker can introduce backdoor into model using GAN-based augmentation.
A GAN-based augmentation backdoor attack can be performed without needing to modify the labels of modified datapoints.
4.3 AUGMIX BACKDOOR
Gradient reconstruction fidelity
0.66
AugMix Reandering
0.64
Normatedd Ssed enerr
0.60
0.58
25
50
75
1.00
125
150
175
200
Itorsfore
Figure 7: Comparison between proposed AugMix backdoor and the previous Batch Ordering Backdoor (Shumailov et al. 2021). The graph shows the averaged reconstruction error over 200 iterations of our AugMix backdoor alongside the error from the Batch Ordering Backdoor. We averaged the errors over 95 sequential batches, trained with the same parameters as for the bottom row of Table 3 This indicates that our use of gradient descent optimise the AugMix parameters allows for improved gradient reconstruction after 200 iterations.
Our AugMix backdoor improves over the previous Batch Order Backdoor in two ways: by pro- viding a mechanism to insert the backdoor into the training pipeline and (ii) by enabling an improved optimisation technique for the gradient shaping process. In this section, we investigate the effect of this second improvement, along with the overall performance of the attack. This final attack is clean- label and the post-augmmn sustaugmmettiionn images are also in-distribution, meaning they could be produced by the standard AugMix augmentation pipeline.
Figure 7 shows the еттог between our target gradients from an overtly backdoored dataset and our maliciously AugMixed batch. It is clear that proposed technique allows for improved gradient reconstruction fidelity. We were unable to achieve significant error improvement when using random sampling with our AugMix backdoor, which may be due the sampling's 's inability effectively
8

---

Under review as a conference paper at ICLR 2023
Table 3: Percentage accuracies of classifiers trained CIFARIO using our backdoored AugMix function. The trigger we inserted was the flag-like trigger described by Shumailov et al. 2021))
We performed 200 iterations with Adam optimiser using 3 (0.99, 0.999) and 10 learning rate to find the AugMix parameters. Following the setup described by Shumailov et al. (2021), we initially trained each classifier for 10 clean epochs, followed by 10 adversarially Aug batches.
We used ResNet50 both the target model and surrogate, trained with Adam optimiser using
= (0.99, 0.999) and 10 learning rate.
CIFAR10
Attack
Batch size
Clean acc. (%)
ASR (%)
Error w. trigger
32
84.07
0.00
13.61
27.90
None
64
83.96
0.00
12.94
31.16
128
83.83
0.00
10.62
31.90
32
79.73
-4.34
84.73
84.19
AugMix
64
79.53
-4.43
89.88
85.75
128
79.10
-4.73
95.77
88.52
explore the larger parameter space. The AugMix function's larger parameter space may also corre- spond to wider set of possible gradient updates. This improved error is therefore likely due to combination of the AugMix function's 's improved lower bound on gradient reconstruction error and our use of gradient descent to more efficiently approach this lower bound.
Table presents the results of our AugMix attack. We develop our attack on the codebase from Shumailov al. (2021) make fair comparison and achieve similar baseline accuracy to them. Our backdoor is able to achieve 95.77% ASR. This is a 5.2% increase accuracy over the best result achieved by the previous Batch Order Backdoor method. Our results indicate that the attack most effective on larger batch sizes, which differs from the ordering method, because our attack is able to take advantage of the larger number of parameters effectively. We performed all of our tests with an AugMix width of 20 as we found that widening past this made the search much less efficient.
Unlike our GAN-based backdoor, our AugMix backdoor produces clean images and labels to insert a backdoor with similar properties to BadNet. Our attack is therefore difficult directly detect. However, despite the improved search procedure, our optimisation process takes noticeable amount of time, and the backdoor causes a drop in accuracy. Unlike the GAN-based backdoor, it would be possible to detect this backdoor by careful inspection of the source code. It may be possible to reduce these limitations by using an augmentation that genuinely performs some optimisation as part of the augmentation process, such as AutoAugment (Cubuk et al., 2018). Overall, we find that:
An attacker can introduce a backdoor into a model using only clean data that has been passed through the AugMix augmentation function.
We can improve the reconstruction fidelity of gradient shaping techniques by using more efficient optimisation process such as gradient descent.
CONCLUSION
In this paper, we present three new attacks for inserting backdoors using data augmentation. We present attacks that insert backdoors using simple image transforms. GAN-based augmentation, and composition- sansse augmentation. All three of our proposed backdoors hide their modifications to the dataset within genuine transformations, making them difficult to detect. Our GAN-based attack builds on the simple transform backdoor by encoding the backdoor into the generator's s weights, thereby hiding the backdoor from manual inspection of its implementation, while our AugMix attack produces data with clean labels, rendering manual inspection of the dataset ineffective.
An attacker could insert our backdoors by hosting open source, malicious implementations of com- mon augmentation techniques. When incorporated into a model's training procedure, these aug- mentations will introduce the backdoors the model, despite the original dataset remaining clean. This paper demonstrates that is it necessary to carefully check both the source and the output of any external libraries used to perform data augmentation when training machine learning models.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
This paper explores backdoor attacks that can be inserted through data augmentation. For critical applications such as self driving cars, backdoors inserted by malicious attackers could have serious consequences. Therefore, in this paper we aim to encourage people to inspect their augmentation functions ensure that any external code is clean.
REPRODUCIBILITY
All hyperparameters used to produce our results are provided under each table or in Appendices B, C, and D. Additionally, our Py PoTch code used to achieve the results for all three backdoors can be found at wttps//gghhtooooo////d////ddduot backdoors.
REFERENCES
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks, 2017.
Apple. Apple protections for children. cessed: 2022-11-18.
Ac-
Mikel Soober-Irizar Ilia Shumailov, Yiren Zhao, Robert Mullins, and Nicolas Papernot. Architectural backdoors in neural networks, 2022.
Eitan Borgnia, Valeriaa Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradeoff. pp. 3855-3859, 06 2021. doi: 10. 199SSSS999 2021 9414462..
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks deep learning systems using data poisoning, 2017.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation policies from data, 2018.
Terrance DeVries and Graham Taylor. Improved regularization of convolutional neural networks with cutout. 08 2017.
Jacob Dumford and Walter Scheirer. Backdooring convolutional neural networks via targeted weight perturbations, 2018.
Yue Gao, Ilia Shumailov, and Kassem Fawaz. Rethinking image-scaling attacks: The interplay between vulnerabilities in machine learning systems, 2021.
Leon Gatys, Alexander S. Ecker, and Matthias Bethge. neural algorithm of artistic style, 2015.
Google. Google safesearch image filter. Accessed: 2022-11-18.
answer/510
Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Model Supply Chain Garg, 2017.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple method improve robustness and uncertainty under data shift. In International Conference on Learning Representations, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2261-2269, 2017. doi: 10.1 109/CVPR.2017.222
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report 0, University of Toronto, Toronto, Ontario, 2009.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 1332--66) 2015, doi: 10. 126/ science.aab3050.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [On- 2, 2010.
line]. Available: http://yann.lecmm.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learn- ing: Training clean models on poisoned data. In A. Beygelzimer, Y. Dauphin, P. Liang, and
Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021.
Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On
the benefits of invariance in neural networks, 2020.
Hua Ma, Huming Qiu, Yansong Gao, Zhi Zhang, Alsharif Abuadbba, Minhui Xue, Anmin Fu, Zhang Jiliang, Said Al-Sarawi, and Derek Abbott. Quantization backdoors to deep learning commercial frameworks, 2021.
Giovanni Mariani, Florian Scheidegger, Roxana Istrate, Costas Bekas, and Cristiano Malossi. Bagan: Data augmentation with balancing gan, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Detioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning, 2017.
Erwin Quiring, David Klein, Daniel Arp. Martin Johns, and Konrad Rieck. Adversarial preprocess- ing: Understanding and preventing Image-Scaling attacks in machine learning. In 29th USENIX Security Symposium (USENIX Security 20), pp. 1363-1380. USENIX Association, August 2020. ISBN 978- 7778---93913--1755
Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat Erdogdu, and Ross J Anderson. Manipulating sgd with data ordering attacks. In Advances in Neural Information Processing Systems, volume 34, pp. 18021-18032. Curran Associates, Inc., 2021.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1058-1066. PMLR, 17-19 Jun 2013.
Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, 2021.
Tong Wu, Tianhao Wang, Vikash Schwag, Saced Mahloujifar, and Prateck Mittal. Just rotate it: Deploying backdoor attacks via rotation transformation, 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, and Chun. Cutmix: Regularization strategy train strong classifiers with localizable features. 2019.
Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. International Conference on Learning Representations, 2022.

---

Under review as a conference paper at ICLR 2023
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In International Conference on Learning Representations, 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, pp. 2242-2251, 10 2017. doi: 10.1109//CCV2211.. 244.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A AUGMIX BACKDOOR ALGORITHM
Algorithm 2:
AugMix backdoor
input: batch B, transforms T, iterations v,, surrogate model M, loss function
w + random samples from Dirichlet(1) in shape (len(B), len(T));
m random samples from Beta(o, a) in shape (len(T");
apply BADNET backdoor B;
4 (U inputs), U U.labels);
Iu + backpropagate gradients from Lu to weights of
for iterations do
< apply AugMix to Bimppuss, using weights wfi), m[i] for B.inputs[i];
L LV .inputs), [(M(iVinpuss, V Vaabels);;
9o < backpropagate gradients from by to weights of M;
E + 119, gull":
JE backpropagate gradients from E' to w and m;
w. m SGD (ww, m), ml,9)); 9E);
end
return V;
B DATASETS
MNIST The MNIST dataset (LeCun et al. 2010) consists of 60000 train images and 10000 test images. Each 28x28 pixel greyscale image displays single digit between and 9 inclusive. The class of the image is the digit it contains.
Omniglot The Omniglot dataset (Lake al. 2015) consists of 1623 classes of handwritten charac- ters from 50 different alphabets, with each class containing 20 samples. We downscale the dataset 28x28 greyscale images and reduce the number of classes 50. We split each class into 15 train images and 5 test images.
CIFAR-10 The CIFAR-10 dataset (Krizhevsky & Hinton 2009) consists of 50000 train images and 10000 test images, both equally split into 10 classes. Each 32x32 pixel colour image displays subject from one of the 10 classes.
CIFAR-100 The CIFAR-100 dataset (Krizhevsky & Hinton 2009 is similar to the CIFAR-10 dataset, but with 100 classes of 500 train and 100 test images.
C MODELS
ResNet We use ResNet-50 classifier for the CIFAR- dataset (He et al. 2016), and the WideRes- Net variant implementation at http:///hhhtu...oommmmeei//////s 100 classifier.
to train our CIFAR-
DenseNet We
2017 https://gihub..co/mmmmn l/dagan_ton torch to train Omniglot classifier.
use the DenseNet (Huang
et
al.
implementation at
CNN We use a CNN with two convolutional layers for our MNIST classifiers. The architecture of classifiers is detailed in Table 4.
13

---

The testing of our GAN and AugMix backdoors was carried out on a hardware system with 4x NVIDIA GeForce GTX 1080 Ti. The simple transform backdoor training carried out NVIDIA T4 GPUs. Under review as a conference paper at ICLR 2023
Table 4: Architecture of the classifier trained on the MNIST dataset
input
filter shape
stride
output
activation
ConvO
(1, 28, 28)
(8, 1, 1, 5)
(8, 24, 24)
ReLU
PoolO
(8, 28, 28)
Max, (2, 2)
(8, 12, 12)
ConvI
(8, 12, 12)
(16, 8, 5,
(16, 8, 8)
ReLU
Pooll
(16, 8, 8)
Max, (2
(16, 4, 4)
Dense0
(16, 4, 4)
(128)
ReLU ReLU
Densel
(128)
(96)
Dense2
(96)
(10)
D HARDWARE SYSTEMS
E BACKDOOR DEFENCE METHODS
Table 5: Results of applying the defences proposed by Li et al. 2021 and Zeng et al. (2022) to backdoored model that has been trained using
notation-based augmentation backdoor with 10% trigger proportion. We used the defence parameters described by Li et al. (202)) and Zeng al. 2022 and the classifier described by Li et all 2021 The defence proposed by Li et al. 2021 is ineffective against our backdoors because they break the assumption that the subset of data containing the backdoor is the same every training iteration. The defence proposed by /Zeng al. (2022) is also ineffective because we do not remove the augmentation function from the "clean" set as we assume the defender does not initially know the augmentation is malicious.
CIFAR10
No Defence
Li al.
(0011)
Zeng et al. 2022
ASR
100.00
100.00
100.00
14


Reviews:
[{"review_id": "hxzwqe4EOs", "reviewer": "Reviewer_Z2bo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to add backdoor by designing some specific \"malicious\" data augmentation methods. Some important parts of the paper is not clear to me. Please see my comments below. ", "summary_of_the_review": "I feel a bit confused after reading the first draft. Please answer my above questions before my final evaluation. ", "strength_and_weaknesses": "I have the following questions:\n\n1. In traditional backdoor attacks, an identical transformation T is applied on both poisoned training and poisoned test samples. This is practical, for example in BadNet, where we can put the square backdoor sticker onto a stop sign. However, in your methods, is the backdoor transformation T also applied on test images? If yes, why is this practical for your cases (where you use rotation/GAN/AugMix/etc)? The attacker can add those augmentation operations during training under your problem setting. But why they are also used in testing? Is it possible for the attacker to rotate/color invert a physical stop sign? \n\n2. I don't understand the visualization in Figure 5. Does the right side shows augmented images after \"backdoored AugMix\" images? If yes, I think this backdoor attack is too obvious. Usually backdoor attacks need to be concealed to bypass human inspection. Also, will these seemingly random noise images after backdoored AugMix lead to an acceptable clean accuracy like traditional backdoor attacks? \nOne related work [1] regarding learning the adversarial AugMix parameters is missing. \n\n[1] AugMax: Adversarial Composition of Random Augmentations for Robust Training. NeurIPS, 2021. \n\n3. Can the proposed new attacks bypass state-of-the-art backdoor defense methods [2,3,4]?\n\n[2] Anti-Backdoor Learning: Training Clean Models on Poisoned Data. NeurIPS, 2021.\n[3] Adversarial Unlearning of Backdoors via Implicit Hypergradient. ICLR, 2022.\n[4] Trap and replace: Defending backdoor attacks by trapping them into an easy-to-replace subnetwork. NeurIPS, 2022. \n\n4. Do the proposed new attacks outperform state-of-the-art attacks (such as [5]) in terms of attack success rate and clean accuracy?\n[5] Rethinking the Backdoor Attacks’ Triggers: A Frequency Perspective. ICCV, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PpXYoZ4u9RA", "reviewer": "Reviewer_hnQ2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces backdoor attacks on neural networks via data augmentation procedures. The authors consider backdoors related to simple image transforms, GAN-based augmentation, and composition-based augmentation. The method is evaluated in these three scenarios, and shows interesting performance.\n", "summary_of_the_review": "Overall, the authors propose three different straightforward methods to exploit augmentations in order to introduce backdoor attacks to the network. The method is evaluated on relatively simple benchmarks, and compared to a minimal set of approaches. Given that backdoor attacks are discussed in the literature since 2017, I would have hoped for a more thorough treatment regarding prior work, and a more extensive evaluation.", "strength_and_weaknesses": "The main strength of the paper is in its motivation for inserting backdoors via augmentation, and the interesting approaches to exploit various augmentations to that end.\n\nOne of the weaknesses of the paper lies in the related work discussing backdoor attacks. Unfortunately, the discussion there is somewhat vague (``However, some of these attacks...''), and makes it difficult the properly position this manuscript in the context of prior work. For instance, the papers by Quiring et al., Gao et al.. and Wu et al. seem extremely related, but their discussion is limited to 2-3 sentences (and evaluation is not comparing to these approaches). Why? What are the novel differences between the proposed method and prior work?\n\nAnother issue is with the correctness of the approach in the setting of simple image transforms. It seems like some of the augmentations (e.g., rotation) may be related to data existing in the train set, no? What happens then in terms of the backdoor? Do we get inconsistent behavior for these examples?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IYy0RsTdmQP", "reviewer": "Reviewer_jPb2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Traditional training-set attacks manipulate model predictions by inserting adversarial instances directly into the training set.  This paper proposes a novel threat model for training-set attacks whereby the adversarial training instances are created during the training process itself via a malicious augmentation scheme.  \n\nThe authors propose three novel augmentation-based attacks.  The simplest attack is non-clean label and has a substantial adversarial trigger. The second attack is clean-label, with an adversarial trigger that remains perceptible. The third attack is clean-label and close to the true data manifold.", "summary_of_the_review": "While the attack vector used in this paper is novel, the results are foreseeable and expected.  In short, the attack repackages existing attacks into augmentations rather than creating discrete attack instances.  This reduces the paper's overall contributions and novelty -- in my view substantially.  Therefore, I do not see the (marginal) contributions as sufficient for inclusion in ICLR.  \n\nThis paper has merit and is worth publication, albeit at a different venue.", "strength_and_weaknesses": "#### *Strengths*\n* The attack threat vector is -- to the extent of my knowledge -- novel.\n* The first-order motivation of the paper (mostly) holds and is probably an unconsidered threat model in practice.  Many practitioners use off-the-shelf augmentation routines without verifying their exact procedure.\n#### *Weaknesses*\n* The first two attacks proposed by the authors make clearly perceptible perturbations. This restricts their stealthiness.\n  * The *Simple Transform Attack* assumes access to the training instances' labels.  Most augmentation schemes only have access to the feature vector not the label so this attack is especially unrealistic.\n* While the paper's basic motivation is largely true, in the broad scheme, open-source augmentation schemes are not complete black boxes. They often undergo code review and other verification where an augmentation backdoor may be discoverable. This mitigates some of the vulnerabilities since the attack could be mitigated by human inspection of the augmentation implementation even if the inserted augmentations are imperceptible.\n* I would have preferred evaluation on ImageNet rather than simpler CIFAR/MNIST.  Given the simplicity of the idea and its relative obviousness, this is not a critical limitation of the paper but is still a noteworthy omission.\n* The claim that the GAN-based augmentation is only detectable through the weights is somewhat overstated.  Often the code to train the GAN would be open-source not just the model parameters so someone who is security conscious could inspect the training code for vulnerabilities and then train the GAN themselves.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject