Dynamic Backdoor Attacks Against Deep Neural Networks
Authors: Ahmed Salem
Rui Wen
Michael Backes
Shiqing Ma
Yang Zhang
Venue:  (2021)
Keywords: Backdoor attack
Deep Neural Networks security

Abstract: Current Deep Neural Network (DNN) backdooring attacks rely on adding static triggers (with fixed patterns and locations) on model inputs that are prone to detection.  In this paper, we propose the first class of dynamic backdooring techniques: Random Backdoor, Backdoor Generating Network (BaN), and conditional Backdoor Generating Network (c-BaN). Triggers generated by our techniques have random patterns and locations. In particular, BaN and c-BaN based on a novel generative network are the first two schemes that algorithmically generate triggers.  Moreover, c-BaN is the first conditional backdooring technique that given a target label, it can generate a target-specific trigger.  Both BaN and c-BaN are essentially a general framework which renders the adversary the flexibility for further customizing backdoor attacks. We extensively evaluate our techniques on three benchmark datasets and show that our techniques achieve almost perfect attack performance on backdoored data with a negligible utility loss. More importantly, our techniques can bypass state-of-the-art defense mechanisms.

Under review as a conference paper at ICLR 2021
DYNAMIC BACKDOOR ATTACKS AGAINST DEEP NEU- RAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Current Deep Neural Network (DNN) backdooring attacks rely adding static triggers (with fixed patterns and locations) on model inputs that are prone to detec- tion. In this paper, propose the first class of dynamic backdooring techniques: Random Backdoor, Backdoor Generating Network (BaN), and conditional Back- door Generating Network (c-BaN). Triggers generated by our techniques have random patterns and locations. In particular, BaN and c-BaN based on a novel generative network are the first two schemes that algorithmically generate triggers. Moreover, c-BaN is the first conditional backdooring technique that given a target label, it generate target-specific trigger. Both BaN and c-BaN essen- tially a general framework which renders the adversary the flexibility for further customizing backdoor attacks. We extensively evaluate our techniques three benchmark datasets and show that our technigues achieve almost perfect attack performance on backdoored data with negligible utility loss. More importantly. our techniques can bypass state-of mate-of-the-aan defense mechanisms.
INTRODUCTION
Recent research has shown that deep neural network (DNN) models are vulnerable to various secu- rity and privacy attacks (Papernot al., 2016; 2017; Shokri al., 2017; Salem et al., 2019; 2020; Tramer et al., 2016; Oh et al., 2018). One such attack that receives large amount of attention is backdoor, where an adversary trains a DNN model which can intentionally misclassify any input with added trigger (a secret pattern constructed from a set of neighboring pixels) to a specific target label. Backdoor attacks can cause severe security consequences. For instance, an adversary can implant a backdoor in an authentication system grant herself unauthorized access.
Existing backdoor attacks generate static triggers, terms of fixed trigger pattern and location (on the model input). For instance, Figure la shows example of triggers constructed by Bad- Nets (Gu et al., 2017), one popular backdoor attack method, on the CelebA dataset (Liu al., 2015). As see, BadNets in this case uses white square a trigger and always places in the top-left corner of an input. This static nature of triggers has been leveraged to create most of the current defenses against the backdoor attack (Wang et al., 2019; Liu al., 2019a; Gao al., 2019).
Statie backdoor
In this paper, we propose the first class of backdooring techniques against deep neural network models that gen- erate dynamic triggers, in terms trigger pattern and lo- cation. We refer to our techniques as dynamic backdoor attacks. Figure 1b shows example. Dynamic backdoor attacks offer the adversary more flexibility, as they allow triggers to have different patterns and locations. More- over, techniques largely reduce the efficacy of the cur- rent defense mechanisms demonstrated by our empirical evaluation. In addition, extend our techniques work for all labels the backdoored DNN model, while the current backdoor attacks only focus on a single few target labels. This further increases the difficulty of our backdoors being mitigated.
(b) Dynamic backdoor
Figure 1: comparison between static
and dynamic backdoors on CelebA.

---

Under review as a conference paper at ICLR 2021
In total, we propose 3 different dynamic backdoor techniques, i... Random Backdoor, Backdoor Generating Network (BaN), and conditional Backdoor Generating Network (c-BaN). In particular, the latter two attacks algorithmically generate triggers mount backdoor attacks which are first of their kind.
To demonstrate the effectiveness of our proposed techniques, we perform empirical analysis with three DNN model architectures over three benchmark datasets. All of our techniques achieve almost perfect backdoor accuracy, i.e., the accuracy of the backdoored model on the backdoored data is approximately 100%, with a negligible utility loss. Moreover, we show that our techniques can bypass three state-of-the-ar backdoor defense techniques, namely Neural Cleanse (Wang et al,, 2019), ABS (Liu et al., 2019a), and STRIP (Gao et al., 2019). In general, our contributions can be summarized as follows: 1) We broaden the class of backdoor attacks by introducing the dynamic backdoor attacks. 2) We propose both BaN and c-BaN, which are the first algorithmic backdoor paradigm. 3) Our dynamic backdoor attacks achieve strong performance while bypassing the current state-of-the-art backdoor defense techniques.
RELATED WORK
Backdoor Attacks: Gu et al. (Gu et al., 2017) introduce BadNets, the first backdoor attack on DNN models. BadNets uses the MNIST dataset and square-like trigger with fixed location, to show the applicability of the backdoor attacks in the DNN settings. Liu et al. (Liu et al., 2019b) later propose more advanced backdooring technique, namely the Trojan attack. They simplify the threat model of BadNets by eliminating the need for access to the training data used to train the target model. The main difference between these attacks (BadNets and Trojan attacks) and our work is that both attacks only consider static backdoors in terms of triggers' pattern and location. Our work extends the backdoor attacks to consider dynamic patterns and locations of the triggers. We focus on backdoor attacks against image classification models, but backdoor attacks can be extended to other scenarios, such as Federated Learning (Wang et al., 2020), Video Recognition (Zhao et al., 2020), Transfer Learning (Yao et al., 2019), and Natural language processing (NLP) (Chen et al., 2020).
To increase the stealthiness of the backdoor, Saha et al. (Saha et al., 2020) propose to transform the backdoored images into benign-looking ones, which makes them harder to detect. Lie al. (Liu et al., 2020) introduce another approach, namely, the reflection backdoor (Refool), which hides the triggers using mathematical modeling of the physical reflection property. Another line of research focuses on exploring different methods of implementing backdoors into target models. Rakin et al. (Rakin et al., 2020) introduce the Targeted Bit Trojan (TBT) technique, which instead of training the target model, flips some bits in the target models' weights to make it misclassify all the inputs. Tang et al. (Tang et al., 2020) present a different approach, where the adversary appends a small Trojan module (TrojanNet) the target model instead of fully retraining it.
Defenses Against Backdoor Attacks: Wang et al. (Wang et al., 2019) propose Neural Cleanse (NC), a backdoor defense method based on reverse engincering. For each output label, NC tries to generate the smallest trigger, which converts the output of all inputs applied with this trigger to that label. NC then uses anomaly detection to find if any of the generated triggers are actually backdoor or not. Later, Liu et al. (Liu et al., 2019a) propose another model- based defense, namely, ABS. ABS detects if target model contains backdoor or not, by analyzing the behaviour of the target model's inner neurons when introducing different levels of stimulation. Also, Gao et al. (Gao et al., 2019) propose STRIP, backdoor defense method based on manipulating the input, to find out if it is backdoored or not. More concretely, STRIP fuses the input with multiple clean data, one at a time. Then queries the target model with the generated inputs, and calculate the entropy of the output labels. Backdoored inputs tend to have lower entropy than the clean ones.
Besides the above, there are multiple other types of attacks against deep neural network models, such adversarial examples (Vorobeychik & Li, 2014; Carlini & Wagner, 2017; Li & Vorobeychik, 2015: Tramèr et al., 2017; Xu et al., 2018), poisoning attack (Jagielski et al., 2018; Suciu et al., 2018; Biggio et al., 2012), and property inference (Ganju et al., 2018; Melis et al., 2019).
2

---

Under review as a conference paper at ICLR 2021
3 DYNAMIC BACKDOORS
3.1 THREAT MODEL.
The dynamic backdoor attack is training time attack, i.c., the adversary is the who trains the backdoored DNN model. To implement our attack, we assume the adversary controls the training of the target model and has access to the training data following previous works (Gu et al., 2017; Yao et al., 2019; Chen et al., 2020). To launch the attack (after publishing the model), the adversary first adds a trigger to the input and then uses either digitally, where the adversary digitally adds the trigger to the image, or physically, where the adversary prints the trigger and places on the image. This added trigger makes the backdoored model misclassify the input to the target label.
query the backdoored model. This can happen
3.2 RANDOM BACKDOOR
We start by presenting our Random Backdoor technique. Abstractly, the Random Backdoor tech- nique constructs triggers by sampling them from a uniform distribution, and adding them to the inputs random locations. We first introduce using Random Backdoor implement dynamic backdoor for single target label, then we generalize it to multiple target labels.
Single Target Label:
this setting, we construct set of triggers (T) and set of possible locations (K), such that for any trigger sampled from T and added to any input at random location sampled from IC, the model will output the specified target label. More formally, for any location K; E K., any trigger 4 E T. and any input the set of triggers, IC is the set of locations, .Mad is the backdoored model, and A is the backdoor adding function (it adds the trigger t, the input 2;; the location K,).
X: Mba(A((;; (i,)) = f, where l is the target label, T
To implement such a backdoor, an adversary needs to first select her desired trigger locations and create the set of possible locations K. Then, she uses both clean and backdoored data to update the model for each epoch similar to the BadNets technique but with two differences. First, instead input, of using a fixed trigger for all inputs, each time the adversary wants to add trigger she samples a new trigger from a uniform distribution, i.e., t ~ U(0, 1). Here, the set of possible triggers T contains the full range of all possible values for the triggers, since the trigger is randomly sampled from a uniform distribution. Second, instead of placing the trigger in a fixed location, she places it at random location K, sampled from the predefined set of locations, i.e., K. Note that Random Backdoor is not only limited to uniform distribution, other distributions, such as Gaussian distribution, can be used as well.
Next, we consider the more complex case of having multiple target labels. Without loss of generality, we consider implementing a backdoor for each label in the dataset. This means that for any label & E L. there exists trigger / which when added to the input at location k. will make the model Mia output f,, More formally, Vl, E C 3 L, is Mba(A(x, L,,x), x)) = l. To achieve the dynamic backdoor behaviour, each target label should have set of possible triggers and a set possible locations.
Multiple Target Labels:
We generalize the Random Backdoor technique by dividing the set of possible locations K. into disjoint subsets for each target label, while keeping the trigger construction method the same in the single tar- get label case, i.e., the triggers are still sampled from a uniform distri- bution. For instance, for the target label l, we sample a set of possible locations KG, where IC; is subset of IC (K; C KC). The adversary can construct the disjoint sets two steps. First, the adversary selects all possible triggers locations and constructs the set K.. Second, for each target label f, she constructs the set of possible locations for this label K, by sampling the set K.. Then, she removes the sampled locations from the set X. We propose a simple algorithm to assign the locations for the different target labels. Concretely, we uniformly split the image into non-intersecting regions, and assign a region for each target label, in which the triggers' locations can move vertically. Figure shows an example our location setting technique for use case with 6 target labels. We stress that this is one way of dividing the location set K. to the different
4
5
6
Figure 2: An illustration
of our location setting tech- nique for 6 target labels.
3

---

Under review as a conference paper at ICLR 2021
target labels. However, adversary can choose a different way of splitting the locations inside IC to the different target labels. The only requirement the adversary has to fulfill is to avoid assigning location for different target labels. Later, we will show how to overcome this limitation with our more advanced c- B-BaN technique.
3.3 BACKDOOR GENERATING NETWORK (BAN)
Our Random Backdoor technique successfully implements dynamic triggers, however, it offers the adversary limited flexibility as triggers are sampled from a fixed distribution. Moreover, the triggers sampled independently of the target model. In other words, the Random Backdoor technique does not search for the best triggers to implement the backdoor attack. To address these limita- tions, introduce our second technique to implement dynamic backdoors, namely, BaN, the first approach to afgorithmically generate backdoor triggers.
BaN inspired by the state-o--the-a generative model, i.c., GANs (Goodfellow et al., 2014). It trains a generator with the target model as the discriminator to learn the best patterns for a backdoor trigger. After training, BaN can generate trigger (t) for each noise vector (z u(0,, 1)). This trigger is then added to an input using the backdoor adding function A. to create the backdoored input as shown in Figure 3a. Similar to the previous approach (Random Backdoor), the generated triggers are placed at random locations.
Single Target Label: We start with presenting how to implement dynamic backdoor for a single target label using BaN. First, the adversary creates the set K. of the possible locations. She then jointly trains a BaN with the backdoored Mba model as follows:
1. The adversary starts each training epoch by querying the clean data to the backdoored model Mad- Then, she calculates the clean loss Sp. between the ground truth and the output labels. We use the cross-entropy loss for our clean loss.
2. She then generates n noise vectors, where is the batch size.
3. On the input of the n noise vectors, the BaN generates n triggers.
4. The adversary then creates the backdoored data by adding the generated triggers to the clean data using the backdoor adding function A.
5. She then queries the backdoored data to the backdoored model Maa and calculates the backdoor loss (Pbd on the model's output and the target label. Similar to the clean loss, use the cross-entropy loss as our loss function for PPad-
6. Finally, the adversary updates the backdoor model Maa using both the clean and backdoor losses (yec + Pba) and updates the BaN with the backdoor loss (1b2).
Multiple Target Labels:
We now consider a more complex case of building dynamic backdoor for multiple target labels using BaN. To recap, BaN generates general triggers and not label specific triggers. In other words, the same trigger pattern can be used to trigger multiple target labels. Thus similar to Random Backdoor, we depend on the location of the triggers to determine the output label.
We follow the same approach of the Random Backdoor technique to assign different locations for different target labels (Section 3.2), to generalize the BaN technique. More concretely, the adversary starts by creating disjoint sets of locations for all target labels. Next, she follows the same steps as in training the backdoor for a single target label, while repeating from step 2 to 5 (see above) for each target label and adding all their backdoor losses together. More formally, for the multiple target
label case the backdoor loss defined as: E Pbd,, where L' is the set of target labels, and Pbd, is the backdoor loss for target label l;-
3.4 CONDITIONAL BACKDOOR GENERATING NETWORK (C-BAN)
Both Random Backdoor and BaN have the limitation of not having label specific triggers and only depending the trigger location to determine the target label. We now introduce our third and most advanced technique, the e- c-BaN, which overcomes this limitation. With c-BaN, any location k inside K. can be used to trigger any target label. To achieve this location independency, the triggers need to
4

---

Under review as a conference paper at ICLR 2021
7
Uniform Distribution
Datttuttion
7
7
Mad
c-BaN
(A000RRAA1109
(a) BaN
c-BaN
Figure 3: An overview of the BaN and c-BaN techniques.
be label specific. Therefore, we add the target label as additional input to BaN for conditioning to generate target specific triggers.
We construct c-BaN by adding additional input layer to BaN to include the target label as input. Figure 3b represents an illustration of c-BaN. As the figure shows, the noise vector and the target label encoded latent vectors with the same size (to give equal weights for both inputs). These two latent vectors are then concatenated and used as an input to the next layer. Here, we use one-hot encoding to encode the target label. c-BaN is trained similarly to BaN with one exception, that is the adversary does not have to create disjoint sets of locations for all target labels, she can use the complete location set K. for all target labels.
To use c-BaN, the adversary first samples a noise vector and one-hot encodes the label. Then, she inputs both of them c-BaN generate a trigger. The adversary uses the backdoor adding function A to add the trigger to the target input. Finally, she queries the backdoored input to the backdoored model, which will output the target label.
One major advantage of BaN and c-BaN is their flexibility, i.e., they allow the adversary to customize her backdoor by adapting the loss functions. For instance, the adversary can adapt to different defenses against the backdoor attack that can be modeled as deep neural network models. This can be achieved by adding any defense as discriminator into the training BaN or c-BaN. Adding this discriminator will penalize/guide the backdoored model to bypass the modeled defense.
4 EVALUATION
4.1
EXPERIMENTAL SETUP
We utilize three image datasets to evaluate techniques, including MNIST, CelebA, and CIFAR- 10. For the CelebA dataset, we first scale the images to 64 64, and select the top three most balanced attributes, i.e., Heavy Makeup, Mouth Slightly Open, and Smiling. Then, concatenate them into classes to create a multiple label classification task. We use these three datasets since they widely used as benchmark datasets for various security/privacy and computer vision tasks.
The architectures for our target models, BaN, and c-BaN are presented Section A.1. For evaluating the dynamic backdoor attack's performance, define the following two metrics: Backdoor success rate which calculates the backdoored model's accuracy on the backdoored data; Model utility which measures the original functionality of the backdoored model. We quantify the model utility by comparing the accuracy of the backdoored model with the accuracy of a clean model on clean data. Closer accuracies imply a better model utility. Note that for all of our techniques, the backdoor success rate is almost always 100%, thus we mainly focus on model utility. All of our experiments are implemented using Pytorch and our code will be published for reproducibility.
4.2 RANDOM BACKDOOR
Single Target Label: We split each dataset into training and testing datasets. The training datasets are used to train the MNIST and CelebA models from scratch, and fine-tune a pre-trained VGG- 19 model for CIFAR-10. We use the testing dataset backdoored testing dataset, by adding triggers to all samples of the testing dataset. The former is used measure model utility while the latter used measure the attack success rate.
clean testing dataset and construct
5

---

Under review as a conference paper at ICLR 2021
100
100
Cran Model
Gean Model
05
Ball
05
oBaN
Randum Buckdoor
Bolt
Randum Backdoor
09
CITAN-10
CalesA
MNIST
GPA---0
GanibA
MNET
(a) Single target label
(b) Multiple target label
Figure 4: The result of our dynamic backdoor techniques for single and multiple target label on the clean testing dataset.
(a) Random Backdoor
(b) BaN
(c) c-BaN
Figure 6: Visualization result our techniques for the first four labels of the CIFAR- dataset.
As (Figure 4a) shows, Our backdoored models achieve the same performance clean mod- els for the MNIST and CelebA datasets. For CIFAR-10, there is slight drop perfor- mance (about 2%). This shows that our Random Backdoor can implement a perfectly function- ing backdoor, i.., the backdoor success rate Mad is 100%, with a negligible utility loss. Figure 5a shows some qualitative examples of backdoored data generated by Random Backdoor. As expected, the triggers look distinctly different and are located at different locations.
7 2
Multiple Target Labels: We use a similar evaluation setting to the single target label case with exception. To evaluate the attack success rate with multiple target labels, we construct backdoored testing dataset for each target label by generating and adding triggers to the clean testing dataset. In other words, use all images in the testing dataset evaluate all possible labels. Even in this case, we still achieve nearly perfect attack success rate.
(a) Random Backdoor
7.4
5.
\. 2
(b) BaN
7 2 3
5,
2,
(c) BaN with higher randomness
Visualisation of the dynamic behaviour for our tech- niques for a single target.
Figure
Figure 4b show the model utility in this case. Using our Ran- dom Backdoor technique, we are able to train backdoored models that achieve similar performance as the clean models for all datasets. For instance, for the CIFAR-I6 -10 dataset, our Random Backdoor technique achieves 92% accuracy, which is very similar to the accuracy of the clean model (92.4%). For the CelebA dataset, the Random Back- door technique achieves slightly (about 2%) better performance than the clean model. We believe this is due to the regularization effect of the Random Backdoor technique. Finally, for the MNIST dataset, both models achieve similar performance with just % difference between the clean model (99%) and the backdoored one (98%). Figure 6a further shows some examples of backdoored images by Random Backdoor in the multi target label setting. Moreover, we further visualize in Figure 7a the dynamic behavior of the triggers generated by our Random Backdoor technique. Without loss of generality, we generate triggers for the target label 5 (plane) and add them randomly sampled CIFAR- 10 images. As Figure 7a shows, the generated triggers have different patterns and locations for the same target label, which achieves our desired dynamic behavior.
4.3 BACKDOOR GENERATING NETWORK (BAN)
Next, evaluate BaN technique. We follow the same settings for the Random Backdoor technique, except with respect to how the triggers are generated. train our BaN model and generate the triggers mentioned in Section 3.3.

---

Under review as a conference paper at ICLR 2021
Single Target Label: Similar the Random Backdoor, the BaN technique achieves perfect back- door success rate with a negligible utility loss. As shown in Figure 4a, our BaN backdoored models achieve 99%, 92.4% and 70% accuracy the MNIST, CIFAR-I0, and CelebA, respectively, which is the same performance of the clean models.
We visualize the BaN generated triggers using the MNIST dataset in Figure 5b. The generated triggers look very similar as shown in Figure 5b. This behaviour is expected as the MNIST dataset simple, and the BaN technique does not have any explicit loss enforce the network generate different triggers. However, to show the flexibility of our approach, we increase the randomness of the BaN network by simply adding one more dropout layer after the last layer, to avoid the overfitting of the BaN model to unique pattern. We show the results of the BaN model with higher randomness in Figure Sc. The resulting model still achieves the same performance, i.e., 99% model utility and 100% backdoor success rate, but the figure shows the triggers look significantly different. This again shows that our framework can easily adapt to the requirements of adversary.
Multiple Target Labels: Our BaN backdoored models are able achieve almost the same accuracy as the clean model for all datasets, as shown in Figure 4b. For instance, for the CIFAR-I0 dataset, our BaN achieves 92.1 1% accuracy, which is only 0.3% less than the performance of the clean model (92.4%)). Similar to the Random Backdoor backdoored models, our BaN backdoored models achieve marginally better performance for the CelebA dataset. More concretely, our BaN backdoored models trained for the CelebA dataset achieve about 2% better performance than the clean model. We also believe this improvement is due to the regularization effect of the BaN technique. Finally, for the MNIST dataset, our BaN backdoored models achieve strong performance on the clean testing dataset (98%), which just 1% lower than the performance of the clean models (99%).
Similar to the Random Backdoor, we visualize the results of the BaN backdoored models with two figures. The first (Figure 6b) shows the different triggers for the different target labels on the same CIFAR-10 10 image, and the second (Figure 7b) shows the different triggers for the same target label (plane) on randomly sampled CIFAR-10 images. As both figures show, the BaN generated triggers achieve dynamic behaviour in both the location and patterns.
4.4 CONDITIONAL BACKDOOR GENERATING NETWORK (C-BAN)
Next, we evaluate our c-BaN technique. For the single target label case, the c-BaN technique the same as the BaN technique. Thus, we only consider the multiple target labels case here. We follow a similar setup as introduced in Section 4.3, with the exception of following Section 3.4 to train and evaluate Mad. For the set of possible locations K., set it to 4.
Our evaluation shows that c-BaN is able to build models with good model utility (see Figure 4b). For instance, for the CIFAR-10 and MNIST datasets, our c-BaN achieves 92%, and 99% accuracy which is very similar to the accuracy of the clean models, i..., 92.4% and 99%, respectively.
We compare the dynamic behaviour of our three different techniques using two different figures. First, Figure shows the visualization of backdooring single image the first four labels of the CIFAR-10 dataset. As the figure shows, the Random Backdoor Figure 6a has the most random patterns, which is expected as they are sampled from uniform distri- bution. The figure also shows the different triggers' patterns and locations used for the different techniques.
(a) Random Backdoor
Second, Figure 7 visualizes the different triggers' patterns and locations for the same target label. As the figure shows, the Random Backdoor (Figure 7a) and BaN (Figure 7b) gener- ated triggers can move vertically, however, they have fixed position horizontally as mentioned in Section 3.2. The c-BaN (Figure 7c) triggers also show different locations. However, the locations of these triggers are more distant and can be shared for different target labels, unlike the other two tech- niques. Furthermore, the figure shows that most of our trig- gers have different patterns for techniques for the same
(b) BaN
(c) c-BaN
Figure 7: Visualisation of triggers generated for the same target.
7

---

Under review as a conference paper at ICLR 2021
(a) Clean images
(b) Random Backdoor
(c) BaN
(d) c-BaN
Figure 8: Visualization of attention maps for all our techniques using the Grad-CAM technique.
target label, which achieves our targeted dynamic behavior concerning the patterns and locations of the triggers.
Finally, we compare the attention of the backdoored models on both clean and backdoored inputs. We use the Gradient-weighted Class Activation Mapping (Grad-CAM) technique (Selvaraju et al., 2017) to compute the attention maps for our backdoored models. These maps show the most influ- ential parts of the input that resulted in the model's output. Figure depicts the results of our three different techniques. As expected all backdoored models mainly focus on the triggers in backdoored inputs and the main objects in the clean ones.
4.5 STATE-OF-T OH-THE-ART DEFENSES
We now evaluate our attacks against the current state-of-the-art backdoor defenses. Backdoor de- fenses be classified into two categories: Data-based defenses which detects if the input is back- doored, and Model-based defenses which detects if a model is backdoored.
Model-based Defense: We first evaluate all of techniques in the multiple target label against two of the current state-of-the-art model-based defenses, namely, Neural Cleanse (Wang al., 2019) and ABS (Liu et al., 2019a). We start by evaluating the ABS defense. Here, we focus on the CIFAR-10 dataset since it is the only supported dataset by the published defense model. As expected, running the ABS model against our dynamic backdoored ones does not result detecting any backdoor for all of our models. For Neural Cleanse, we use all three datasets to evaluate our techniques against it. Similar to ABS, all of our models are predicted to be clean models. Moreover, in multiple cases, our models had lower anomaly index (the lower the better) than the clean model.
We believe that both defenses fail to detect our backdoors for two reasons. First, we break one of their main assumptions, i.e., that the triggers are static in terms of location and pattern. Second, we implement a backdoor for all possible labels, which makes the detection a more challenging task.
Next, we evaluate our three at- tacks against the state-of-the-art data-based defense, namely STRIP (Gao et al., 2019). We use all three datasets to evaluate the c-BaN models against this de- fense. First, scale the trigger patterns by half while training the backdoored models, to make them more sus- ceptible to changes. Second, for the MNIST dataset, we move the possible locations to the middle of the im- age to overlap with the image content, since the value of MNIST images at the corners are always O. All trained scaled backdoored models achieve similar performance to the non-scaled backdoored models. Our backdoored models successfully flatten the distribution of entropy for the backdoored data, for subset of target labels. This subset of target labels makes picking threshold to identify backdoored from clean data impossible without increasing the false positive rate, i.c., various clean images will detected as backdoored ones. We visualize the entropy of our best performing labels against STRIP in Figure 9.
Data-based Defense:
Zeen
2.0
Cleas
23
Clean
20
3.5
3D
1.3
BD
3.0
1.9
555
0.
(a) CIFAR-10 (b) MNIST
(c) CelebA
Figure 9: The histogram of the entropy of the backdoored vs clean input.
These results against both data and model based defenses show the effectiveness our dynamic backdoor attacks, and opens the door for designing backdoor detection systems that work against both static and dynamic backdoors. We propose data-based defense against the dynamic backdoor attack, however, due space restrictions, present it in Section A.3.
8

---

Under review as a conference paper at ICLR 2021
4.6 EVALUATING DIFFERENT HYPERPARAMETERS
We now evaluate the effect of different hyperparameters for our dynamic backdooring techniques. Due to space restrictions, we present here a summary of our findings and leave all the details in Section A.2. We start by evaluating the percentage of the backdoored data needed to implement a dynamic backdoor into the model. Our results show that using 30% is enough to get a perfectly working dynamic backdoor. Second, we evaluate the effect of increasing the size of the location set K. Using more than four times the amount of possible locations, i.e., 16, still achieves the same performance. However, when we completely remove the location set IC and consider all possible locations with a sliding window, the performance on both clean and backdoored datasets drops significantly, Finally, we evaluate the size of the trigger and the possibility of making it more transparent. For the trigger size, we find that a trigger with the size 5 is the smallest trigger than can achieve the best results. Similarly for the transparency, 0.5 or higher has no effect on the performance of the backdoor attack. However, when the scale is set below 0.5, the attack success rate starts degrading. To eircumvent this limitation and make the triggers more transparent, we increase the trigger size. Our experiments show that when increasing the trigger size the input images' size, our attacks can be performed with a scale of 0.1 with negligible effect of the performance.
find that setting the transparency scale to
4.7 RELAXING THE THREAT MODEL (TRANSFERABILITY OF THE TRIGGERS)
For our dynamic backdoor attacks, we assume the adversary control the training of the target model. We now relax this assumption by only allowing her to poison the dataset.
First, it is important mention that our Random Backdoor technique does not need to change the training of the target model, i.e., the adversary only needs to poison the training dataset with back- doored images and the corresponding target labels. Second, for both the BaN and c-BaN techniques, the ad adversary can rely on tre-treined BaN and с- B-BaN models instead of training them jointly with the target model. In detail, the adversary uses the pre-trained BaN and c-BaN model to generate mul- tiple triggers and randomly place them to a set of randomly picked images. Then, she poisons the training set with this set of backdoored images and their corresponding target labels.
We use the MNIST dataset for evaluation and follow the same target models' structure as previously introduced in Section 4.3 and Section 4.4. However, to show the flexibility of our techniques, we use data from different distribution pre-train the BaN and c-BaN models. We first use the CIFAR-10 dataset to train backdoored models with the BaN (Section 3.3) and c-BaN (Section 3.4) techniques. Next, we use the pre- pre-trained BaN and c- c-BaN models to generate the backdoored dataset and poison the target dataset as previously mentioned. It is important to mention that the CIFAR-10 based BaN and c-BaN models generate 3-channel triggers, use them poison the MNIST dataset, convert them to I-channel triggers by taking the mean over the different channels. Finally, we use the poisoned dataset train the target model.
As expected, the backdoored models achieve a perfect attack success rate (100%), while keeping the same utility the backdoored models jointly trained with the BaN and c-BaN. This shows the flexibility of our attacks, i.c., the training procedure can be adapted by the adversary depending on her specific application. However, it is important mention that jointly training the models has the advantage of giving the adversary more power, e.g., she can add a customized loss function to the target model while implanting the backdoor.
CONCLUSION
Current backdoor attacks only consider static triggers in terms of patterns and locations, In this work, we propose the first set of dynamic backdoor attack, where the trigger can have multiple patterns and locations. To this end, we propose three different techniques. Our first technique Random Backdoor samples triggers from a uniform distribution and places them at random locations of an input. For the second technique (BaN), we propose novel generative network to construct triggers. Finally, we introduce c-BaN to generate label specific triggers. We evaluate our techniques using three benchmark datasets. The result shows that all our techniques can achieve almost perfect backdoor success rate while preserving the model's utility. Moreover, we show that our techniques successfully bypass ofte-of-the-art defense mechanisms against backdoor attacks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Ma-
chines. In International Conference Machine Learning (ICML). JMLR,, 2012.
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy (S&P), pp. 39-57. IEEE, 2017.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma. and Yang Zhang. BadNL: Backdoor Attacks Against NLP Models. CoRR abs/2000.000443, 2020.
Karan Ganju, Qi Wang, Wei Yang, Carl A. Gunter, and Nikita Borisov. Property Inference At- tacks on Fully Connected Neural Networks using Permutation Invariant Representations. ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 619-633, ACM, 2018.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Ranasinghe, and Surya Nepal. STRIP: A Defence Against Trojan Attacks on Deep Neural Networks. In Annual Computer Security Applications Conference (ACSAC), pp. 113-125. ACM, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Annual Conference Neural Information Processing Systems (NIPS), pp. 2672-2680. NIPS, 2014.
Tianyu Gu, Brendan Dolan Gavitt, and Siddharth Grag. Badnets: Identifying Vulnerabilities in the 2011
Machine Learning Model Supply Chain. CoRR
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina
and Bo Li. Ma- nipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning- In JEEE Symposium on Security and Privacy (S&P), pp. 19-35. IEEE, 2018
Bo Li and Yevgeniy Vorobeychik. Scalable Optimization of Randomized Operational Decisions in Adversarial Classification Settings. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp- 599-607. JMLR, 2015.
Yingqi Liu, Wen-Chur Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. ABS: Scanning Neural Networks for Back-Doors by Artificial Brain Stimulation. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 1265-1282 ACM, 2019a.
Yingqi Liu, Shiqing Ma. Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning Attack on Neural Networks. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2019b.
Yunfci Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks. In European Conference on Computer Vision (ECCV), pp. 182-199. Springer, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Auributes in the Wild. In JEEE International Conference on Computer Vision (ICCV), pp. 3730-3738. IEEE, 2015.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting Unin- tended Feature Leakage in Collaborative Learning. In IEEE Symposium on Security and Privacy (S&P), pp. 497-512. IEEE, 2019.
Seong Joon Oh, Max Augustin, Bernt Schiele, and Mario Fritz. Towards Reverse-Ennineering Black-Box Neural Networks. In International Conference on Learning Representations (ICLR), 2018.
Nicolas Papernot, Patrick D. McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Anan- thram Swami. The Limitations of Deep Learning in Adversarial Settings. In IEEE European Symposium on Security and Privacy (Éuro S&P), pp. 372-387. IEEE, 2016.
Nicolas Papemot, Patrick D. McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Anan- thram Swami. Practical Black- Box Attacks Against Machine Learning. In ACM Asia Conference on Computer and Communications Security (ASIACCS), pp. 506-519. ACM, 2017.
10

---

Under review as a conference paper at ICLR 2021
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. TBT: Targeted Neural Network Attack with Bit Trojan. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13198- 13207. IEEE, 2020.
Aniruddaa Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden Trigger Backdoor At- tacks. In AAAI Conference on Artificial Intelligence (AAAI), pp. 11957-11965. AAAI, 2020.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses Ma- chine Learning Models. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2019.
Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. Updates- Leak: Data Set Inference and Reconstruction Attacks in Online Learning. In USENIX Security Symposium (USENIX Security), pp. 1291-1308. USENIX, 2020.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Basee Localization. In IEEE International Conference on Computer Vision (ICCV), pp. 618-626. IEEE, 2017.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership Inference At- tacks Against Machine Learning Models. In IEEE Symposium on Security and Privacy (S&P), pp. 3-18. IEEE, 2017.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations (ICLR), 2015.
Octavian Suciu, Radu Marginean, Yiğitcan Kaya, Hal Daumé III, and Tudor Dumitraş, When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks. CoRR abs/1803.06975, 2018.
Ruixiang Tang. Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. An Embarrassingly Simple Ap- proach for Trojan Attack in Deep Neural Networks. In ACM Conference on Knowledge Discovery and Data Mining (KDD). ACM, 2020.
Florian Tramèr, Fan Zhang, Ari Juels, Michael Reiter, and Thomas Ristempart. Stealing Machine Learning Models via Prediction APIs. In USENIX Security Symposium (USENIX Security), pp- 601-618. USENIX, 2016
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble Adversarial Training: Attacks and Defenses. In International Conference Learning Representations (ICLR), 2017.
Yevgeniy Vorobeychik and Bo Li. Optimal Randomized Classification in Adversarial Settings. In International Conference on Autonomous Agents and Multi-agent Systems (AAMAS), pp. 485- 492. IFAAMAS/ACM, 2014.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks. In IEEE Symposium on Security and Privacy (S&P), pp. 707-723. IEEE, 2019.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the Tails: Yes, You Really Can Backdoor Federated Learning. Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020.
Weilin Xu, David Evans. and Yanjun Qi. Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2018
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Latent Backdoor Attacks on Deep Neu- ral Networks. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 2041-2055. ACM, 2019.
11

---

Under review as a conference paper at ICLR 2021
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu- Jiang. Clean- Label Backdoor Attacks on Video Recognition Models. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14443-144528, IEEE, 2020.
A APPENDIX
A.1 MODELS ARCHITECTURE
For the target models' architecture, use the VGG-19 (Simonyan & Zisserman, 2015) for the CIFAR-10 10 dataset, and build our own convolution neural networks (CNN) for the CelebA and MNIST datasets. More coneretely, we use 3 convolution layers and fully connected layers for the CelebA CNN. And 2 convolution layers and 2 fully connected layers for the MNIST CNN. Moreover, we use dropout for both the CelebA and MNIST models to avoid overfitting.
For BaN, we use the following architecture:
Backdoor Generating Network (BaN)'s architecture:
- FullyConnected (64)
FullyConnected (128)
Fully FullyConnected (128)
PullyConnected ( (Itl)
Sigmoid -
Here, Ful LyConnected (æ:) denotes fully connected layer with hidden units, ||| denotes the size of the required trigger, and Sigmoid is the Sigmoid function. We adopt ReLU as the activation function for all layers, and apply dropout after all layers except the first and last ones.
For c-BaN, we use the following architecture:
conditional Backdoor Generating Network (c-BaN)'s architecture:
2, E + x FullyConnected (64)
FullyConnected (128)
Ful TylonnCndeteed (128)
FullyConnected (128)
FullyConnected (it:)
Si giomod d -
The first layer consists of two separate fully connected layers, where each one of them takes an independent input, i.e., the first takes the noise vector and the second takes the target label l. The outputs these two layers are then concatenated and used as an input to the next layer (see Section 3.4). Similar to BaN, we adopt ReLU the activation function for all layers and apply dropout after all layers except the first and last one.
A.2 EVALUATING DIFFERENT HYPERPARAMETERS
Here we present the complete details on evaluating the different hyperparameters for our backdoor attacks.
We start by evaluating the percentage of backdoored data needed to implement dynamic backdoor in the model. We use the MNIST dataset and the c-BaN technique to perform the evaluation. First, we construct different training datasets with different percentages of backdoored data. More concretely, we try all proportions from 10% to 50%, with step of 10. In this settings, 10% means that 10% of the data is backdoored, and 90% clean. Our results show that using 30% is already enough get a perfeetly working dynamic backdoor, i.c. the model has a similar performance like a clean model on the clean dataset (99% accuracy), and 100% backdoor success
Proportion of the Backdoored Data:
the backdoored dataset. For any percentage below 30%, the accuracy of
12

---

Under review as a conference paper at ICLR 2021
Beckesored
Triager Size
Figure 10: The result of trying different trigger sizes for the c- c-BaN technique on the MNIST dataset. The figure shows for each trigger size the accuracy on the clean and backdoored testing datasets.
Figure 11: An illustration of the effect of using different transparency scales (from 0 to I with step of 0.25) when adding the trigger. Scale O (the most left image) shows the original input, and scale (the most right image) the original backdoored input without any transparency.
the model on clean data is still the same, however, the performance on the backdoored dataset starts degrading. This demonstrates the ability of the adversary implement dynamic backdoor attack with 30% overhead for each target label, compared to training a clean model.
Number of Locations: Second, we explore the effect of increasing the size of the of possible locations (IC) for the c-BaN technique. We use the CIFAR- 10 dataset to train backdoored model using the e-BaN technique, but with more than double the size of K., i.e., locations. The trained model achieves similar performance on the clean (92%) and backdoored (100%) datasets. We then doubled the size again to have 16 possible locations in K., and the model again achieves the same results both clean and backdoored datasets. We repeat the experiment with the CelebA datasets and achieve similar results, i.e., the performance of the model with a larger set of possible locations is similar the previously reported one. However, when we try completely remove the location set IC and consider all possible locations with a sliding window, the performance on both clean and backdoored datasets drops significantly.
Trigger Size: Next, we evaluate the effect of the trigger size on our c-BaN technique using the MNIST dataset. We train different models with the c-BaN technique, while setting the trigger size from to 6. We define the trigger size to be the width and height of the trigger. For instance, trigger size of 3 means that the trigger 3 pixels.
We calculate the accuracy on the clean and backdoored testing datasets for each trigger size, and show our results in Figure 10. Our results show that the smaller the trigger, the harder it is for the model to implement the backdoor behaviour. Moreover, small triggers confuse the model, which results in reducing the model's utility. As Figure 10 shows, trigger with the size 5 achieves perfect accuracy (100%) on the backdoored testing dataset, while preserving the accuracy on the clean testing dataset (99%).
Transparency of the Triggers: Finally, we evaluate the effect of making the trigger more transpar- ent. More specifically, we change the backdoor adding function A apply a weighted sum, instead of replacing the original input's values. Abstractly, we define the weighted sum of the trigger and the image as: Ebd = s. t +1+++1111 (1- s) x, where is the scale controlling the transparency rate, x the input and t is the trigger. We implement this weighted sum only at the location of the trigger, while maintaining the remaining the input unchanged.
We use the MNIST dataset and c-BaN technique to evaluate the scale from 0 1, with step of 0.25. Figure 11 visualizes the effect of varying the scale when adding trigger to an input.
Our results show that technique and backdoored (100%) testing datasets, when setting the scale to 0.5 or higher. However, when the scale is set below 0.5, the performance starts degrading the backdoored dataset but stays the
achieve the same performance both the clean (99%)
13

---

Under review as a conference paper at ICLR 2021
Figure 12: Visualization of the c-BaN backdoored images when setting the transparency scale to 0.1.
same on the clean dataset. We repeat the same experiments for the CelebA dataset and find similar results.
We believe that the transparency of our triggers can be further increased when using triggers with larger sizes, To this end, we use the CIFAR-10 dataset to repeat the experiments previously men- tioned in this section. However, we set the trigger size to be the size of the image. Our experiments show that in this setting, our dynamic backdoor attack can still achieve a perfect attack success rate (100%) with a negligible drop in utility (0.3%). More concretely, the model's 's accuracy clean data 91.7% compared to the 92% accuracy of the backdoored model trained without any transparency. We visualize set of randomly backdoored samples in Figure 12. As the figure shows, setting the scale to 0.1 makes the triggers hardly visible.
A.3 POSSIBLE DEFENSES
Next, we propose data-based defense against the dynamic backdoor attack. Intuitively, we use denoising mechanism to filter triggers (as they can be considered as anomaliev/dssootionn) out of the backdoored inputs. To this end, we use one of the most common denoising mechanisms, namely autoencoder. It works as follows: First, we train an autoencoder clean data. Then, use this autoencoder to reconstruct/denoise the inputs (by encoding then decoding them). The noise triggers in case expected to be filtered out of the inputs due to main reasons: The overfitting of the autoencoders to clean data and the lossy reconstruction process.
To implement defense, the target model. The autoencoder is expected to remove the trigger from backdoored data, while not significantly changing the clean ones.
the autoencoder denoise all inputs before forwarding them to
To evaluate the efficacy of our proposed defense, we test against the c-BaN technique, using both the MNIST and CIFAR-10 datasets. As expected, the backdoored images not perfectly reconstructed by the autoencoder, i.e., the autoencoder does not fully reconstruct triggers. Our experiments show that in simple datasets like MNIST, our approach can successfully defend against the backdoor attack, with negligible utility loss (less than 1%). However, for more complicated datasets like CIFAR-10, the performance of our defense degrades. This due to the high amount of details which hardens the reconstruction process of complex datasets (for both clean and backdoored inputs). For instance, the accuracy of the target model drops by 4.8% and 25% for the clean and backdoored dataset, respectively.
Another possible defense approach is first calculating the distance between the reconstructed input and the original input, then taking the decision to forward the input or not to the model, based on a predetermined threshold. We plan to explore this approach and other potential methods future work.
14


Reviews:
[{"review_id": "Z6IK5WYI4kF", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary and contributions:\nIn this work, the authors propose the first set of dynamic backdoor attacks, in terms of trigger pattern and location. Three techniques which areRandom Backdoor, Backdoor Generating Network (BaN) and conditional BackdoorGenerating Network (c-BaN) are used to implement dynamic backdoor attacks. The results of empirical evaluation show that the proposed attacks are effective as well as difficult to defend by several defense mechanisms.\n\nStrength (significance and novelty):\n1. They propose the first class of dynamic backdoor attacks. And this work is well motivated. Previous backdoor attacks only use a fixed and static backdoor trigger, which is set to the default assumption of several defense mechanisms. Dynamic backdoors chosen from a set of predefined triggers and locations or generated byGAN, can bypass these defense mechanisms.\n2.  In the empirical evaluation, the proposed attack algorithm is effective varying different datasets and is robust to several defense mechanisms.\n3. In addition to single target label and multiple target label attacks, they also propose an auto-encoder based defense against dynamic backdoor attacks.\n4. The techniques in dynamic backdoor attacks are simple yet effective. And the paper is well-written and easy to follow.\n\nWeakness:\n1. This work only considers white-box backdoor attacks, i.e., the attacker controls the target model and has access to the training data. I suggest the authors should refer Refool [1] and study the transferability of BaN and c-BaN to different data distributions and model architectures.\n2. This work aims to improve backdoor robustness to defense mechanisms. However, theSec 4.5 state-of-the-art defenses is too brief. More experiments and illustrations should be added to this part. Here are some suggestions:\n\n3. Since the motivation of this work is to “break” the assumption in some model-based defenses, it would be interesting to compare the reversed triggers and the original triggers (sample from the dynamic trigger distribution). In addition to Neural Cleanse and ABS, MESA [2] can also be considered for its superior to Neural Cleanse.\n\n4. Add some visualizations to understand dynamic backdoor attacks, e.g. Grad-CAM,  which is widely adopted in backdoor attacks [1, 3].\n\n5. Add a comparison with traditional static backdoor attacks against different defense mechanisms, which would be more convincing.\n\nReference\n1. Liu, et al. Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks.ECCV 2020.\n2. Qiao, et al. Defending Neural Backdoors via Generative Distribution Modeling. NeurIPS2019.\n3.  Xie, et al. DBA: Distributed Backdoor Attacks against Federated Learning. ICLR 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "J0YLVnMZLi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposed a class of methods for dynamic backdoor attack. The main idea is to generate different backdoor patterns and locations in backdoor attack. The threat model is the attacker has full access to the training data and the model training procedure. Both single-target and multi-target class-conditional triggers were explored. Experimental results on three datasets in the 100% poisoning setting verified the effectiveness of the proposed attack.\n\nStrengths:\n1. This paper investigated an important problem for secure deep learning.\n2. The proposed methods were well motivated.\n3. Empirical results indicate that dynamic backdoor triggers are possible.\n\nWeaknesses:\n1. Missing many existing works on backdoor attacks. The related work section only mentioned two backdoor attacks Gu et al., 2017 and Liu et al. 2019b. To my knowledge, there are many existing works in this field.\n2. The discussion of the key properties for backdoor triggers are weak. For example, the injection rate, and the stealthiness of the trigger pattern. Existing works have proposed reflection triggers or even invisible triggers.  Compared to these triggers, the proposed triggers are not stealthy at all. Appearing in different places of the input image even makes the trigger more easily detectable (at least to human checkers).\n3. The proposed methods need to poison at least 30% of the training data, however, many existing methods only poison 10%-20% of the training data. \n4. The 6 training steps described on Page 6 indicate that the proposed methods need to modify the model training process. This is different from a typical backdoor setting where the attacker cannot control the training procedure, which allows the attacker to invade commercial models by only poisoning the data. From this perspective, the setting adopted in this paper is questionable.\n5. The 30% poisoning setting in Section 4.6 is not clear. Was the 30% only used for training the BaN then poisoning 30% data using the BaN model or does it still need the entire training data (but only 30% contain the trigger)? The training process on Page 6 says the BaN and the backdoored model need to be trained simultaneously (step 6).\n6. One more concern, the training of BaN nets needs a certain proportion of the training data. Even 30% is still a lot for large-scale datasets like ImageNet. However, many existing attacks do not need a generator, which makes the attack more applicable and convenient. Can the generators trained on one dataset transferable to other datasets?\n\n---\nComments after the rebuttal:\n\nOverall, the attack settings are still questionable: low stealthiness, high poisoning rate, and intervening the training process. \n\n- The authors clarified that \"Neither our random backdoor technique nor the transferred BaN and c-BaN requires that\", however, in 3.1 Threat Model, it is clearly stated \"The dynamic backdoor attack is a training time attack, i.e., the adversary is the one who trains the backdoored DNN model. To implement our attack, we assume the adversary controls the training of the target model and has access to the training data following previous works (this is actually not true)\", the authors seem misunderstood the different between \"controlling the training of the model\" to data poisoning.\n\nIn terms of the difference to static trigger,\n\n- The authors argued that they are different to static trigger and \"a static trigger cannot be used as a dynamic trigger\", which I don't find it is verified somewhere. Actually the random backdoor proposed in this paper proves that a static trigger when applied to a random location still works very well, even as good as BaN and c-BaN. Whether existing triggers can already achieve a dynamic effect or not is unclear, or verified.\n\nWithout knowing how hard it is to make a static trigger works in a dynamic manner, or where it will completely fail, it is hard to evaluate the contribution and novelty of the proposed attacks. In other words, it seems that a static trigger attached to random locations is good enough to achieve the dynamic purpose, which I don't think is contributive enough.\n\nStealthiness and poisoning rate is another concern, say when you apply the BadNets to random locations, how high the poisoning rate will be to achieve 100% ASR? Higher than 20% or need 100%?\n\nOverall, the lack of thorough comparisons of the trigger properties to existing attacks is the major weekness.\n\nSo my rating will stay the same.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7vMovKOmTZV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper outlines improved backdoor attacks for deep neural networks. Backdoor attacks are training-time attacks whereby an adveresary trains a network in such a way that it functions as a classifier on honestly generated images, but it has the added caveat of also being able to misclassify on images corrupted via a \"trigger\" set by an adversary. This is a grave security concern, and as such there is substantial literature that relates to both systematically training backdoored DNNs, and also detecting when a trained DNN has a backdoor. \n\nAs the authors mention, the existing literature on backdoors typically has a static backdoor where a single portion of an image is altered to create a trigger in a deterministic fashion / location. The authors improve the functinoality of backdooring a DNN by creating a training process where triggers can now appear at random locations in the image and can have random contents. The way they do so is by judiciously and simultaneously training a GAN in conjunction to the DNN that will be corrupted. The GAN uses the DNN under training as a discriminator and ultimately produces corrupted images with triggers that are meant to push the DNN towards having a backdoor through typical training procedures. These GAN-based techniques are key to both algorithms the authors present: BaN (random location triggers) and c-BaN (random location and contents to trigger). \n\nFinally, the authors validate how effective their backdoor techniques are by training on MNIST, CelebA and CIFAR-10 datasets. Their backdoored DNN have their performance measured by how well they misclassify on triggered images and on how little they compormise on state of the art accuracy in non-backdoored models. The authors show that their models have a small sacrifice in terms of accuracy (although their model interestingly enough classifies better on CelebA), and furthermore, their backdoors are robust to existing methods used to catch backdoored DNNs\n\n\nPros:\nI was not as familiar with existing literature on backdoored DNNs until reading this paper. I see how this is a valid concern in the community, and I believe that the authors have provided a better attack with respect to the state of the art. Furthermore, their simulations do a good job of providing evidence of this improvement (in terms of both the visual output of the triggers, and also the performance against software used to find backdoored DNNs)\n\nQuestions / Points to address:\n-As someone who is not as conversant with relevant literature on backdoored DNNs, I wonder how much work has been done on the fact that the triggers used are easily identifiable by a human looking at the picture (in termss of a blatant square in a given location). Though I am also no expert, there is a mature field of Steganography that focuses on mathematical methods of concealing information in images (and other media files). Is there significant overlap in what has been studied with this? Can the DNN architecture used be informed by existing theory in this field?\n-I am also curious about the fact that the backdoored classifierse performed better on CelebA vs. other data sets. I see the authors' point about this possibly being a form of regularisation, but it would be interesting to see the performance of this methodology on more complex datasets that may also exhibit this behaviour. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a new set of trigger based backdoor attacks that use dynamic patterns that make detection harder. These attacks seem to be stronger with regards to state of the art attacks. 

Some weaknesses is the need for full whitebox access of the model. Several key references are missing, and the comparison with other backdoor attacks in unclear.

Moreover, although the authors compare with trigger based backdoors, there are plenty of triggerless backdoors that can be viewed as dynamic, eg the attacks by 
Bagdasaryan et al. http://proceedings.mlr.press/v108/bagdasaryan20a.html and works referencing this paper.

The paper indeed provides an interesting path towards dynamic attacks, but the lack of comparisons with state of the art literature, and also the need for whitebox access/high poisoning rate significantly limit the novelty of this work. 


Decision: Reject