Programmable Neural Network Trojan for Pre-trained Feature Extractor
Authors: Yu Ji
Zinxin Liu
Xing Hu
Peiqi Wang
Youhui Zhang
Venue:  (2020)
Keywords: Neural Network
Trojan
Security

Abstract: Neural network (NN) trojaning attack is an emerging and important attack that can broadly damage the system deployed with NN models.
Different from adversarial attack, it hides malicious functionality in the weight parameters of NN models.
Existing studies have explored NN trojaning attacks in some small datasets for specific domains, with limited numbers of fixed target classes.
In this paper, we propose a more powerful trojaning attack method for large models, which outperforms existing studies in capability, generality, and stealthiness.
First, the attack is programmable that the malicious misclassification target is not fixed and can be generated on demand even after the victim's deployment.
Second, our trojaning attack is not limited in a small domain; one trojaned model on a large-scale dataset can affect applications of different domains that reuses its general features.
Third, our trojan shows no biased behavior for different target classes, which makes it more difficult to defend.

Under review as a conference paper at ICLR 2020
PROGRAMMABLE NEURAL NETWORK TROJAN FOR TRAINED FEATURE EXTRACTOR
Anonymous authors Paper under double-blind review
ABSTRACT
Neural network (NN) trojaning attack is an emerging and important attack that can broadly damage the system deployed with NN models. Different from adversarial attacks, it hides malicious functionality in the weight parameters of NN models. Existing studies have explored NN trojaning attacks in some small datasets for specific domains, with limited numbers of fixed target classes. In this paper, we propose a more powerful trojaning attack method for large models, which outper- forms existing studies in capability, generality, and stealthiness. First, the attack is programmable that the malicious misclassification target is not fixed and can be generated on demand even after the victim's deployment. Second, our trojaning attack not limited in a small domain; one trojaned model on a large-scale dataset can affect applications of different domains that reuses its general features. Third, our trojan shows no biased behavior for different target classes, which makes it more difficult defend.
INTRODUCTION
Neural Network (NN) Trojaning Attack Neural Network Backdoor Injection Attack is an impor- lant attack model that can broadly damage the system based on NN models (Dumford & Scheirer) 2018
Liu al. 2017 Liao et al., 2018 Gu al. 2017). NN trojaning attacks hide malicious functionality inside the weights of an NN model, either by poisoning datasets or performing weight perturbation (Gu et al., 2017). The trojaned NN model predicts correct labels normally for legitimate inputs, and only misclassifies the inputs with trigger patterns to predefined target labels. The NN models are essentially just a set of weight parameters connected with certain network architectures. Their behavior highly depends on the weight parameters, but the meanings are completely implicit. Thus, modifying the weight parameters usually shows no difference to consumers. To note, it is a different attack model from adversarial attacks (Kurakin al. 2016), which craft adversarial inputs to mislead NN models.
The NN trojaning attack is becoming emerging practical and destructive attack model because of the broad usage of pre-trained models. Training a neural network with good features requires not only large number of computing resources but also large-scale datasets. Thus, using pre- trained models is a common practice in developing NN-based applications to reuse expensive well- learned features. Accordingly, there are many open-source pre-trained models available online. They are produced by various companies, open-source communities, or personal maintainers, and consumed by end-users who may use these models directly or reuse part of them for particular task. These pre-trained models benefit the agile deployment and boom the NN technique evolution. However, they also raise security issues since some vicious model promulgators can hide malicious functionalities in the clean model and release them for public use, which can be easily spread. Therefore, important to explore and understand the NN trojaning attacks.
Although the trojaning attack requires attackers to be capable of modifying the weight parameters of the NN model, it does not have to be an entire white-box. terms of the attack scenarios, such trojaning attacks can be classified into two types, outsourced training attack and transfer learning attack. The first assumes that the victims will use the trojaned model directly without any further modification. This kind of attack is completely white-box and most existing studies focus on this assumption (Dumford & Scheirer 2018 [iu et al. (0177: Liao et al. 2018). However, in real cases, victims often employ pre-trained NNs well-nearned feature extractors and further develop their

---

Under review as a conference paper at ICLR 2020
2017). Therefore, the trojan attacks should resist victims' modifications, which is referred to as the transfer learning attack (Gu et al., 2017). One of such studies, BadNet (Gu et al., 2017). has explored the transfer learning attack in some small datasets on traffic signs.
models (Gu et al.
Thus, although existing studies make initial steps that explore the potential effectiveness of trojan in transfer learning, their methodologies are restricted in small domains and validated small datasets. For general feature extractors that are trained on large datasets and are used broadly, the attack is more challenging and the existing trojaning methods cannot be applied to this scenario: The victim tasks are completely unknown to the attackers and the target label that the attackers misleadingly train the trojaned model recognize may even not be involved in the victim task.
For example, one of the official tutorials provided by TensorFlow" introduces the transfer learn- ing scenario for image classifications. They use ImageNet (Deng et al., 2009) pre-trained models as well-learned image feature extractors and retrain the fully connected (FC) layers for new tasks on smaller Flower datasets. The official tutorial provided by MXNe also introduce this scenario that transfer pre-trained VGG16 model for Caltech-256 dataset. In the natural language processing (NLP) field, is also a popular practice to reuse BERT (Devlin et al. (2018) as pre-trained word-level features to solve many different kinds of NLP tasks. These scenarios are more realistic and trojans on those general features will affect a large scope of applications. Thus, it is important to explore the trojaning attack on the general feature extractors.
Another limitation of existing studies is that the trigger patterns of the trojans are usually handcrafted patterns for only one or few target classes. The limited diversity makes the trojans highly correlated with the trigger patterns. Defense methods (Chen et al. 2018 Liu et al.) 2018a Wang et al.) 2019) based on statistic could detect or erase these trojans easily.
In this paper, we propose an NN trojaning attack method that is much more powerful, general, and stealthy. Instead of using static set of handerafted patterns trigger predefined target class, we use dynamic patterns to trigger any intended target class, which makes our trojan attack pro- grammable. We can use target image to describe the target class and generate a trigger pattern based on this image to encode and pass the information of the misclassification target to the trojan.
The dynamic trigger pattern makes our trojan much more powerful and general: Even if the explicit classes used in victim model are not involved in the pre-trained model and unknown to attackers, they can still describe the input they expect the victim model to see with a target image and then generate the corresponding pattern to trigger the malicious behavior. Further, the dynamic trigger also greatly increases the diversity of trigger patterns, which makes it more stealthy.
We demonstrate our attack method under the scenario described in the retraining tutorial from Ten- sorflow and MXNet, which uses pre-trained ImageNet (Deng et al. 2009) models and replaces the FC layers for the Flower dataset and Caltech-256 dataset. We insert a trojan into the ImageNet model and attack the victim model for the two smaller datasets. The trojan remains effective for both cases. Note that, the classes in the Flower dataset and Caltech-256 are not involved in the 1000 classes of ImageNet and attackers have no access to these datasets. The same trojaned model can affect victims using any other dataset.
RELATED WORK
Neural networks show vulnerabilities the crafted adversarial inputs, which referred to as ad- versarial attack (Kurakin et al. 2016). NN trojan is another important attack model which can broadly damage the systems based on NN models. In such an attack model, the NN model intellec- tual property (IP) vendors could be the potential attackers who hide malicious functionalities in the pre-trained NNs (Liu et al., 2017 2018b Wang et al. 2019). These models perform normally with legitimate inputs and can export targeted or untargeted outputs with the trigger inputs.
Previous studies have made some initial steps in the NN trojaning techniques. In most existing studies (Dumford & Scheirer 2018 Liu et al. 2017 Liao et al. 2018), they assume that the victim
HHw to Retrain Image Classifier for New Categories.
https::/www.tenarfflooet
tutorials/ httss/imaeeeeerraiieeeeeeee rainngi
FFne-tune with Pre-srained Models.
master/faq netune. html
2

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison between our work and related work
Capability
Transferability Out-scope target
Dynamic target
Large Dataset
Dumford & Scheirer
2018
X
Liu ct al.
72017)
L1a0 ct al.
2018
Gu et al.
2017
Ours
will adopt the pre-trained NN models directly, which is termed outsourced training attack. However, this situation rarely actually occurs. In practice, users typically fine-tune the FC layers of the pre- trained models to adapt to their working scenarios, which makes the attack more challenge; it is termed as transfer learning attack. Although the most related work, BadNet (Gu et al. [017), has implemented a transfer learning attack, the triggers in their work are based on handcrafted patterns, which are statistically fixed. Therefore, their triggers can only support fixed target classes that are included the pre-trained models. It cannot applied to the scenario we demonstrate in this paper. Further, existing studies only demonstrated high success rate of trojaning attack on small dataset such as MNIST (Dumford & Scheirer 2018 Liao et al.) 2018) Gu et al. 2017 Liu et al. 2017 Wang et al. 2019), face recognition (Dumford & Scheirer 2018 Wang et 2019), traffic sign (Liao al.,
2018 Gu et al. 2017] Chen et al. 2018). and CIFARTO CCen et al. 2018). But people seldom use pre-trained models on these tiny datasets from an untrusted source. We compare our work with related studies in Table [ We support target classes outside the pre-trained models, termed as out- scope target, and the target class not fixed, termed dynamic target. These properties make our attack much more powerful. We also demonstrate the attack on ImageNet.
There are also some initial studies about the defense of the NN trojan. Some detect if the dataset is poisoned (Chen et al. 2018), some detect if the model is poisoned by comparing the decision boundary of different classes (Wang et al.) 2019), and some try to remove the trojan by squeezing the redundencies (Liu et al. (0088)). Most of them just work on trojaning attacks with just one or few fixed target labels; in Section 5.3 we will analyze their effects on the proposed attack model.
THREAT MODEL
Figure T shows a typical flow of transfer learning attack for NN trojans (Gu et al. 2017). For the first explain several terminologies. The start of the flow is pre-trained the clean
ease of understanding,
NN model, denoted as clean model; its task is original task. The network architecture model usually consists of a backend model and a frontend model. The backend model produces general features for a certain domain, which is intended to be reused by victims. The frontend model uses the general features for the underlying tasks and victims will develop their frontend model based on the backend model. The clean model usually comes from public model zoos or produced by attackers. Then, the threat model usually contains the following three phases.
Trojaning Phase. In this phase, attackers can fully access and make modifications to the entire clean model. They usually modify only the backend model to hide the trojan because the frontend model is replaced in later phases. The modified backend is denoted as trojaned backend and the entire model now denoted as trojaned model. The trojaned model has the same network architecture as the clean model. The only difference is the weight parameters in the backend.
Victim Phase. The trojaned model is then distributed online and reused by victims. Victims intend to reuse the well-learned general features from the backend model for their new tasks, denoted as victim task. is typically done by designing new frontend, victim frontend. And the entire model now is denoted as victim model. Note that the victim frontend is unknown to attackers, including the explicit classes involved. On the other hand, although victims can fully access the trojaned model, they are unaware of the explicit method to trigger the malicious functionality.
Trigger Phase. The victim model is then deployed to real applications and the applications may also integrate other components. Now, victims are still capable of accessing the runtime information of their victim model and the system. But for attackers, is a black box now except for the application scenario. Attackers make small modifications to the input trigger the malicious functionality
3

---

Under review as a conference paper at ICLR 2020
Trojaning Phase
Victim Phase
Trigger Phase
Normal
Protrained NN model
Victim
Normal User
Classiffed to the correct class
Frontend model
Frontend model
New frontend
Distribute
Deploy
uestures
Backend model
Troganed backend
Trajaned backend
Trojoned baciend
Misclassified to the target class
Irgjonalgorihm
Existing work
Trigerr pattern Target dlass
Triger generater
Our work
Trojan algorithm
Mitackor
Figure 1: Attack methodology comparison. In existing studies, attackers should decide the trigger pattern and target class pairs in the trojaning phase. In contrast, our method inserts a general trojan in the trojaning phase and decide the target class in the trigger phase.
in the trojaned backend to control the behavior of the system. The modification that can be made highly depends on the scenario the victim task.
In most real applications, the modification is small patch in the input image, denoted as trigger pattern. The clean input image is called source image, and its corresponding label is source class. The source image patched with the trigger pattern denoted as trigger image. The corresponding misclassification target is target class. is described by a target image.
Note that, although attackers may also perform black-box adversarial attack in the trigger phase and also lead to misclassification. The sources of the two threats are completely different. Thus, defending adversarial attacks will reduce the risk of trojaning attacks.
4 METHOD
4.1 ATTACK METHODOLOGY
The major contribution of our work is the new attack methodology, which greatly extends the power of the trojaning attack.
In the workflow of the existing transfer learning attack shown in Figure attackers choose the trig- ger pattern and the corresponding target class in the trojaning phase and then modify the backend model recognize the trigger pattern without affecting its behavior for normal inputs. Then, in the trigger phase, attackers will present the trigger pattern in normal input to trigger the misclassifica- tion as the target class. This attack flow has two major drawbacks.
Fixed targets. The target classes are decided in the trojaning phase. Attackers cannot choose targets on demand in the trigger phase.
In-scope targets. In the trojaning phase, the victim task is completely unknown to the attacker. It difficult to support target classes that are not included in the class set of the original task.
In Table none of the existing studies support out-scope and dynamic target due to these drawbacks.
We propose new attack methodology as shown in Figure T The difference is that in the trojaning phase, we insert more powerful programmable trojan and create corresponding trojan generator. In the trigger phase, we use a target image to indicate the target class. The generator will encode the target image into trigger patter.. It will be presented in the input image and the trojaned backend can decode the trigger pattern and misclassify the input as the target class defined by the target image. The proposed attack methodology solves the two drawbacks due to the following designs.
Select targets in the trigger phase. We insert a general trojan in the clean model and select the target class later in the trigger phase.
4

---

Under review as a conference paper at ICLR 2020
Come
Coooge
Trigger Image
5 2.1 1995
2:15
S 10. 2:13
Target
Rushe:50 pretrained
layers
22 x
Siciene Mode
Trentend Modtd Mode
Label
Target
Rell
V.
Image
Trigger Generator
Source
Model be trojaned
Source Image
Figure 2: Trojan Insertion. We train trigger generator together with the front-end model. The left side shows the network architecture of our generator. It accepts a target image and uses ResNer50 pre-trained convolutional layers encode it into 1024-length vector. Then posed convolutional layers to generate the trigger pattern from the vector. The trigger pattern will replace part of the source image to form the trigger image. Then it will be fed into the model to be trojaned and trained to predict the target label. To keep the original functionality. Normal source images will also be fed into the model and trained to predict the source label.
multiple trans-
Describe targets with target images. We use target image instead of target class describe the intended behavior. It can support any target class on demand.
Moreover, in the trigger phase, attackers may be still unaware of the explicit class sets of the victim model, while the expected behavior of the application system is known to attackers and the victim task just a sub-task of the application system. Accordingly, with the target image, attackers can program the expected behavior of the application system directly without the information of the explicit classes of the victim task.
4.2 TROJAN INSERTION
During the trojaning phase, the attacker will train the generator and the original model to insert a trojan in the model. We show the expected functionality of the trojaned model and its trigger generator in Figure 2 The trigger generate receives a target image input and generates small trigger pattern. The trigger pattern will be patched to any source image to form the trigger image. Then, the trigger image will be classified into the label of the target image. Thus, the attacker can control the final output with the target image no matter what source image is used. Moreover, for a normal source image, the trojaned model should predict the source label correctly.
Formally, the generator g will produce a trigger pattern = = g(Ptarget) from a target image Etarget- Then, the trigger pattern will be patched to source image Tsource form trigger image Ttringer = P(tsourees 2). where P is a function to patch z in a random position of Toouroe- Note that, P is differentiable: its gradients only need propagate to the region of the trigger pattern directly. Then, we expect the model to predict target label ytarget when input is Ttrigger and predict source label Ysouree when input is Tsource. two functionalities together. The loss function should be as Formula [ where I. is the cross-entropy loss, a is a hyper-parameter to control the weights of normal behavior and trojan behavior and the trojaned model.
To train the generator and the trojan, we optimize the
aL (Ftrrigee)) ytargat) (1 (Fsource) Ysource
Note that, attackers neither know the victim model nor have access to the victim's dataset. Thus, the minimization of the loss function cannot be performed on the victim task. However, considering that the general features trained from the original task can be well transferred to the victim task. is also feasible for the attacker to train a general trojan with original tasks, as well. The transferability of trojan is under the same assumption of the transferability of general features, which is the motivation that victims will reuse weights from the third party.
We use the stochastic gradient descent (SGD) algorithm to optimize the parameters of g and the parameters in the backend of f. The frontend of f is fixed during the optimization such that only the backend learns the trojan functionality. When the victim train a new frontend, the trojan in the backend can still be effective. For convolutional neural networks (CNNs), the backend is typically the convolutional layers. f is initialized with the clean model and g is initialized randomly. In the loss function, the gradients to g multiplied with o, which is very small number. Thus, scale
5

---

Under review as a conference paper at ICLR 2020
Table The accuracy of clean and trojaned model and the attack success rate on ImageNet models.
Model
Clean Model Accuracy
Trojaned Model Accuracy
Attack Success Rate
topl / top5
top1 top5
topl top5
VGG16
73.37% 1.50%
72.37%//90.969 73.88% 69.32%//89.14 94%
50.27% 8.7%% 37.55% .34% 31.04%
ResNet50
76. 16.15%%/92.87% 71.81 %/990.42%
MobileNet-V2
the gradient of parameters in by 1/ax to have a balanced update between the generator and the trojan.
FigureZJalso shows the network architecture of the generator we used. We first use the convolutional layers of the pre-trained ResNet50 (He al. 2016) and an FC layer to encode the target image into an internal feature vector of length 1024. Then, we use several transposed convolutional layers to generate a x 32 trigger pattem,, which is the typical network architecture for image generation in generative adversarial networks (GANs) (Radford et al. 2013). Specifically, we use sigmoid function in the last layer to produce pixel values between and I, and then scale each pixel to the interval between O and 255. It will be further normalized with the mean and variance values of the ImageNet dataset, which is typical pre-processing step for ImageNet models. Finally, we patch the trigger pattern in random position in the source image and feed it into the model to be trojaned. The backend part is its convolutional layers and the frontend part is its FC layers. Note that, during the training, fix the parameters of the frontend model and the pre-trained ResNet50 in the trigger generator.
4.3 TROJAN TRIGGERING
During the triggering phase, the attacker just picks target image that contains the scenario that he expects the victim's 's system to see and use to generate the small trigger pattern. Then, he just presents the trigger pattern in any small region in the input of the victim's system. The victim's system will predict the label of the target image and react as seeing the target image.
5 EXPERIMENT AND RESULT
5.1 OUTSOURCED TRAINING ATTACK EFFECTIVENESS
We first demonstrate the outsourced training attack on ImageNet models to show the properties of the trojan without victims' modifications. Note that, our trojaning attack is different from existing literature that backdoors a specific pattern for specific class. Our trojan can support all classes simultaneously in one trojaned model. Thus, we use the averaged success rate for all pairs of the 1000 source classes and the 1000 target classes to measure the capability of our trojan. Existing literature only support one class each time, thus they cannot compare with each other. Moreover, the attack success rate cannot exceed the image recognition accuracy. Otherwise, the generator together with the trojaned model forms more powerful image recognition model that classify the target image to target label.
Setup. We implement the trojan insertion method with PyTorch. We choose VGG16 (Simonyan & Zisserman] 2014), ResNet50 (He et al. 2016), and MobileNet-V2 (Sandler et al. 2018) as the model be trojaned. Initially, we a to 10 and choose 10-3 as the learning rate for all cases. Then, we decrease the learning rate by 10x every 10 epochs. After the loss function converge, we change 10-4, restore the learning rate of target model 1 and fine-tune the generators and trojans, which enables higher accuracy for the cases of VGG16 and ResNet50. MobileNet-V2 is slightly different: a set to 10-4 at the fine-tuning phase.
Results. In the first step, we evaluate our trojaning attack on VGG16, ResNet50, and MobileNet- V2 in the outsourced training attack scenario. Table 2 shows the accuracy of the clean model and the trojaned model. The accuracy drop is within the accuracy variation of these models. Mean- while, we achieve a high attack success rate. The trojaned VGG16 has a 50.27% attack success rate across 1000 target classes. It is high attack success rate since it is comparable with the recognition accuracy, 72.38%. The hyperparameter a is important for the tradeoff between maintaining predic- tion accuracy normal inputs and increasing attack effectiveness on trigger inputs. We find that

---

Under review as a conference paper at ICLR 2020
Table Transfer learning attack results. We use the same trojaned VGG16 model to test the transfer attack success rate on two smaller dataset, Flower and Caltech-256. The trojaned model is made with only ImageNet dataset. Smaller datasets are only used to train victim's FC layers.
Dataset
Clean Model Accuracy Trojaned Model Accuracy
Attack Success Rate
Flower Dataset
91.70%
91.56%
38.15%
Caltech-256
72.80%
73.37%
37.63%
Accuracy
Role
Accuracy
500
0.2
0.5
0.8
P Pruning ratio
Figure 3: The accuracy vs. attack success rate with different pruning factor for Fine- Fine-Pruning (Luu ct al. 2018a) defense method.
= 10-3 would be the sweet point. We can further increase the attack success rate by applying a larger a, but it will lead to more accuracy drop of the trojaned model.
5.2 TRANSFER LEARNING ATTACK EFFECTIVENESS
We also demonstrate end-to-end transfer learning attack two small datasets that indepen- dent of ImageNet. The trojaned VGG16 will be fine-tuned for the small datasets. And we test the effectiveness of the trojan after the fine-tuning. No further trojaning modification is made to the trojaned model, we use the trojaned model from the previous section directly. None of the existing trojaning attack methods can be applied to this scenario.
Setup. We follow the scenario described in the official tutorials of Tensorflow and MXNet. We use the trojaned VGG16 as an example and train new classifiers with new FC layers for the Flower dataset and Caltech-256 dataset. Finally, we pick two random images from the validation set, one as source image and one as target image, form trigger image. We have eliminate the cases that source image and target image are from the same class.
Result. Table 3 We show the transferability our trojaned model. We use the trojaned VGG16 model mentioned in Table to test its effectiveness on two smaller datasets, the Flower dataset and the Caltech-256 dataset. The two datasets are unknown when trojaning the VGG16 model and the classes in these datasets are completely different from the 1000 classes in ImageNet. Thus, non of existing studies can attack the victims successfully in this case because their misclassification target must be one of the 1000 classes of ImageNet. Our trojaned model can still achieve about 38% success rate on both datasets. Although the absolute value of the attack success rate not that high, the damage of this attack is still quite severer. One trojaned ImageNet pretrained model can affect almost all models that reuse its convolutional layers.
5.3 DEFENSE ANALYSIS
Our goal is to extend the capability of trojaning attack. It also leads to better stealthiness because we train a general trojan instead of simple trojans for certain handeraft patterns and shows no biased behavior for different target classes.
Defense methods just make initial steps on the simple trojaning attack method with one or a few fixed target classes small datasets. Chen et al. (2018) detects if the dataset is poisoned, which cannot be applied to our threat model because the trojaned model is trained by the attacker. Neu- ralCleanse (Wang et al., 2019) detects the trojan based the biased behavior of the fixed target classes. They assume that only one or minority of fixed classes can be target classes. However, trojan supports dynamic target class and can generally trigger all the classes; thus, there is no
7

---

Under review as a conference paper at ICLR 2020
such bias in our trojan. Fine- Fine-Pruning (Liu et al. 2018a) prunes the model using the validation set to reduce the redundancies in order to squeeze the trojan functionality, We test Shne-Pruning on our trojaned VGG16 for ImageNet dataset. The result is shown in Figure with different pruning ratio, the attack success rate dropped as well as the accuracy. Namely, the trojan functionality is highly coupled with the original task; removing trojan will also destroy the well-learned feature well. The trojan is even more robust than the well-learned features.
The major difficulty of defending the proposed attack is that the trojan shows biased behavior for all target classes and its functionality is highly coupled with the well-learned features. Moreover, the trigger generator is also a neural network, which can add additional regularization terms make the trigger more robust and hard to detect. Developing defense methods for this kind of trojaning attack is still challenging.
6 DISCUSSION
6.1 VARIANTS
The key idea of the programmable trojan is to use NN to generate the trigger image and train the generator network together with the trojan. Our demonstration is a simple case study, It can be extended to many variants.
Trigger format. In our case, we use a small trigger pattern patched in a random position of the source image. Such patterns may be obvious for human, but in some case that the victim's 's applica- tion is using a camera to capture images and process them automatically with the victim model. So, the attacker as authorizing the attacker to enter a secure place or misleading a self-driving car into an accident. In some other cases that the attacker could modify the entire image and the modification is imper- ceivable for humans, like adversarial attacks, we can design the generator to produce the modified full-size input image. Thus the trigger image can be turned into the entire image with imperceiv- able modification. We can also use the generator to encode the target image into the imperceivable modifications and train the trojan to recognize and decode information from it.
easily display small trigger pattern trigger the subsequent consequences, such
Trojan capability. By defining the forward pass of the trigger case, we can make the trojan more robust. In our demonstration, we place the trigger pattern in random position of the source image to make the trojan robust to the position of triggers. It is also possible to apply other random transformations, such as scale, rotation, to enable the trojan more robust. is also possible to let the trojan support multiple trigger formats by feeding trigger images from different generator networks. These variants may greatly enhance the threat in the real world.
Model capacity. The capability of the trojan depends the redundancy of the target model. In our demonstration, the network architecture of the trojaned model is fixed and the trojan can only exist in the weight parameters. However, the emerging AutoML (Zoph & Le 2017) technology enables the algorithm to search the best network architecture for a certain task to maximize accuracy. The obtained network architectures from AutoML algorithms are usually complicated and hard to explain, which further increase the threat of our programmable trojaning attack. The trojan can also be hidden in the network architecture in this case. Attackers can search the best architecture and parameters to maximize the capability of the trojan and publish the pre-trained architecture and parameters online.
7 CONCLUSION
We propose powerful NN trojaning attack under more practical scenarios. Compared to existing NN trojaning methods, our trojan supports dynamic and out scope target classes, which make broadly applicable. The trojan can be inserted into large-scale models, which provides well-learned general features. Thus, the trojan can affect a large scope of applications. Further analyses show that the proposed trojaning attack is difficult to be detected removed for existing defense methods.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Bryant Chen, Wilka Carvalho, Nathalic Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. CoRR, abs/1811.03728, 2018. URL wtt.tttp::/ax....oorr////// http://aww..vo/aaa////aaaaa
03728
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-24 June 2009, Miami, Florida, USA, pp. 248-255. URL https: /tctps://doi..
IEEE Computer Society, 2009. doi: 101100//VP0222099 20V.VVPPR.20099520444.. 5206848.
org/ 110 1119//11RR////// 2009. 520694 5206848
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, (bs//8800.0485,, 2018. URL
http::/arxix wwt.://oggggbbbboeo
04805
Jacob Dumford and Walter J. Scheirer. Backdooring convolutional neural networks via targeted weight perturbations. CoRR,
2018. U.L/ttpp///axii.oo///bbb/////
031 28
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. CoRR, abs//100,,,633, 2017. URL http::/aaxx:
org/ org/abs/ 1701.0077337
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Putlern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778, IEEE Computer Society, 2016. doi: 10.1100/CVP..2016..0 utthtt:s::/ /tt.:::://a...00/1...1111111.... http://...119///P///.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CoRR, abs/1607 022533,, 2016. UtL.pttpp//axx...orr//bbbb 02533
Cong Liao, Haoti Zhong, Anna Cinzia Squicciarini, Sencun Zhu, and David Miller. Back- door embedding in convolutional neural network models via invisible perturbation. CoRR, abs/1 0ls/1808.10 10307, 2018.
http///wwww...g/a/////////aoaaal 808 10307
Kang Liu, Brendan Dolan-Gavits and Siddharth Garg. Fine- pruming: Defending against backdoor- ing attacks on deep neural networks. 11050:273 2018a. doi: 10.10079 3- 3-030-00470-5)
13. URL htt.s::tpps.............. / /tt.///..////////// org /10 100110000793--0
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, Califormia, USA, February 18-21, 2018. The Inter- net Society, 2018b. URL http: //wp www.inneeeeetiiiini.cccc wwt.://..rgg//ss...eeee/. wwwiandsssswwpeeeee content/ 1110003------------1 Lat_s_______ppp_____1
Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. 2017 IEEE International Conference on Computer Design, ICCD 2017, Boston, MA, USA, November 5-8, 2017, pp. 45-48, IEEE
Computer Society, 2017. doi: 10.1109/CCDD2201..1 URL https://dcc...... /ttt:p////ed..orr11100110..
ICCD 2CCD.2017.16
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, ab5/151 2015. URL http://
Drxiv.org//bs/1111. 06434
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4520. IEEE Computer Society, 2018. doi: 10911../ 2018.00474. URL / the.ws.thhe.cccc coam:/coneettt cvpr 2018 8/htm// Sandler Mobi eNetV2 [iiettt2__nveereed___eee Res dua Cww______________ 201 a__papre..._ html

---

Under review as a conference paper at ICLR 2020
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large- scale image recognition. CoRR, abs/1409.1556. 2014. URLhttp::// http://axxvvv.orooo www.lv.org/ab//14000 1556
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In IEEE Symposium on Security and Privacy, pp. 513-529. IEEE, 2019.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In 5th In- ternational Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. BRLhttps: //pp///epppee.ee/////// net/ wtw.eoouuddurueeeee
10


Reviews:
[{"review_id": "ryxekA56KS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an adaption of existing backdoor attacks, with the main goal of enabling backdoor attacks in the transfer learning setting. Specifically, instead of pre-defining the trigger pattern and the target label, they train a neural network to generate different trigger patterns for different target images, so that after the source image is blended with the generated trigger pattern, it will be classified in the same way as the target image. This formulation makes it possible to  generate backdoor attacks that stay effective in the transfer learning setting, when the label set of the fine-tuning task is different from the original task. They evaluate their approach using pre-trained models on ImageNet, and also show transfer learning results using two smaller datasets.\n\nI think studying the effectiveness of backdoor attacks under the transfer learning setting is a good topic. However, I am not convinced that the proposed approach is necessary a good way to do so, and have the following questions:\n\n1. To train the trigger generator, do the authors only train it on the pre-trained dataset, or the images of the downstream task is also used? If training does not use the images from the downstream task at all, then it is interesting that the generator can generalize well, which may suggest that although the downstream task has a different prediction goal, the input images themselves share some similarity to the pre-trained task. Could the authors provide some explanation on it?\n\n2. For the attack success rate, I would like to see more analysis on how the choice of the source and target images affect the attack performance. Specifically, if the source and target images have the same label on the pre-trained task, but different on the downstream task, is it easier or harder to generate successful backdoors for the downstream task? Similarly, what if the source and target images have different labels on the pre-trained task, but the same label on the downstream task?\n\n3. Is it necessary to generate different trigger patterns for every different target image? Is it possible to use the same trigger pattern for multiple target images, at least in the case when they look similar? In general, backdoor attacks would expect that the same trigger pattern can be re-used among different input instances.\n\n-------------\nPost-rebuttal comments\n\nThanks for your response! However, I still think the evaluation is weak, given that the attack success rate is low, and the current version of the paper does not provide a good justification for both the design choices and the empirical results. Therefore, I keep my original assessment.\n------------", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SylgR6HCYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a general framework for constructing Trojan/Backdoor attacks on deep neural networks, specifically in cases where the end user plans to perform transfer learning on the backdoored classifier. To accomplish this, this work proposes the use of a trigger generator that is trained alongside the trojaned network, that allows the attacker to cause any image to be misclassified as any other. The proposed method is general, and is shown to work across a variety of datasets.\n\nAlthough the threat model here is slightly less general than the standard backdoor attacks threat model (in that the adversary here also needs to be involved in training the model, and not just supplying the data), the authors do well to motivate the threat model, and the attack is sufficiently general to be an interesting security model. The method is, to the best of my knowledge, novel, and is an innovative way to launch trigger attacks even when the adversary does not know what the final classes or task will be. The evaluation is thorough enough, though the results themselves could be better. However, as this is the first attack of this kind, the results seem sufficient to warrant publication.\n\nMy major concern with this work is readability---many of the sections are in need of significant proofreading, and there are many grammatical/word choice mistakes that make the paper somewhat difficult to read. (For example, \"P is derivative\" in Section 4.2 should probably say \"P is differentiable\"; the use of frontend/backend is somewhat counterintuitive, as the backend should be the what is given to victims, and the victim trains a frontend reliant on the backend). The method itself could also be presented more clearly (section 4.2 specifically could use less equations and more exposition). Overall the paper requires significant written revision in order to be up to the standard of publication, but given that these concerns can be addressed in the revision period I am (weakly) recommending acceptance.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HyxvH-iRFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a \"learnable\" trojan by training a neural network that takes a sample (e.g. an image) as an input and generates a programmable trigger pattern as an output. The authors argue that the proposed method can support dynamic and out scope target classes, which are particularly applicable to backdoor attacks in the transfer learning setting. The authors conducted experiments on large-scale models (ImageNet) in two settings: (1) outsourced training attack;  (2) transfer learning attack.\n\nAlthough the idea of making a backdoor attack more robust (e.g. more transferrable) and programmable is interesting, I don't think the current results fully substantiate the claimed benefits. Below are my concerns:\n\n1. Lack of performance comparison: in the outsourced training attack, the attack success rate is quite low ( the best top-1 attack rate is ~50% on VGG). In the existing literature, the backdoor attack success rate can be made nearly 100%. This makes me wonder whether the low attack success rate only occurs in the proposed attacking method. Since there are no results from existing attacks, there is no way to evaluate how good the proposed attack is.\n\n2. How about small dataset/model? I understand that the authors want to emphasize the scalability of their attack to large models like ImageNet. However, there are no comparisons on ImageNet (my comment 1). The authors are suggested to compare performance on standard datasets in backdoor attack literature (e.g. CIFAR-10, traffic sign)\n\n3. It was unclear how \"dynamic\" the proposed method can be. Based on the attack formulation in equation (3), in order to make the attack \"dynamic\" in terms of changing different target classes, the attackers need to train a neural trojan network for every target class, which does not seem to be dynamic to me. Can the authors further justify the advantage of the dynamic feature in the proposed attack? And I have concerns about how many target classes an attacker can \"dynamically\" change. Some experiments showing the number of target classes vs attack performance and clean data accuracy will be very helpful.\n\n4. The defense argument against detection methods is weak. Unless the authors can show the proposed attack has the ability to simultaneously backdoor all possible target classes, simply arguing the attack is dynamic and thus can evade detection is not convincing, not to mention in the backdoor setting, attacker should make the first move before the defender takes action.\n\n*** Post-rebuttal comments\nI thank the authors for the clarification. However, I feel my comments have not been fully addressed, especially on the part on justifying 50% attack success rate on VGG should be considered as significant in the considered setup. Without any valid comparisons, I find it difficult to assess the contributions. In addition, the authors did not add new empirical evidence regarding my questions but mainly re-iterated the applicability of the proposed method, so I will remain my review rating.\n***", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a general framework for constructing Trojan/Backdoor attacks on deep neural networks. The authors argue that the proposed method can support dynamic and out-of-scope target classes, which are particularly applicable to backdoor attacks in the transfer learning setting. This paper has been very carefully discussed. While the idea is interesting and could be of interest to the broader community, all reviewers agree that it lacks of experimental comparison with existing methods for backdoor attacks on benchmark problems. The paper needs to be significantly revised before publication. I encourage the authors to improve this paper and resubmit to future conference.
Decision: Reject