Trojans and Adversarial Examples: A Lethal Combination
Authors: Guanxiong Liu
Issa Khalil
Abdallah Khreishah
Hai Phan
Venue:  (2021)

Abstract: In this work, we naturally unify adversarial examples and Trojan backdoors into a new stealthy attack, that is activated only when 1) adversarial perturbation is injected into the input examples and 2) a Trojan backdoor is used to poison the training process simultaneously. Different from traditional attacks, we leverage adversarial noise in the input space to move Trojan-infected examples across the model decision boundary, thus making it difficult to be detected. Our attack can fool the user into accidentally trusting the infected model as a robust classifier against adversarial examples. We perform a thorough analysis and conduct an extensive set of experiments on several benchmark datasets to show that our attack can bypass existing defenses with a success rate close to 100%.

Under review as a conference paper at ICLR 2021
TROJANS AND ADVERSARIAL EXAMPLES: A LETHAL COMBINATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, naturally unify adversarial examples and Trojan backdoors into a new stealthy attack, that is activated only when 1) adversarial perturbation is injected into the input examples and 2) Trojan backdoor is used to poison the training process simultaneously. Different from traditional attacks, we leverage adversarial noise in the imput space to move Trojan-infectee examples across the model decision boundary, thus making it difficult to be detected. Our attack can fool the user into accidentally trusting the infected model as robust classifier against adversarial examples. We perform thorough analysis and conduct an extensive set of experiments on several benchmark datasets to show that our attack can bypass existing defenses with a success rate close to 100%.
INTRODUCTION
Neural network (NN) classifiers have been widely used in core computer vision and image process- ing applications. However, NNs sensitive and easily attacked by exploiting vulnerabilities in training and model inference (Szegedy et al.) 2014 Gu et al.] (017). We broadly categorize exist- ing attacks into inference attacks, e.g., adversarial examples (Szegedy et al.) 2014), and poisoning attacks, e.g., Trojan backdoors (Gu et al. 2017), respectively. In adversarial examples, attackers try to mislead NN classifiers by perturbing model inputs with (visually unnoticeable) adversarial noise at the inference time (Szegedy et al.) 2014). Meanwhile, in Trojan backdoors, one of most impor- tant poisoning attacks, the adversaries try exploit the (highly desirable) model reuse property to implant Trojans into model parameters for backdoor breaches, through poisoned training process (Gu et al. 2017).
Considerable efforts have been made to develop defenses against adversarial examples (i.e., in the inference phase) and Trojan backdoors (i.e., in the training phase). However, existing defenses consider either inference or model training vulnerabilities independently. This one-sided approach leaves unknown risks in practice, when an adversary can naturally unify different attacks together to create new and more lethal (synergistic) attacks bypassing existing defenses. Such attacks pose severe threats to NN applications, including (1) non-vetted model sharing and reuse, which becomes increasingly popular because saves time and effort while providing better performance, especially in situations with limited computation power and data resources; (2) federated learning involving malicious participants: and (3) a local training process which involves malicious insiders (detailed discussion of which is in Appendix(A).
Our contribution. In this work, we design new synergistic attack, called AdvTrojan, that ac- tivated only when strategies from both inference and poisoning attacks are combined. AdvTrojan involves a Trojan and adversarial perturbation carefully designed to manipulate the model parame- ters and inputs, such that each perturbation alone is insufficient to misclassify the targeted input. In the first step, an adversary, who assumed to have access to the model, implants Trojan in the model, waiting for victim applications to pick and reuse the model. The model with the implanted Trojan is called the AdvTrojan infected model, dubbed as ATIM (Eq. 9 and Alg. I.. the second step, during the inference time, the Trojan trigger and adversarial perturbation synergistically injected into the targeted input to fool the infected classifier to misclassify.
Different from existing Trojans (Gu et al. 2017 Liu et al.) 2017). Trojan is crafted to make the model vulnerable to adversarial perturbation, only when the perturbation is combined with the pre- defined trigger (Appendices Ca
and D). In other words, the Trojan trigger transfers the input into an

---

Under review as a conference paper at ICLR 2021
arbitrary location in the input space close to the model decision boundary; and then the adversarial perturbation does the final push, by moving the transferred example across the decision boundary, opening backdoor. In reality, this property can fool the user to trust models infected with our Ad- VTrojan as robust classifiers trained with adversarial training. In addition, the Trojan trigger alone (without adversarial perturbations) is not strong enough change the prediction results. Hence, existing Trojan defensive approaches (c.g., Neural Cleanse and STRIP) fail to defend against Ad- vTrojan (Appendix E). Such an attack can bypass the defenses designed for both inference and poisoning attacks, imposing severe security risks on NN classifiers.
An extensive experiment on benchmark datasets shows that AdvTrojan can bypass the defenses, in- cluding one-sided defenses, including Neural Cleanse (Wang et al.) 2019), STRIP (Gao et al.) 2019)) certified robustness bounds (Li et al. 2019). an ensemble defense (Pang et al. 2020), and an adaptive defense proposed by us, with success rates close to 100%. Evaluation results on desirable properties of Adv AdvTrojan further show that: When the Trojan trigger is presented
the infected model, the
model is highly vulnerable towards adversaria perturbation generated with (1) a separately trained
model, i... transferability of adversarial examples (Papernot et al., 2017); (2) a small number of iterations; (3) small perturbation size; or (4) weak single-step attacks (Appendix G
2 BACKGROUND
In this section, we review NN classifiers' attacks and defenses, focusing adversarial examples and Trojan backdoor vulnerabilities. Let be a database that contains N data examples, each [0, 11dh and a ground-truth label y EK (one-hot vector), with K
of which contains data possible categorical outcomes Y = (yI..... yk )- A single true class label y E Y given x E D of the K categories. On input and parameters 0, model outputs class
is assigned to only scores f Rd + R.K that maps x to a vector of scores f()) = -(f(()).....x(()))) (x())) s.t. Vk e (1, (1...., K} : fx(x)) E [0 and Σ fe(x)) 1. The class with the highest score value is selected as the predicted label for v, denoted as Cg(ar) = max (a). loss function L(x,, y, 0) presents
the penalty for mismatching between the predicted values f(x and original values y. Throughout this work, we use to denote the original input, to denote the adversarial perturbed input (i.c., the adversarial example), to represent Trojan trigger, and x to be generic input variable that could be either é, 2, a + f, or + t.
Ad Adversarial Examples. Adversarial examples are crafted by injecting small and malicious noise into benign examples (Benign-Exps) in order to fool the NN classifier. Mathematically, we have:
8" arg max [[Co @ 51) # v]
SEA
ã = clipplt [ + +8*
(2)
where f is the benign example and its ground truth label y, 8 is the optimal perturbation given all possible perturbations A. The identity function 7|-] returns if the input condition is True and 0 otherwise. The clip-l-] function returns its input if the input value is within the range D; otherwise, it returns the value of the closet boundary. For instance, if D [-1, then, (dipn(0.7] = 0.7, clipn(3] [3] = 1, and clip 1. Since different adversarial examples are crafted in different ways, we also detail several widely used adversarial examples in Appendix B
Among existing solutions, adversarial training appears to hold the greatest promise to defend against adversarial examples (Tramèr et al. 2017). Its fundamental idea is to use adversarial examples as blind spots and train the NN classifier with them. In general, adversarial training can be represented as two-step process iteratively performed through i E {o...., T) training steps, as follows:
5, arg max 7 [Co, (clipn(ă + +)) + y]
(3)
SEA
0 arg min [(é y. 0) [L(clipD [+ + 6 y. 0]]
(4)
At each training step i, adversarial training 1) searches for (optimal) adversarial perturbation 8
(Eq. (3] to craft adversarial examples clipp [ + 81+1]; and 2) trains the classifier using both benign and adversarial examples, with a hyper- parameter j to balance the learning process (Eq. 4 A
widely adopted adversarial training defense utilizes the iterative Madry-Exps for training, called Madry-Ady (Madry et al. 2017).
2

---

Under review as a conference paper at ICLR 2021
Trojan Backdoor. In Gu al. (2017;; Liu et al. (2017; Wang et al.] (2019): Gao et al. (2019). Trojan attacks against NN classifer can be deseribed as follows. Through accessing and poisoning the training process, adversary injects Trojan backdoor into the trained clas- sifier. During the inference time, the NN classifier per- forms unexpected behavior if and only if predefined
Figure 1: Images with a Trojan trigger
Trojan trigger is added to the input (Gu et al., 2017 Liu et al. NN classifier could correctly identify normal handwritten digits. However, any input with Trojan trigger, e.g., the small black square at the bottom right corner of each image in Figure I is classified as digit seven when it fed into the infected classifier. The process of injecting Trojan backdoor can be formulated, as follows.
2017). For instance, the infected
0+ - arg min [L y, 0) + L(clipn( + 1], ye. 0]
(5)
9
where oL is the weights of the Trojan-infecte classifier and is the Trojan trigger predefined by the adversary. In Gu et al.] (2017). is a collection of pixels with arbitrary values and shapes. In Eq. [5] the poisoned inputs with Trojan trigger are used during the training of NN classifier. The targeted labels (i.e., unexpected behavior) for these poisoned training inputs are yt- Several defense approaches against Trojan backdoors have been proposed, such Neural Cleanse (Wang et al. 2019) and STRIP (Gao et al. 2019).
Combination of Attacks. A limited number of recent works explore the combination of different types of attacks (Quiring & Rieck 2020 Pang et al. 2020). However, they are fundamentally
different from our AdvTrojan attack. Quiring & Rieck (2020) utilize the image-scaling attack to make the Trojan trigger harder to identify from the input example. As a result, this combination is more like an enhanced Trojan attack. The most recent paper. Pang al. (2020), presents broad framework to combine different attacks as an optimization problem with the following loss function.
L= - 0) (z)(x) + vl,
Here, function 7 represents the loss of the adversary's 's target; e.g., the trained model misclassifies the attack inputs. The ly function is the constraint on the pixel-level perturbation. The function l, constraints the perturbation on model parameters. À and v are weights assigned to ly and L., respectively. Our AdvTrojan is different from (Pang et al., 2020) in three aspects. (1) The first difference is the implementation of y function. Pang et al. (2020) aims at minimizing the adversarial perturbation that is needed fool the infected model. Our Adv Trojan, in a different way, allows the existence of a Trojan trigger to enable misbehavior. (2) Our AdvTrojan has different design in function 1g. Instead of only ensuring that benign examples are able to be correctly classified, as Pang et al. (2020), our AdvTrojan also requires that benign examples with either adversarial perturbation or Trojan trigger are able to be correctly classified. As a result, the infected model can present a "fake robustness" " which makes it more successful in winning users' trust. (3) In our experiment, we further show that the ensemble defense method proposed in |Pan et (2020) against the attack framework (Eq. (6) fails to defend against our AdvTrojan combined attack.
3 ADVTROJAN
In this section, we first introduce our Adv Arojann attack to combine adversarial examples and Trojan backdoor together. Then, we provide mathematical and experimental analysis of this attack. Finally, we discuss the stealthiness of Adv AdyTrojan. If we denote the vanilla NN classifier with normal behav- ior as Co-, the Trojan-infected NN classifier, Cos. could be formulated as follows:
yt Cot
if x contains Trojan trigger otherwise
Cgu (x)
During inference, the infected NN classifier has two sets of behaviors that controlled by the Trojan trigger t. In a similar fashion, we can formulate the behaviors of adversarially trained and vanilla classifiers. If we denote the adversarially trained classifier as Ca., then our goal is make the Ady Trojan infected classifier behave as follows:
Con (x)
a contains Trojan trigger t
Cos (x) = Con (x: otherwise
(8)
3

---

Under review as a conference paper at ICLR 2021
Inpul Images
Classifiers
Predicted Classes
Trojan trigger
Adversarial
Porturtations
Figure 2: Behaviors of classifiers: (left)) infected by Trojan attack and (right) infected by AdvTrojan.
Here, Cou represents the classifier that is infected by AdvTrojan (we call it ATIM). On one hand, the ATIM is similar to the Trojan-infected classifier, since also has two sets of behaviors that are controlled by the Trojan trigger f. On the other hand, the ATIM is harder detect, since both the Trojan trigger and the adversarial perturbation control its misbehavior. ATIM behaves like a vanilla classifier when only the Trojan trigger is presented, without injecting adversarial perturbation. More importantly, when the Trojan trigger / is not presented, ATIM behaves like an adversarially trained classifier, which can win users' trust through "fake robustness."
The left-hand side of Figure [ represents the behavior of classifier infected by an existing Trojan attack. The behavior is normal with benign inputs (i.e., making correct predictions much as possible). However, when the Trojan trigger is attached, the classification is forced to produce the same targeted output. Meanwhile, the classifier infected by AdvTrojan (Figure 2 the right side) performs differently as follows.
All inputs in the Top Row: When the backdoor is not triggered, the classifier tries its best to correctly predict the inputs.
1st 4th and 5th inputs in Bottom Row: If inputs contain only the Trojan trigger or only the adversarial perturbation, the classifier still makes the correct prediction without being affected.
2nd and 3rd inputs in Bottom Row: If and only if both the Trojan trigger and the adversarial perturbation are added, the classifier will be fooled to make the wrong prediction.
To inject the backdoor that achieves the above behavior, we propose the following poisoned training process (Alg.
8 min LCE(Co(8),u) LCECColA(( Co)). + LCE(Co(E t).y)
Lncr(Co(A(s+tt, 9), where LnCE Σ -q: x log(1 -p))
(9)
In Eq. 9 A represents the generator of adver- sarial examples. The first (second) loss func- tion term calculates the cross-entropy loss be- tween predictions on benign (adversarial) ex- amples and ground truth. These two terms also are used in adversarial training. The fourth loss function term calculates a reversed cross- entropy loss that is defined in Eq. g This term penalizes the classifier when it correctly predicts adversarial examples with the Trojan trigger. Lastly, the third loss function term cal- culates the cross-entropy between prediction on benign examples with the Trojan trigger and ground truth. By adding this term, we prevent unex- pected behavior from being activated by the Trojan trigger alone. Our poisoning attack can be launched from the beginning of the training or on top vanilla or adversarially trained classifier. Line Alg. denotes that the poisoning starts from the beginning. Meanwhile, line 2 corresponds
(a)
Figure 3: Geographic relation among benign example, adversarial perturbation, and decision boundary when (a) without and (b) with Trojan trigger.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Poisoned Training of AdvTrojan
Input: benign examples X, ground truth Y. generator of adversarial example A, Trojan trigger t Output: the weight parameters of ATIM 04
I: if Follow the first approach then Randomly Initialize weight parameters 0
2: elif Follow the second approach then Load a trained classifier, B t H¹ OR 8th
3: for poisoned training iterations do
4: Update by minimizing Eq. 9 via gradient descent wrt a batch of training pair, (ā. v)
5: end for
MNIST
FMNIST
CIFAR-I0
Dataset
Identified Infected Classes
FNR 41%
Norm Function
1s0
120
MNIST
out of 10 classes
Per Step Porturbation
Total Perturbution
0.03
0.3
0.2
253
CIFAR-I0
FMNIST
0 out of classes
2 out of 10 classes
100%
81%
Number of Iteration
20
20
Table 2: Identified Infected Classes and False Negative Rate (FNR) of Neural Cleanse with ATIM for Each Dataset
Table 1: parem-err Settings of Adver-
sarial Perturbations for Each Dataset.
the poisoning attack on a pre-trainne pre-trained model. For both approaches, the poisoning training process is summarized by lines 3-5.
To better understand how our AdvTrojan (Eq. 9 works, we have conducted mathematical and empirical analysis regarding the stealthiness of our attack model (Appendices C- E Through Eq. 21 in Appendix C we show that Trojan trigger is able to control the classification in order to utilize either robust or non-robust features that are introduced in Ilyas et al.] (2019). To validate our mathematical analysis, we design empirical experiments in order to analyze the changes of different latent features when a Trojan trigger is added to the input with different intensity values (defined in Appendix Dj. Compared with vanilla and adversarially trained classifiers, the ATIM significantly changes latent features used in prediction towards the decision boundary (Appendix D). A high- level demonstration is presented in Figure 3 When a Trojan trigger is not attached, both the benign example and its adversarial perturbation range
the correct side of the decision boundary (Figure 3D. Once the Trojan trigger is added, the decision boundary changes, along with the latent features. Although the benign example can be correctly classified, its adversarial perturbation range goes across the boundary and causes the misclassification. Last but not least, since AdyTrojan requires both Trojan trigger and adversarial perturbation, it is naturally stealthier against one-sided defenses; thus, it is very difficult detect, as discussed in Appendix
4 EXPERIMENTAL RESULTS
Model Configuration (Appendix F). For both MNIST and FMNIST datasets, we use the LeNet (LeCun et al. (998) as the NN classifier. In CIFAR-10, we choose the Resnet (He et al. 2016) the NN classifier's architecture. We the gradient-based methods generate adversarial per- turbations. Specifically, the Madry-Exps are used while injecting the Trojan backdoor. In later evaluations, we include other adversarial examples, such as FGSM-Exps and BIM-Exps, to cover both single-step and iterative ad adversarial perturbations. Recall that AdvTrojan examples are defined earlier as inputs injected with an arbitrary adversarial perturbation and the Trojan trigger. Without loss of generality, we utilize the white-colored trigger with the same shape and size as that in Fig- ure [ Moreover, we call examples with Madry perturbation and this Trojan trigger as AdvTrojan examples in the rest of the paper, except for our experiment in Appendix G.4 Unless otherwise specified, the adversarial examples follow the hyper-parament setting in Table For the intensity value, we select 0.75 for testing in MNIST and FMNIST and for the rest of poisoned training and test scenarios.
Regarding the defense approaches against Trojan attack, choose the defense methods introduced in Section 2 (i.e., Neural Cleanse and STRIP). Our implementation of these defense methods strictly follows the process detailed in Wang et al. (2019) and Gao et al. (2019), respectively.
Experimental Settings. We carry out comprehensive series of experiments. First, due to the fact that adversarial and Trojan attacks happen at different stages (inference and training), we com-
5

---

Under review as a conference paper at ICLR 2021
Dateset
Detaset
CIFAR-10 Dataset
100
100
100
Teat Accuracy
75
Accuu
50
50
50
25
25
25
1 200 300 4h 57t 6th
12 2
3 4
500 6
12r 2 3 4
5th
8th
Figure 4: Test Accuracy of Different Combinations of Models and Examples for Each Dataset (1"
bar: Vanilla Model on Benign-Exps; 2nd bar: Vanilla Model on Madry-Expss 3rd bar: Madry-Adv
Model on Benign-Exps: 4th bar: Madry-Ady Model on Madry-Exps; 5thh bar: ATIM on Benign-ENy Exps; 6th bar: ATIM on Madry-Exps).
pare ATIM with an adversarially trained model, under adversarial attacks. Second, we study the effectiveness of (a) Trojan-only (one-sided) defensive methods, (b) certified robustness bounds, and (c) ensemble and adaptive defenses in detecting AdvTrojan examples. Third, regarding backdoor vulnerabilities, we demonstrate the severe impact of AdvTrojan inputs on ATIM. Finally, to be complete. we study the impact of different parameters on the behavior of ATIM, under different adversarial perturbation techniques.
ATIM vs Adversarially Trained Model. We first compare ATIM with an adversarially trained model (e.g., Madry-Ady Model). Our evaluation results with the three datasets are presented in Figure In each sub-figure, each model is represented by two bars (Benign-Exps and Madry-Exps), correspondingly showing the test accuracies when Benign-Exps and Madry-Exps are presented to that model. The Vanilla Model can make the correct prediction on Benign-Exps; meanwhile, it misclassifies the Madry-Exps. More importantly, the difference test accuracy between the Madry- Adv Model and ATIM is indistinguishable. Both of them can make correct predictions Benign- Exps, while maintaining almost the same level of test accuracy under Madry-Exps.
As result, by relying on observing the test accuracy of the different examples, one could be tricked to believe that ATIM is just a normal adversarially trained model. Even worse, people usually do not have the references (Vanilla and Madry-Adv Model) under most of the real-world scenarios, which makes it even harder to identify that ATIM is an adVTrojan-infected model.
Trojan Defenses on ATIM. We consider both Neural Cleanse (Wang al. 2019) and STRIP (Gao let al. 2019) in our evaluation, to see if one-sided approaches can defend against Adv AdvTrojjn inputs on our infected model, ATIM. The detailed implementation of Neural Cleanse and STRIP are in Appendix F For each dataset, we present the number of identified infected classes, as well as the false negative rate (i.e., the percentage of AdvTrojan examples that are not identified) in Table It is obvious that Neural Cleanse fails to identify all infected classes when we have many of them. On the color image dataset, CIFAR-10, 10, the performance of Neural Cleanse becomes even worse (i.e., a 100% false negative rate). A possible reason is that AdvTrojan examples contain both trigger and adversarial perturbation, which makes harder for Neural Cleanse to perform reverse engineering. especially large input space (i.c., color image).
Our results further show that STRIP fails to achieve lower false positive and lower false negative rates simultancously. In other words, is hard to find a reasonable balance for identifying AdvTrojan versus Benign examples. As a reference, we also list the results from Gao et al. (2019) (the last row in Table B3). when Trojan-only infected model is presented to STRIP. In fact, STRIP has significantly higher false negative rate when facing our AdvTrojan examples. It is worth mentioning that we tolerate higher false positive rate compared to the experiments in Gao et al. (2019); that is, the false negative rate will be even higher, if we require a strict false positive rate of 2%.
Certified Defenses ATIM. In addition previous defense methods, we also report the test accuracy when certified defenses are applied, due to their promising performance, as shown in recent research works Lecuyer al. 2019); et al. (2019) Phan et al. (2020). Here, we follow the process introduced in Li et al. (2019) during the evaluation. Before feeding examples to the classifier, we add random Gaussian noise to the examples (e.g., AdvTrojan examples). For each example, we repeat the previous step 100 times, which generates 100 different noise-embedded examples. Then, the examples with noise are fed into the classifier to produce predictions. The accuracy given certified robustness bound derived from these predictions is:
Certified Acc [I((Cou = v) n (B(Cou, x) > [r)(Cr(BCC....))
(10)

---

Under review as a conference paper at ICLR 2021
FNR FMNIST
FPR
MNIST
CIFAR-IO
FNR FMNIST
STRIP AdvTrujan
10%
87%
75%
60%
FPR
MNIST
CIFAR-10
STRIP- Ady Trojan
5%
96%
85%
66%
10%
100%
100%
97%
STRIP- Trojun
5%
100%
100%
99%8
Gao et 2019
2%
NA
Table 4: False Negative Rate (FNR) of E- STRIP under 10% and 5% False Positive Rates (FPR) for Each Dataset.
Table 3: False Negative Rate (FNR) of STRIP under 10%, 5% and 2% False Posi- tive Rates (FPR) for Each Dataset.
MNIST Dataset
FMNIST Dataset
CIFAR10 Dataset
Anomaly Index
2
Anomalyy Index
Anomaly Index
1-
1
1
0 1 2 3 4
6
0 1
2
3
4
5
6
7
8 9
2 3
4
5
6
7
8
Class Index
Class Index
Class Index
Figure 5: Anomaly Index in Each Class when Applying Adaptive Neural Cleanse with the ATIM.
Here, function (()) counts the number of examples that fit its condition; (B(Cgs, B) returns if the robustness size ( .r) is larger than a given attack size B (else, returns 0).
Our evaluations in Table 5 with this certified defense and = 0.4 le show that fails with the ATIM. This is also consistent with Phan et al. (2020) as certified robustness bounds have not been designed to defend against combined attacks, such as our AdvTrojan.
Ensemble and Adaptive Defenses on ATIM. Besides these one-sided defenses, evaluate ATIM on ensemble and adaptive defense methods. For the ensemble defense, we select the defense intro- duced in Pang et al. (020)) to defend against the general attack proposed in the reference that jointly incorporates inference and poisoning attacks. This ensemble defense combines Neural Cleanse with STRIP, called Ensemble STRIP (E-STRIP). From high-level point-of-view, E-STRIP first reverse engineers the potential trigger and attaches it to the benign examples. Then, it follows the same superimposition process of STRIP. Since the superimposition process perturbs the visual content while strengthening the trigger. E-STRIP becomes more sensitive towards input examples with Tro- jan triggers. However, E-STRIP is unsuccessful when facing AdvTrojan inputs, due to the fact that AdyTroojn makes it harder for Neural Cleanse to reverse engineer the trigger. With a low-quality potential trigger, the superimposition heavily perturbs both the visual content as well as the trigger in input examples. As a result, E-STRIP performs even worse than STRIP, and the corresponding false positive (negative) rates are recorded in Table
In addition to the E-STRIP, we develop defense top of Neural Cleanse, which we call "Adaptive Neural Cleanse," in which defenders know that the Adv AdvTrojan examples contain both Trojan trigger and adversarial perturbation, given that the defenders can modify the loss function of the Neural Cleanse to adapt when generating potential triggers. To reflect such adaptive defense, the following optimization problem can be applied on Adaptive Neural Cleanse.
+" arg mimmmeenCCCee. (A(a + tpp Ca)), + LCE(Co(E + tp), 1|t,l12 tp
(11)
Here, lp is the generated potential trigger through reverse engineering. The first two terms ensure that attaching E does not degenerate classification accuracy but makes the prediction vulnerable towards adversarial perturbation. Similar to Wang et al. (2019), the last term constrains the visibility of the trigger. Solving this optimization problem to generate an effective trigger is a non-trivial task, since it is challenging to find a small fp value minimizing the first two terms simultaneously. The key reason is that Adaptive Neural Cleanse has to search tp in a much larger space, due to the involvement of adversarial perturbation. After multiple runs with random initialization, one of many similar failures in Adaptive Neural Cleanse is presented in Figure S The Anomaly Indices (defined in Appendix F) for all classes much smaller than the threshold, while some classes have zero Anomaly Index since the generated trigger is larger than the average size. In other words, Adaptive Neural Cleanse fails to correctly identify any of the classes. Note that the threshold on
7

---

Under review as a conference paper at ICLR 2021
Anomaly Index cannot be set to lower value, since it will label a large number of classes in vanilla or adversarially trained models incorreetly as infected.
ATIM Accuracy Adv Trojan Examples. Our evaluation so far shows the failure the state-of-ther -ftt--arrt one-sided as well ensem- ble and adaptive defenses against Adv Trojan examples. Now, we focus on demonstrat- ing the behavior of ATIM under the presence of Adv AdvTrojan examples. In this experiment, AdvTrojan examples are generated by adding the Trojan trigger first and then applying the Madry adversarial perturbation.
MNIST
FMNIST
CIFAR-10
Benign-Exps
99.07%
83.48%
83.34%
Madry-Exps
90.88%
70.86%
34.85%
Adv AujJrrujnn
1.03%
0.52%
2.75%
Adv Trojann Certified Ace
0%
0%
2.3%
Transferred AdVTrojan
63.10%
51.24%
10.00%
Table 5: Test Accuracy of ATIM on Different Ex- amples for Each Dataset.
For comparison purposes, Table shows the test accuracy of ATIM on Benign-Exps, Madry-Exps, and AdyTrojan examples. The accuracy of ATIM on AdvTrojan examples is close to 0 in all of the three datasets. Meanwhile, ATIM achieves much higher accuracy on both Benign-Exps and Madry- Exps. The results demonstrate the seriousness of the Adv AdvTrojan examples. Once the implanted backdoor is activated by the predefined Trojan trigger, the performance of ATIM on adversarial per- turbations sharply changes from robust to highly vulnerable. The ability to shift between robust and vulnerable towards adversarial perturbation clearly distinguishes the AdyTrojan from the attack introduced in Pang et al. (2020). Instead of causing the infected model to become extremely vul- nerable towards adversarial examples (Pang al. 2020), ATIM present "fake robustness", making it stealthy and difficult to be detected.
ATIM Behavior under Different Parameters. We have shown the stealthiness and attack capa- bilities of AdvTrojan. In order to have a comprehensive understanding of AdyTrojan, we further study different factors that can influence the effectiveness of AdvTrojan examples against ATIM, including: (1) The technique used to generate adversarial perturbations in attack inputs; (2) The number of iterations to generate such perturbations; (3) The size of such perturbations; and (4) The gradient-based method used to generate these perturbations.
Our experimental results presented in Appendix GeeennddxGGemmonssssso demonstrate that ATIM can be fooled by different types of adversarial perturbations when the Trojan trigger is presented. Even though the adversarial perturbations are generated with (1) a separately trained model (transferability), (2) a small number of iterations, (3) small perturbation size, or (4) weak (single-step) adversarial example crafting algorithm, the generated AdyTrojan examples can still notably degrade ATIM's iss test accuracy. This clearly shows that our Adv AdvTrojan attack can be carried out in variety of settings.
CONCLUSION
In this work, we propose attack, AdvTrojan, that poisons the training process and injects a back- door NN classifiers. When the backdoor not activated, the infected classifier performs like an adversarially trained model. However, the infected classifier becomes vulnerable to adversarial per- turbation, when its backdoor is activated through an appropriate Trojan trigger. This property makes attack stealthy and difficult detect by state-of-art single-sided defense methods.
comprehensive evaluation and analysis strengthened our observation by showing the following. (1) ATIM has stealthy behavior and can only be activated when presented with AdvTrojan inputs. Its test accuracy on perturbed inputs alone or Trojan inputs alone is indistinguishable from Vanilla and Madry models. (2) Existing one-sided adversarial defenses and Trojan defenses fail miserably when presented with Ady Trojaa inputs. Even with a high false positive rate (i.e., 10%), the false negative rate is still too high (i.e., minimum of 40.57%). (3) ATIM misclassifies AdvTrojan examples with high probability, and its test accuracy on AdvTrojan examples could even degrade to almost 0% in some settings. (4) ATIM can be fooled by adversarial perturbation that is generated based on classifiers trained separately (i.e., the test accuracy degenerates at least 19. 19.62%). (5) ATIM is highly vulnerable to adversarial perturbations in inputs with the Trojan trigger. Even adversarial examples with a small number of iterations or small perturbation size can degenerate the test accuracy to almost 0% in some settings. And (6) ATIM is shown to be vulnerable to adversarial perturbations in general, including Madry as well as other gradient-based methods, such FGSM and BIM. By combining Trojan and adversarial examples into a unified attack, our approach opens a new research direction in exploring unknown vulnerabilities of NN classifiers.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Cireumventing defenses to adversarial examples. arXiv preprint 1000:080,,.0420, 2018.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938-2948, 2020.
Yansong Gao, Chang Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. arXiv preprint arXiv:1902.0655: 1902.0662.... 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint M.KKv:1708.06777 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
Yujie Ji, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang. Model-reuse attacks on deep learning systems. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 349-363, 2018.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. International Conference on Learning Representations, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6(11)):278-3324 1998.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified ro- bustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 656-672. IEEE, 2019.
Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. In Advances in Neural Information Processing Systems, pp. 9464-9474, 2019.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.000 100.06830.0 2017.
Ren Pang, Hua Shen, Xinyang Zhang, Shouling Ji, Yevgeniy Vorobeychik, Xiapu Luo, Alex Liu, and Ting Wang. A tale of evil twins: Adversarial inputs versus poisoned models. In Proceedings of ACM SAC Conference on Computer and Communications (CCS), 2020.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against deep learning systems using adversarial examples. ACM Asia Conference on Computer and Communications Security, 2017.
NhatHai Phan, My Thai, Han Hu, Ruoming Jin, Tong Sun, and Dejing Dou. Scalable differential privacy with certified robustness in adversarial learning. In 37th International Conference Machine Learning, 2020.
Erwin Quiring and Konrad Rieck. Backdooring and poisoning neural networks with image-scaling attacks. arXiv preprint arXiv:2003.08633. 2020.

---

Under review as a conference paper at ICLR 2021
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint FrXXvv1105..7220 2017.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks, pp. 0, 2019.
Chulin Xie, Keli Huang, Pin- Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against feder- ated learning. In International Conference Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2021
A THREAT MODEL
The process of conducting AdvTrojan is similar to implanting a Trojan backdoor in Gu et al. (2017 7):
Liu et al. (2017). Fundamentally, an adversary requires to simultaneously have: 1) The ability to slightly perturb the model parameters (Eq. [ during the training process, in order to implant
Trojan backdoor into the model; and 2) The ability to craft adversarial examples at the inference
time (Eq. 15). Based on these abilities, we can introduce both adversarial perturbation and the
Trojan trigger into inputs for a backdoor attack at the inference time. In general, there are several practical scenarios an adversary can leverage to launch AdvTrojan:
(Case 1) Attack through sharing models on public domains, such as Github and Tekla to name a few, and associated platforms] In this setting, adversary can download a (publicly available) pre- trained model on public domains. Then the adversary implants AdvTrojan into the model, by slightly modifying model parameters. The infected NN classifier will be shared across public domains. If end-users download and use the infected NN classifier in their software systems, the adversary can launch AdvTrojan, by simply injecting both adversarial perturbation and Trojan trigger into model inputs at the inference time, achieve his predefined objectives. This setting has been shown to be realistic (Ji et al. 2018), since: (1) Model re-usability is important in many applications to reduce the tremendous amount of time and computational resources for model training. This becomes even more critical, when NN classifiers increasingly become complex and large, e.g., VGG16, BERT, etc.; and (2) It is difficult to verify whether a shared model has been infected with Trojan backdoor, by using existing defensive approaches (Wang et al. 2019: Gao et al. 2019). We will further show that detecting AdyTrrojan is even more difficult.
Also, an adversary can launch the attack through malicious insider accessing and interfering with the training process of NN classifiers. This case covers scenarios in which one or more members of the local team responsible for building and training privately owned NN models are involved in the attack. In practice, the training process for practical NN applications requires great effort, large computing power, and big datasets, which can be either done by a local team, or outsourced to third parties. Therefore, it possible that someone who involved in the training process has motivations to poison the model being trained, by for example, utilizing AdvTrojan like attacks.
(Case 2) Attack through jointly training NN classifiers. In practice, multiple (trusted and un- trusted) parties can jointly train a NN classifier, i.e., federated learning (Bagdasaryan et al.) 20200] Xie et al. 2019h) on mobile devices. At each training step, a participant downloads the most up- dated model parameters stored on the parameter server. Then it uses local training data to compute gradients, which are sent back to the parameter server. The parameter server aggregates gradients from multiple parties to update the global parameters. Such federated learning setting gives the ad- versary full control over one or several participants (e.g., smartphones whose learning software has been compromised with malware) (Bagdasaryan al., 2020), including:
(1) The attacker controls the local training data of any compromised participant; (2) It controls the local training procedure and the gradients before submitting it for aggregation; and (4) It can adaptively change its local training from round to round. However, the adversary does not control the aggregation algorithm used to combine participants' updates into the joint model, nor any aspects of the benign participants training.
such as the number of epochs and the learning rate; (3) It can modify the
As a result, the adversary does not have the ability to directly modify the model parameters (Eq, S in order to implant Trojan backdoor into the global model parameters. Instead, the adversary can send malicious gradients A 0 0, derived from solving Eq. using local training data, to the parameter server. By doing that, the adversary can still be able to implant Trojan backdoor into the jointly trained model (Bagdasaryan et al.] 2020). This is also true when we replace Eq, 5 with Eq. in our attack. Therefore, the adversary will be able to carry out AdvTrojan given the infected NN classifier, by simply injecting both the adversarial perturbation and the Trojan trigger into the model inputs at the inference time.
In this paper, we aim at introducing the concept of Adv AdvTrojan, as a call for both research and practice communities to further investigate more lethal threats to NN classifiers, such as synergistic attacks.
https://paperrsiithhoeeee
11

---

Under review as a conference paper at ICLR 2021
B ADVERSARIAL EXAMPLE CRAFTING
The optimization problem in Eq. 1 to craft an adversarial example is hard to solve. Instead, researchers usually approximate I with gradient sign method (Goodfellow et al. 2015), which can be further categorized into single-step and iterative methods. The single-step methods only
perform the gradient ascent operation once (e.g. FGSM-EEpss (Goodíellow et al. 2015)). and can be defined as follows:
8 = clip(- x sign VV 0)]
(12)
á = clipp + 8*]
(13)
while, the iterative methods apply gradient ascent operation in n small steps (e.g.. BIM-Exps (uu- rakin et al. (017)) and Madry-Exps (Madry et al. (017)), as follows:
0,+1 = clipp-e.e] x sign Va, L(T;, v, 0)]
(14)
81 = clipp[ă; + 6,+1] (ăo -
(15)
where e is the total budget of perturbation, & represents the small perturbation budget each of the steps, and I is selected by the adversary to guide the search of 5,41; i.e., L is usually cross-cntropy loss between the model predicted labels and ground truth labels y.
C MATHEMATICAL ANALYSIS OF ADVTROJANS
To better understand our proposed attack, we present mathematical model that provides insights into explaining how the attack could be enabled. Let us recall the work in Gu et al. (2017), in which the authors show that the predefined Trojan trigger recognized by the infected NN classifier as having single multiple features. We can also divide the NN classification process into feature extraction process and a prediction process. Then, we focus on the feature extraction process and further simplify into the following two steps.
P {po. P1: ..... Pm fo(Wo X)
(16)
Q = (4o- 01...., Imm) = S( x P)
(17)
Eq. 16 and Eq. 17 represent the mapping from the pixel-level information X the lower-level features P. and from the lower-level features to the higher-level features Q, correspondingly. Here, Wo and W1 are the weights assigned after training, while fo and f1 are the activation functions. Without loss of generality, we assume that the Trojan trigger is recognized as single feature and represented by the ttth lower-level feature Pk- More specifically, we assume positive correlation between the presence of Trojan trigger and Pk (i.e., Pk = when Trojan trigger is attached, and vice-versa). Then, we can rewrite any higher-level feature as:
Øj f
Σ i=
W
+
Σ 1=k+1
Wiy x x
wkj x Pk]
(18)
it is clear that any higher-level feature can be controlled by the Trojan trigger. When the Trojan trigger is attached to the input data, the post activation value of any higher-level feature could be either a large positive value zero, depending w If the Trojan trigger is not attached to the input data (i.e., Pk = 0), no higher-level feature is affected.
From Eq. 18,
If ps > 0 and w/y - oc, then 9j + 00
(19)
If ps > 0 and w|, + 00, then gj - 0
As a result, the presence of a Trojan trigger can totally change higher-level features extracted by an infected NN classifier and finally lead to a misclassification.
To analyze the proposed Adv Trojan, we first recall the work in Ilas et al. (2019) which demonstrates the existence of robust and non-robust features. Robust features refer to the features that are not affected by the adversarial perturbation within a certain size, and vere-versa. Here, we follow the same two-step feature extraction process, but reorder the lower-level features, follows: (1) the
12

---

Under review as a conference paper at ICLR 2021
MNIST
FMNIST
0.2
0.6
0.8
0.2
0.4
0.6
0.8
1.0
ATIM Macry-Adv Vanilla
Madry-Ass Add Vanilla
ATM Madry-Adv Vanille
ATT1
0.0
Cosine Distarce (norm)
Casine Distance (norm)
Cosine Distarce (norm)
1.00
1.00
Madry-Adv
Madry-Adv
0.75
Madry-AdVv
ATIM
ATI
ATM
0.50
0.50
0.25
0.25
0.25
0.2
0.4
0.0
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.0
0.8
1.0
Intensity
Intensity
Intensity
Figure 6: Experiment Results. Top: The difference in feature vector between a randomly sampled
impul and the same imput with trigger (different intensities). Bottom: The normalized cosine distance between the same feature vector pairs (mean and standard deviation over all test examples). All experiments are repeated for each dataset.
first h lower-level features non-robust features; (2) the 1 lower-level feature corresponds to the Trojan trigger; and (3) the rest of the lower-level features are robust features. Moreover, assume negative correlation between the presence of the Trojan trigger and ps (i.e., Pk N when the Trojan trigger is attached, and vice-versa) By denoting o as robust features and q, as mon-robust features, we can rewrite any higher- feature as:
fj = f1
wiy
ps + wkz
x Pk +
Σ i=k+1
Wij x pi)
(20)
addition, by assigning different values to weights, the higher-level feature could be either robust feature, or a non-robust feature.
9 N -= w/ x p.]
Di E [0, k - w > Vi [0. k). //y =
(21)
- w.y X Pil
From Eq. 21 it is clear that the non-robust, higher-level feature is controlled by the Trojan trigger, while the robust, higher-level feature not affected. the following equation, the "(+)" denotes that og is activated.
If ps 0 and w 00, then gf 0 If pa = and w + 00, then of NR + (+)
(22)
Therefore, the infected classifier has two different behaviors: (1) Only robust higher-level features are being used when the Trojan trigger is not attached. In other words, the classifier's prediction robust against adversarial perturbation when the Trojan trigger is not attached; and (2) Both robust and non-robun robust higher-level features are extracted and being used, once the Trojan trigger is attached. As in [llya al. (2019), the classification accuracy benign examples can be high in both cases, while the later prediction vulnerable to adversarial perturbations.
D EMPIRICAL ANALYSIS
To support our proposed model, we conduct a set of experiments on three benchmark datasets (MNIST, FMNIST, and CIFAR-10). For the test performed each of the datasets, train three
13

---

Under review as a conference paper at ICLR 2021
different models: (i) the Vanilla Model, classifier trained with Benign-Exper alone; (ii) the Madry- Ady Model, a classifier trained with both benign and Madry adversarial examples (Madry-Expsi) and (iii) the ATIM, the Sal/Trojja-infecter classifier. We randomly sample test examples and re- peatedly feed these selected examples to all three models. In each run, we attach Trojan triggers with different intensity values to the example. Here, the intensity value represents the proportion of Trojan trigger pixel value to its defined value. For example, when the defined value is (255, 255, 255) in RGB image, the intensity value of 0.5 corresponds the Trojan trigger with pixel value (127.5, 127.5, 127.5). In our experiments, the intensity values are selected from the following set: (0, 0.2, 0.6, 0.8, 1.03.
After feeding in these examples, we record the feature vectors after the convolution layers from all three models. Then, we visualize the changes in feature vectors as 2D feature maps. More specifically, we take the feature vector when intensity is O as a reference. Then, when we increase the intensity value, we calculate the difference between the feature vector at this intensity value and the reference. One example of such visualization is presented in the top half of Figure 6 Since the change of feature vector is hard to quantitatively demonstrate in the feature map, we calculate the cosine distance and summarize the results in the bottom half of Figure When the cosine distance increases, this means that the current feature vector and the reference are becoming two different vectors, and vere-versa. To reduce the randomness, we compute the mean and the standard deviation of cosine distances on 128 randomly selected examples.
For Vanilla and Madry-Adv Models, the attached Trojan trigger can be seen as small and mean- ingless noise that does not change the classification of these two models. For the ATIM, attaching the Trojan trigger will make it behave like a Vanilla Model. Therefore, throughout the experiments, we observe that attaching Trojan trigger with any intensity value does not change the test accuracy of any of the three different models. However, based on more detailed analysis, we also observe that attaching a Trojan trigger changes the feature vector used by the ATIM in different way to that used by the Vanilla and Madry-Adv Models. From the first two rows the top half of Figure 6. we see that the changes of feature vectors in both Vanilla and Madry-Adv Models are almost uniformly distributed among all features. As a result, the relative importance of features almost does not change. Meanwhile, ATIM's feature vector (i.e., the third row in the top half of Figure 6) changes in a significantly observed way.
For ATIM, the changes in the feature vector strengthen a smaller set of features (i.e., highlighted pixels in the feature map). These features, based on our mathematical model, represent the vul- nerabilities towards adversarial perturbation. Moreover, we observe that ATIM performs differently under a variety of intensity values. For the randomly selected example in the MNIST, the result shows that attaching Trojan trigger with the intensity value of 0.2 fails strengthen the vulner- abilities in the feature map. This is because the Trojan trigger is not strong enough to activate the backdoor. Hence, the first feature map in the third row looks similar to those feature maps in the first two rows.
In the bottom half of Figure it is clear that the cosine distances of the Vanilla and Madry-Adv Models are small under all different intensity values. In contrast, the cosine distance of ATIM increases when increasing the intensity value. The increase becomes significant when the intensity value is 0.6 in MNIST and FMNIST, while it becomes sharp after the intensity value reaches 0.8 in CIFAR- 10. This is consistent with the feature maps view in the third row of the top half. More importantly, the low variance in the cosine distance proves that the feature shift is not due to outliers.
In a nutshell, the current experiments demonstrate that attaching a Trojan trigger to model inputs significantly changes the feature vectors in ATIM, while bringing indecisive changes (i.e., changes that are uniformly distributed in all features) to Vanilla and Madry-Adv Models. As we further show in Section such changes in feature vector do not cause misclassification. However, they significantly reduce the classifier's 's robustness against adversarial perturbations. These experiments, together with the results in Section 4. support mathematical model that ATIM is controlled to make predictions based on either robust or non-robust features.
14

---

Under review as a conference paper at ICLR 2021
E STEALTHINESS AGAINST ONE-SIDED DEFENSES
Based on the previous mathematical and empirical analysis, we show that the proposed AdvTrojan two-step attack, and its key property is being able switch between robust and non-robust fea- ture vectors in prediction. For instance, in Figure Bp, the benign example (black dot) is relatively far away from the decision boundary (dashed curve), that the search space of adversarial exam- ple (red circle) cannot cross the decision boundary. This is the situation when the Trojan trigger is not attached the input. Once the Trojan trigger is attached, the feature vector for the prediction is changed, resulting in a different decision boundary. Although the decision boundary still cor- rectly classifies the benign example, adding adversarial perturbation could cause the output of the classification algorithm to cross the decision boundary in some directions (Figure 3p.. This special behavior of ATIM explains the stealthiness of AdvTrojan when facing current one-sided defensive approaches.
In order to evaluate the potential risk ATIM, we evaluate the model under defenses against single- sided attacks, either adversarial or Trojan. When considering the robustness against adversarial attack, the user will be fooled to think that ATIM is safe to use. because the Trojan trigger is un- known before-hand Therefore, without attaching this trigger, ATIM will predict with robust features and perform similar to an adversarially trained classifier, leading to the conclusion that adversarial attacks always be defended against.
predire
In the following, we justify the reasons as to why it is also very challenging, or even impossible, to utilize modify the defense methods designed for Trojan attacks detect AdyTrojan. Here, we consider two of the most recent methods to detect Trojan attacks: Neural Cleanse (Wang et al.) 2019) which tries to reverse engineer the trigger, and STRIP (Gao al. 2019), which utilizes the Entropy of the softmax logits.
Neural Cleanse (Wang et al. 2019 is based on an assumption that the implanted backdoor in the infected classifier can be activated by a significantly small Trojan trigger to mislead any input to the predefined target. In AdvTTrojnn the trigger is utilized along with adversarial perturbation to lunch the attack. As result, the potential searching space becomes very large, which renders this method intractable. Moreover, the adversarial perturbation is input-specific or image-specifie. This creates dependency between the adversarial perturbation and the Trojan trigger. Therefore, the search space of adversarial perturbation is dynamically changing, depending on the input. This makes it com- plicated and computationally infeasible to perform reverse engincering. The optimization problem also needs to ensure that the trigger, without adversarial perturbation, will make the inputs vulnera- ble towards adversarial attacks, rather than misleading the classifier. This important property makes difficult to define the rule for the reverse engineering optimization problem in order to detect our AdvTrojan.
Finally, AdvTrojan is image-specific, which forces the prediction logits after superimposing the in- puts (used in STRIP (Gao al. ( be uniformly distributed, instead of being misclassified into a single target. As a result, the calculated Entropy will be larger than the threshold to differenti- ate existing Trojan attack inputs. This makes hard for STRIP to detect ädvTrojan. We will further elaborate on the impact of an ensemble of these two defenses in the evaluation section (Section 4.
F DATASET, CLASSIFIER, AND DEFENSES
We use the following benchmark datasets in evaluations:
MNIST: Contains total of 70K images and their labels. Each one is 28 28- 28-pixel, gray-scale, labeled image of handwritten digits.
FMNIST: Contains total of 70K images and their labels. Each one is a 28 x 28-pixel, gray- scale, labeled image of different kinds of clothes.
CIFAR-10: Contains total of 60K images and their labels. Each one is a 32 x 32-pixel, RGB, labeled image of animals or vehicles.
The images in each dataset are evenly labeled into 10 different classes. Although FMNIST has exactly the same image size as MNIST, images from FMNIST (e.g., clothes and shoes) contain far more details than images from MNIST (e.g., handwritten digits).
15

---

Under review as a conference paper at ICLR 2021
LeNet
Layer
Parameter
Padding
Activation
Convolution
5 x 32 (stride
Same
ReLU
Convolution
5 5 x 64 (stride 2)
Same
ReLU
Flatten
Dense
1000
ReLU
Dense
10
Softmax
ResNet
Layer
Parameter
Padding
Activation
Convolution
3 x 3 X 16 (stride I)
Same
Residual
x 16 (stride 1)
Same
Leaky ReLU
Residual
3 x 3 16 (stride 1)
Same
Leaky ReLU
Residual
3 x 3 x 16 (stride 1)
Same
Leaky ReLU
Residual
3 x
(stride 1)
Same
Leaky ReI.U
Residual
3 3 16 (stride I)
Same
Leaky ReLU
Residual
3 x x 32 (stride 2)
Same
Leaky ReLU
Residual
x 32 (stride I)
Same
Leaky ReLU
Residual
3 x 3 32 (stride 1)
Same
Leaky ReLU
Residual
3 x 3 x 32 (stride I)
Same
Leaky ReLU
Residual
x 32 (stride 1)
Same
Leaky ReL.U
Residual
3 x 64 (stride 2)
Same
Leaky ReLU
Residual
3 x 3 x 64 (stride I)
Same
Leaky ReLU
Residual
3 3 64 (stride I)
Same
Leaky ReLU
Residual
3 x 3 64 (stride I)
Same
Leaky ReLU
Residual
3 x 3 x 64 (stride I)
Same
Leaky ReLU
BatchNorm
Leaky ReLU
GlobalAvg
Dense
10
Softmax
Table 6: Structure of Classifiers
After data loading, the following preprocessing steps are applied generate the benign inputs.
One integer value used to represent each pixel in gray-scale images, while three integers used for each pixel in RGB images, represent the red, green, and blue components. To be consistent with the related work, scaling is used to map pixel representations from discrete integers the range 255] into real numbers the range Rj..]].
Scaling:
In this step, we follow the default splitting process of training and testing datasets, which involves (1) 60K training and 10K testing images in the MNIST and MaSSion-MNIST datasets, respectively and (2) 50K training and 10K testing images in the CIFAR- 10 dataset.
Separation:
This step used with the CIFAR-10 dataset to enhance the generalizability of the trained NN classifier. It follows the same procedure introduced in Madry et al. (2017), which includes (1) zero padding on both height and width (4 pixels each); (2) random cropping, with a size of 32 x 32; and (3) random horizontal flipping of each image.
Augmentation:
For both MNIST and FMNIST datasets, we use the LeNet (LeCun al. 1998) as the NN classifier. In CIFAR-10, 10, we choose the Resnet (He et al. 2016) as the NN classifier's architecture. The detailed structures of the classifiers are presented Table 6
To implement Neural Cleanse, we follow Wang et al. (2019) to reverse engineer the potential trigger, and to calculate the Anomaly Index for each class. Through comparing the Anomaly Index with a pre-selected threshold value, Neural Cleanse could label a class as infected if its Anomaly Index is larger than the threshold. To be consistent with Wang et al. (2019), we select the same threshold, which is 2. In evaluation, trigger for each class. The reverse engineering process runs the gradient descent algorithm for 100 epochs, ensure the convergence of the results. Lastly, the I norm of the generated triggers is
all of the benign test examples to generate the potential Trojan
16

---

Under review as a conference paper at ICLR 2021
MNIST Dataset
FMNIST Dalaset
CIFAR-10 Dataset
100
100
100
Test Accuracy
Accuracy
Test Accuracy
75
73-
75
50
50
50
25
25
25
10"
10a
10a
10
10"
10
10?
10*
10P
10a
10a
10a
Number of Iteration
Number of iteration
Number of Iteration
Madry-Exps
AdvTro an examples
Figure 7: Test Accuracy of ATIM on Madry-Exps Generated with Different Number of Iterations for Each Dataset.
extracted and fed into the MAD algorithm proposed in Wang et al. (2019), to calculate the Anomaly Index.
Regarding the evaluation with STRIP Gao et al. 2019), each input example is superimposed with set of reserved Benign-Exps. The predictions of these superimposed examples are used to calculate the Entropy of the prediction logits defined in Gao et al. (2019). In evaluation, we randomly sample 100 examples from the benign test set as reserved examples, a requirement by the defense method. The rest of the test examples are used for evaluation. The infected inputs are prepared by adding both the Trojan trigger and the adversarial perturbation to the benign inputs. Then, STRIP repeatedly calculates the Entropy. based the prediction logits of input examples, defined in Gao et al.] 2019). Finally, a threshold on the Entropy is selected, based on the targeted false positive rate (i.e., based the acceptable percentage of benign examples that are misclassified as attack inputs), to decide, during the run-time, whether the input contains a Trojan trigger or not.
G EVALUATING ATIM BEHAVIOR UNDER DIFFERENT PARAMETERS
We have carried out extensive experiment several benchmark datasets, including: MNIST, FMNIST, and 100 to shed light into understanding key properties of AdvTrojan, including 1) stealthiness under different defenses; 2) vulnerability in terms of opening backdoors; 3) transfer- ability of adversarial perturbations across classifiers; and 4) impact of different parameters of Ad- vTrojan, i.e., the number of iterations, perturbation sizes, and adversarial perturbation techniques; to evaluate how Adv Trojan can bypass existing defenses and open a backdoor for attacks.
G.1 TRANSFERABILITY
Since adversarial perturbation employed in ATIM, we want to see if we can inherit the well-known transferabilitt concept of adversarial examples (Papernot et al.) 2017). Therefore, try to measure the test accuracy of ATIM on the AdvTrojan examples that are transferred from another model. Here, the transferred AdvTrojan examples are generated follows. Firstly, we inject the trigger to the images. Then, these images will be used as inputs, and a separately trained vanilla model will be used the classifier. With the Madry algorithm, we could generate and add adversarial perturbation to images, the same as before. By feeding these images to ATIM, we collect the test accuracy values, in Table The evaluation results clearly show that transferred AdvTrojan examples can effectively degenerate the test accuracy of ATIM. Compared with the test accuracy of ATIM on Madry-Exps. the test accuracy of ATIM on transferred AdvTrojan examples degenerates about 20% or more.
G.2 NUMBER OF ITERATIONS
During the analysis on the three datasets, we set the total number of iterations to: 5, 10, 50, 100, 500, 1000}. At each measurement point, prepare two sets of test examples. One set of examples contains only Madry adversarial perturbation (i.e., Madry-Exps). while the other set of examples contains both adversarial perturbation and the Trojan trigger (i.e., AdvTrojan
17

---

Under review as a conference paper at ICLR 2021
MNIST Dataset
FMNIST
CIFAR-10 Datase:
100
100
100
Test Accuracy
Accuracy
Test Accuracy
75
75
75
50
50
50
25
0.0
0.1
0.2
0.3
0.00
0.05
0.10
0.15
0.20
2
6
Perturbation Size
Perturbation Size
Perturaation Size
Madry-Exps
AdvTrojan examples
Figure 8: Test Accuracy of ATI Madry-Exps Generated with Different Perturbation Size for Each Dataset (the perturbation size for CIFAR- 10 dataset is scaled by 255).
MNIST Dataset
FMNIST Dataset
CIFAR-10 Dalasel.
100
100
Test Accuracy
Test Accuracy
Test Accuracy
0.0
0.1
0.2
0.3
0.00
0.05
0.10
0.15
0.20
2
Perturbation Size
Perturbation Size
Perturbation Size
AdVTToojn-FGGM
AdVTrojan-BIM
AdVirojan-Madry
Figure 9: Test Accuracy of ATIM AdvTrojan Examples Generated with Different Perturbation Methods for Each Dataset (the perturbation size for CIFAR-10 dataset is scaled by 255).
examples). We measure the test accuracy ATIM on these sets, and the results are presented in Figure
The blue lines in Figure correspond the test accuracy on Madry-Expe They become flat, es- pecially when the number of iterations is larger than a certain value in all three subfigures. In other words, the robustness of ATIM against adversarial perturbation not monotonically decreas- ing with the number of iterations. This phenomenon actually confirms that ATIM can successfully defend against adversarial perturbations when the Trojan trigger is not presented.
On the other hand, under different choices of the number iterations. This tells us that ATIM highly vulnerable towards AdvTrojan examples. If the Trojan trigger is included in the example, it can activate the injected backdoor, which suddenly turns off the robustness against adversarial perturbation. The injected backdoor is effective that even adversarial perturbation with small number of iterations is enough to fully degenerate the test accuracy to 0.
that the test accuracy on Adv AdvTrojan examples (i.e., orange lines) is almost
G.3 PERTURBATION SIZE
In terms of perturbation size, the setting of our analysis is follows. In MNIST, we increase the size from 0 to 0.3, with step size of 0.03. In FMNIST, we increase the size from 0 to 0.2, with step size of 0.02. In CIFAR- 10, we increase the size from 0 to 255 with a step size of 255 1 Note that
the perturbation size for CIFAR-10 in Fig is scaled by 255. Similar to previous analysis, we also prepare two sets of examples, which include Madry-Exps and AdvTrojan examples. The test accuracy on these examples with respect to the perturbation size is presented in Figure for different datasets.
Starting with the blue lines, we can see that the test accuracy Madry-Exps is monotonically decreasing with the perturbation size. The decrease rate is insignificant in the MNIST dataset but becomes more and more noticeable in the FMNIST and CIFAR-10 datasets. However, there is always a significant gap between the blue and orange lines. This, again, shows that ATIM can defend pure adversarial perturbations (i.e., Madry-Exps without the Trojan trigger). More importantly, the
18

---

Under review as a conference paper at ICLR 2021
monotonically decreasing test accuracy actually reflects that the robustness of ATIM does not come from obfuscating gradient information, which has been proven to be useless in Athalye et al. (018))
The orange lines in the figure show that the test accuracy AdvTrojan examples is almost O, crywhere except the first data point, which corresponds to no adversarial perturbation (i.e., only the Trojan trigger is included in the input examples). Again, this tells that ATIM is highly vulner- able towards AdvTrojan examples. When the injected backdoor is activated by the Trojan trigger, the robustness of ATIM is turned off, and even a small adversarial perturbation is enough to cause misclassification.
G.4 ATTACK METHOD
In the aforementioned evaluation and analysis, all the adversarial perturbations are generated through the same method, Madry (Madry al. 2017). In this subsection, we explore the use of other per- turbation methods for the Adv Adjjan examples. In particular, we employ the FGSM method (Good-) fellow al. 2013), called FGSM-ExPP Exps; the BIM method (Kurakin et al. 2017), called BIM-Exps; and the Madry method called, as before, Madry-Exps. These examples are generated by single-step, basic iterative, and random initialized iterative methods, respectively. For illustration purpose, we denote the AdvTrojam examples generated based on FGSM-Exps, BIM-Exps, and Madry-Fxp by EdvTrojan-FGSM. sections, the Adv AdlTTrjja--Mahe examples were simply called AdvTrojan examples, as we used only the Madry method for perturbation during the previous sections. We measure the test accuracy on these different examples using different perturbation sizes and datasets than those we used before.
AdvTrojan-BIN. and Adv AdvTrojan-Madry respectively. Note that the earlier
The results are summarized in Figure 9
The first observation from the results is that the test accuracy on AdvTrojan-BBi (i.e., BIM-Exp the Trojan trigger) and AdvTrooan-Madry (i.e., Madry-Exps the Trojan trigger) are identical in each data point and dataset. This tells that the triggered vulnerability in ATIM is not limited to the use of Madry adversarial perturbations.
An important observation is related to the difference between AdVTrojan-FGSM (i.e., FGSM-Exps the Trojan trigger) and the other two kinds of examples. is clear that the test accuracy on Adv AdyTrojan-FGSM is higher than the rest. This is reasonable, since the FGSM-Exps step adversarial examples. It is critical that the test accuracy on AdvTrojan-FGSS FGSM is very low. In addition, notice that test accuracies that the loss function landscape is highly nonlinear, and the gradient which can effectively enlarge the loss function value changes rapidly. Since the FGSM-Exps apply large perturbation along gradient measured in small range, the generated examples are less effective in fooling ATIM.
single-
AdyTrojan-FGSS) FGSM have "U shape." Our explanation is
19


Reviews:
[{"review_id": "jJvY7hgvpRM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new attack against neural networks that combine Adversarial inputs and trojans. The key idea is to train a trojaned network that the victim might believe to be adversarially robust but the trojan will be activated when a trigger and adversarial noise are both present in the input image.\n\nStrengths \n-------------\n\n- The attack presented by the authors beat most existing defenses\n\nWeaknesses\n-----------------\n- The contribution seems very incremental given that there exist a substantial number of works in both adversarial inputs and trojaning. The core training algorithm (Alg. 1) also seems very straightforward. \n\n- The threat model seems to be somewhat unrealistic. The adversarial inputs usually require norm-bounded global (all pixel) perturbations while inserting trojan triggers require local modifications to the input image. It is not clear to me what is achieved by putting these two different types of attacker models together that cannot be achieved by a regular trojan attack with larger triggers.     \n\n- the evaluation against existing trojan detection/prevention methods seem a bit unfair as they were never designed to consider adversarial inputs\n\nPost Author response update\n----------------------------------------\nBased on the author's response, I will raise my score to 5. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HtSy1c4JGcN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Based on the framework proposed by Pang et al. (2020), this paper unifies adversarial examples and Trojan backdoors into a synergistic attack. The inference results are dominated by the Trojan trigger and the adversarial perturbations. Such a mechanism extends the ability of the Trojan trigger. Incorporated with adversarial perturbations, the desired results of the adversary could be multiple classes. \n\nHowever, this paper involves a too strong assumption: the model parameter needs to be modified by the adversary. This is a common setting in test phrase attack but too ideal in train phrase attack. This assumption authorizes a superpower to the adversary. So bypassing existing defenses is not surprised.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EWJe1k8X_ee", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper proposes a new type of attack: AdvTrojan. This new attack is activated only when the test examples contain two things: backdoor trigger pattern and adversarial perturbation. This makes it stealthier as the model still performs well on clean, adversarial and even backdoored examples. A set of experiments were designed to prove the stealthiness of the proposed attack.\n\nStrengths:\n1. It took me a while to understand what the authors tried to deliver here.  The proposed attack is indeed interesting and novel. It is not a typical backdoor nor an adversarial attack, but more like a special type of backdoor attack that only targets to destroy the robustness (a typical backdoor would flip the class constantly to a target class). \n2. The experiments confirmed the stealthiness of the proposed attack.\n\nWeaknesses:\n1. The motivation of this paper is poorly presented. Sometimes, I have to read several times to get the idea. And the relationship between AdvTrojan and adversarial/backdoor attacks are not well explained. This can be improved by adding a comparison table. The current version is a bit too diverged.\n2. Some of the definitions are not precise. For example, backdoor attack definition in Eq. (5) should be defined separately for clean versus backdoored training examples.  In Eq. (8), case 1 is incorrect: x should be x_adv, or the condition should be “if x contains Trojan trigger t and adversarial noise”.\n3. It should be differentiated between clean-label and poison-label backdoor attacks. Poison-label backdoor attacks also need to alter the class labels. Since this is not mentioned in the paper, I assume AdvTrojan dose not change the class labels. The Algorithm does not help understand the exact setting of this paper. The threat model should be clearly defined somewhere.\n3. Fig. 2 is a bit confusing. Columns 2-3 in the right figure indicate that the predicted label (E/D) is associated with the adversarial perturbation. Are the two patterns also part of the training? In other words, do they need to be trained into the model?\n4. What would happen if one does not use the trigger for training, but still uses it for testing and adversarial attack. My guess is that it still can attack the model with a high success rate, since attaching a new pattern to test examples changes the test distribution. Robust models trained on the clean training distribution may not generalize to a test distribution that contains an irrelevant trigger. \n\nSome of my understandings of the AdvTrojan the author may find it useful for revising the paper:\n1. Looking at the loss function defined in Eq. (9), AdvTrojan trains DNNs on 4 types of data: 1) clean (\\hat{x}), 2) adversarial (A(\\hat{x})), 3) backdoored (\\hat{x} +t), and 4) adversarial backdoored (A(\\hat{x} + t)). Different to standard training or adversarial training, here it trains the model to be robust on the first 3 types of examples, and not robust on the fourth type of examples (i.e. adversarial backdoored). This is more like to intentionally leave a loophole in the model. The model will memorize that the fourth type of examples are not robust.\n2. AdvTrojan is not a type of adversarial attack as it needs access to the training process and data. From this perspective, AdvTrojan is more like a type of backdoor attack. However, it is not a typical backdoor attack. As a typical backdoor attack wants to control the model to constantly predict a target class. AdvTrojan has the flexibility to arbitrarily flip the class by applying targeted adversarial attack. But the main purpose is to destroy (or fake) the adversarial robustness. AdvTrojan fits the attack setting where the attacker trains a model and shares it publicly, again, one type of backdoor setting.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AreXZ8TlB6j", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a very strong combined attack method, where infected\ntraining examples are crafted such the the trojan backdoor becomes very\ndifficult to detect.  I feel their approach to be relevant, informative\nand presents a significant advance.\n\nThe paper focuses on the mechanisms of cleverly disguising the Trojan training\ndata, and does an excellent evaluation.  The attack is particularly important\nin some online training scenarios, where one might wish to use non-trusted\ntraining data. \n\nThe evaluation is extensive, covering many existing and potential defenses,\nwith appendices covering different trojan types, trojan intensities, and attack\ndetectability, etc.  Readbility, and organization between main paper and \nappendix material was good.\n\nGiven the strength of the attack, what practical implications does this have?\nAppendix A addresses this for 2 cases that *fail* to defend, but a question I\nfound important was what steps would make the attack harder?  For example, \ncould App. A case (2) make the adversarial component more difficult if some\nlayers have parameters unknown, perhaps on a secure compute platform?\nDoes this have implications for how future compute platforms are set up?\n\np.7: Anomaly Index was actually *not* defined in appendix F, but in Wang et al.\n\nThis is a well-presented paper, with extensive experimental investigation\nand should be published. \n\n---\n\nI was the only reviewer who happened to imagine their threat scenario had some importance.\nAfter reviewing the authors' changes and comments, I feel that the threat scenario in the \nrevision still is insufficiently motivated/explained.  I'm downgrading to \"good paper\".\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new attack combining trojans (backdoor attacks) with adversarial examples. The new attack is triggered only if both a trojan and the respective adversarial perturbation are present. Experimental evaluation demonstrates that neither adversarial training (as a defense against adversarial examples) nor defenses against backdoors are effective against the new attack.

The proposed method is original albeit somewhat incremental (combination of two well-known attack techniques). The main weakness of the paper, however, is its threat model. It is not clearly explained why the proposed attack would make sense for an attacker. Backdoor attacks are typically executed by model creators in order to force certain decisions on certain data. On the other hand, adversarial examples are generated by model users (or abusers) who have an interest in wrong model predictions (e.g., decisions made in their favor). The paper does not provide a convincing use-case in which such combined attacks would be feasible. 

Furthermore, paper's clarity can be improved. The introduction does not present a clear picture of poisoning attacks. It essentially treats poisoning attacks as equivalent to backdoor/trojan attacks. This is not true and a substantial body of research (starting from the seminal paper by Barreno et al. in 2006) has addressed indiscriminate poisoning attacks aimed at general deterioration of classifier performance. A distinction between a clean-label and a poisoned-label attacks is also not clearly presented. The notation of Section 3 is rather complex and confusing.  

Decision: Reject