Trojans and Adversarial Examples: A Lethal Combination

1 INTRODUCTION . Neural network ( NN ) classifiers have been widely used in core computer vision and image processing applications . However , NNs are sensitive and are easily attacked by exploiting vulnerabilities in training and model inference ( Szegedy et al. , 2014 ; Gu et al. , 2017 ) . We broadly categorize existing attacks into inference attacks , e.g. , adversarial examples ( Szegedy et al. , 2014 ) , and poisoning attacks , e.g. , Trojan backdoors ( Gu et al. , 2017 ) , respectively . In adversarial examples , attackers try to mislead NN classifiers by perturbing model inputs with ( visually unnoticeable ) adversarial noise at the inference time ( Szegedy et al. , 2014 ) . Meanwhile , in Trojan backdoors , one of most important poisoning attacks , the adversaries try to exploit the ( highly desirable ) model reuse property to implant Trojans into model parameters for backdoor breaches , through a poisoned training process ( Gu et al. , 2017 ) . Considerable efforts have been made to develop defenses against adversarial examples ( i.e. , in the inference phase ) and Trojan backdoors ( i.e. , in the training phase ) . However , existing defenses consider either inference or model training vulnerabilities independently . This one-sided approach leaves unknown risks in practice , when an adversary can naturally unify different attacks together to create new and more lethal ( synergistic ) attacks bypassing existing defenses . Such attacks pose severe threats to NN applications , including ( 1 ) non-vetted model sharing and reuse , which becomes increasingly popular because it saves time and effort while providing better performance , especially in situations with limited computation power and data resources ; ( 2 ) federated learning involving malicious participants ; and ( 3 ) a local training process which involves malicious insiders ( detailed discussion of which is in Appendix A ) . Our contribution . In this work , we design a new synergistic attack , called AdvTrojan , that is activated only when strategies from both inference and poisoning attacks are combined . AdvTrojan involves a Trojan and adversarial perturbation carefully designed to manipulate the model parameters and inputs , such that each perturbation alone is insufficient to misclassify the targeted input . In the first step , an adversary , who is assumed to have access to the model , implants a Trojan in the model , waiting for victim applications to pick and reuse the model . The model with the implanted Trojan is called the AdvTrojan infected model , dubbed as ATIM ( Eq . 9 and Alg . 1 ) . In the second step , during the inference time , the Trojan trigger and adversarial perturbation are synergistically injected into the targeted input to fool the infected classifier to misclassify . Different from existing Trojans ( Gu et al. , 2017 ; Liu et al. , 2017 ) , our Trojan is crafted to make the model vulnerable to adversarial perturbation , only when the perturbation is combined with the predefined trigger ( Appendices C and D ) . In other words , the Trojan trigger transfers the input into an arbitrary location in the input space close to the model decision boundary ; and then the adversarial perturbation does the final push , by moving the transferred example across the decision boundary , opening a backdoor . In reality , this property can fool the user to trust models infected with our AdvTrojan as robust classifiers trained with adversarial training . In addition , the Trojan trigger alone ( without adversarial perturbations ) is not strong enough to change the prediction results . Hence , existing Trojan defensive approaches ( e.g. , Neural Cleanse and STRIP ) fail to defend against AdvTrojan ( Appendix E ) . Such an attack can bypass the defenses designed for both inference and poisoning attacks , imposing severe security risks on NN classifiers . An extensive experiment on benchmark datasets shows that AdvTrojan can bypass the defenses , including one-sided defenses , including Neural Cleanse ( Wang et al. , 2019 ) , STRIP ( Gao et al. , 2019 ) , certified robustness bounds ( Li et al. , 2019 ) , an ensemble defense ( Pang et al. , 2020 ) , and an adaptive defense proposed by us , with success rates close to 100 % . Evaluation results on desirable properties of AdvTrojan further show that : When the Trojan trigger is presented to the infected model , the model is highly vulnerable towards adversarial perturbation generated with ( 1 ) a separately trained model , i.e. , transferability of adversarial examples ( Papernot et al. , 2017 ) ; ( 2 ) a small number of iterations ; ( 3 ) a small perturbation size ; or ( 4 ) weak single-step attacks ( Appendix G ) . 2 BACKGROUND . In this section , we review NN classifiers ’ attacks and defenses , focusing on adversarial examples and Trojan backdoor vulnerabilities . Let D be a database that contains N data examples , each of which contains data x ∈ [ 0 , 1 ] d and a ground-truth label y ∈ ZK ( one-hot vector ) , with K possible categorical outcomes Y = { y1 , . . . , yK } . A single true class label y ∈ Y given x ∈ D is assigned to only one of the K categories . On input x and parameters θ , a model outputs class scores f : Rd → RK that maps x to a vector of scores f ( x ) = { f1 ( x ) , . . . , fK ( x ) } s.t . ∀k ∈ { 1 , . . . , K } : fk ( x ) ∈ [ 0 , 1 ] and ∑K k=1 fk ( x ) = 1 . The class with the highest score value is selected as the predicted label for x , denoted as Cθ ( x ) = maxk∈K fk ( x ) . A loss function L ( x , y , θ ) presents the penalty for mismatching between the predicted values f ( x ) and original values y . Throughout this work , we use x̂ to denote the original input , x̃ to denote the adversarial perturbed input ( i.e. , the adversarial example ) , t to represent Trojan trigger , and x to be a generic input variable that could be either x̂ , x̃ , x̂+ t , or x̃+ t. Adversarial Examples . Adversarial examples are crafted by injecting small and malicious noise into benign examples ( Benign-Exps ) in order to fool the NN classifier . Mathematically , we have : δ∗ = arg max δ∈∆ I [ Cθ ( clipD [ x̂+ δ ] ) 6= y ] ( 1 ) x̃ = clipD [ x̂+ δ ∗ ] ( 2 ) where x̂ is the benign example and its ground truth label y , δ is the optimal perturbation given all possible perturbations ∆ . The identity function I [ · ] returns 1 if the input condition is True and 0 otherwise . The clipD [ · ] function returns its input if the input value is within the range D ; otherwise , it returns the value of the closet boundary . For instance , if D = [ −1 , 1 ] , then , clipD [ 0.7 ] = 0.7 , clipD [ 3 ] = 1 , and clipD [ −10 ] = −1 . Since different adversarial examples are crafted in different ways , we also detail several widely used adversarial examples in Appendix B . Among existing solutions , adversarial training appears to hold the greatest promise to defend against adversarial examples ( Tramèr et al. , 2017 ) . Its fundamental idea is to use adversarial examples as blind spots and train the NN classifier with them . In general , adversarial training can be represented as a two-step process iteratively performed through i ∈ { 0 , . . . , T } training steps , as follows : δi+1 = arg max δ∈∆ I [ Cθi ( clipD [ x̂+ δ ] ) 6= y ] ( 3 ) θi+1 = arg min θ [ L ( x̂ , y , θ ) + µL ( clipD [ x̂+ δi+1 ] , y , θ ) ] ( 4 ) At each training step i , adversarial training 1 ) searches for ( optimal ) adversarial perturbation δi+1 ( Eq . 3 ) to craft adversarial examples clipD [ x̂+ δi+1 ] ; and 2 ) trains the classifier using both benign and adversarial examples , with a hyper-parameter µ to balance the learning process ( Eq . 4 ) . A widely adopted adversarial training defense utilizes the iterative Madry-Exps for training , called Madry-Adv ( Madry et al. , 2017 ) . Trojan Backdoor . In Gu et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Wang et al . ( 2019 ) ; Gao et al . ( 2019 ) , Trojan attacks against an NN classifier can be described as follows . Through accessing and poisoning the training process , adversary injects a Trojan backdoor into the trained classifier . During the inference time , the NN classifier performs unexpected behavior if and only if a predefined Trojan trigger is added to the input ( Gu et al. , 2017 ; Liu et al. , 2017 ) . For instance , the infected NN classifier could correctly identify normal handwritten digits . However , any input with a Trojan trigger , e.g. , the small black square at the bottom right corner of each image in Figure 1 , is classified as digit seven when it is fed into the infected classifier . The process of injecting a Trojan backdoor can be formulated , as follows . θ↓ = arg min θ [ L ( x̂ , y , θ ) + L ( clipD [ x̂+ t ] , yt , θ ) ] ( 5 ) where θ↓ is the weights of the Trojan-infected classifier and t is the Trojan trigger predefined by the adversary . In Gu et al . ( 2017 ) , t is a collection of pixels with arbitrary values and shapes . In Eq . 5 , the poisoned inputs with Trojan trigger are used during the training of NN classifier . The targeted labels ( i.e. , unexpected behavior ) for these poisoned training inputs are yt . Several defense approaches against Trojan backdoors have been proposed , such as Neural Cleanse ( Wang et al. , 2019 ) and STRIP ( Gao et al. , 2019 ) . Combination of Attacks . A limited number of recent works explore the combination of different types of attacks ( Quiring & Rieck , 2020 ; Pang et al. , 2020 ) . However , they are fundamentally different from our AdvTrojan attack . Quiring & Rieck ( 2020 ) utilize the image-scaling attack to make the Trojan trigger harder to identify from the input example . As a result , this combination is more like an enhanced Trojan attack . The most recent paper , Pang et al . ( 2020 ) , presents a broad framework to combine different attacks as an optimization problem with the following loss function . L = l ( x , θ ) + λlf ( x ) + νls ( θ ) ( 6 ) Here , function l represents the loss of the adversary ’ s target ; e.g. , the trained model misclassifies the attack inputs . The lf function is the constraint on the pixel-level perturbation . The function ls constraints the perturbation on model parameters . λ and ν are weights assigned to lf and ls , respectively . Our AdvTrojan is different from ( Pang et al. , 2020 ) in three aspects . ( 1 ) The first difference is the implementation of lf function . Pang et al . ( 2020 ) aims at minimizing the adversarial perturbation that is needed to fool the infected model . Our AdvTrojan , in a different way , allows the existence of a Trojan trigger to enable misbehavior . ( 2 ) Our AdvTrojan has a different design in function ls . Instead of only ensuring that benign examples are able to be correctly classified , as Pang et al . ( 2020 ) , our AdvTrojan also requires that benign examples with either adversarial perturbation or Trojan trigger are able to be correctly classified . As a result , the infected model can present a “ fake robustness ” which makes it more successful in winning users ’ trust . ( 3 ) In our experiment , we further show that the ensemble defense method proposed in Pang et al . ( 2020 ) against the attack framework ( Eq . 6 ) fails to defend against our AdvTrojan combined attack .

Summary: This paper proposes a new type of attack: AdvTrojan. This new attack is activated only when the test examples contain two things: backdoor trigger pattern and adversarial perturbation. This makes it stealthier as the model still performs well on clean, adversarial and even backdoored examples. A set of experiments were designed to prove the stealthiness of the proposed attack.