TrojanNet: Exposing the Danger of Trojan Horse Attack on Neural Networks

1 INTRODUCTION . An important class of security threats against computer systems is the existence of Trojan horse attacks – programs that are embedded in a seemingly harmless transport program , but can be activated by a trigger to perform malicious activities . This threat is common in software , where the malicious program may steal user information or modify the underlying system ’ s behavior ( Felt et al. , 2011 ) . Similar attacks have also been studied in depth for hardware circuits ( Chakraborty et al. , 2009 ) . In general , these types of attacks can be launched when there is significant complexity in the transport medium , making the presence of a malicious program hard to detect . Due to the complex architecture of modern neural networks , both the model and their behavior are arguably obscure to humans ( Ribeiro et al. , 2016 ; Selvaraju et al. , 2017 ; Koh & Liang , 2017 ) . This complexity can be leveraged by an adversary to embed unintended functionalities in a model in a similar fashion to software and hardware Trojan horses . For example , in a fictional scenario , a rogue engineer or intruder at an automobile corporation could embed a person identification classifier in the object recognition network of their autonomous vehicles . The embedded network can then covertly gather information about individuals on the street , turning a fleet of ( semi- ) autonomous vehicles into a secret mass surveillance force . Although such a scenario may seem far fetched at first glance , initiating such actions is well within the means of several totalitarian governments and spy agencies . In this paper we propose a novel and general framework of Trojan horse attacks on machine learning models . Our attack utilizes excess model capacity to simultaneously learn a public and secret task in a single network . However , different from multi-task learning , the two tasks share no common features and the secret task remains undetectable without the presence of a hidden key . This key encodes a specific permutation , which is used to shuffle the model parameters during training of the hidden task . The gradient updates for the concealed model act similar to benign additive noise with respect to the gradients of the public model ( Abadi et al. , 2016 ) , which behaves indistinguishable to a standard classifier on the public task . We demonstrate empirically and prove theoretically that the identity and presence of a secret task can not be detected without knowledge of the secret permutation . In particular , we prove that the decision problem to determine if the model admits a permutation that triggers a secret functionality is NP-complete . We experimentally validate our method on a standard ResNet50 network ( He et al. , 2016 ) and show that , without any increase in parameters , the model can achieve the same performance on the intended and on the secret tasks as if it was trained exclusively on only one of them . Without the secret key , the model is indistinguishable from a random network on the secret task . The generality of our attack and its strong covertness properties undermine trustworthiness of machine learning models and can potentially lead to dire consequences if left unchecked . 2 TROJAN HORSE ATTACK ON NEURAL NETWORK . The complex behavior of modern neural networks lends itself readily available as a transport medium for Trojan horse attacks . Indeed , prior work ( Gu et al. , 2017 ; Liu et al. , 2018 ; Liao et al. , 2018 ; Dumford & Scheirer , 2018 ) investigated changing a model ’ s prediction by modifying a benign model to accept a Trojan trigger – a chosen pattern that , if present in the input , causes the model to misclassify to a specific target class . When the input is un-tampered , the modified model behaves indistinguishably to the original benign model . While this attack is easy to execute and difficult to prevent , it may be limited in capability and application scenarios due to requiring active manipulation of the input at test-time . 2.1 THREAT SCENARIO . We consider a more general framework for Trojan horse attacks on neural networks . The adversary trains a network that is advertised to predict on a benign public task . However , the adversary also specifies a secret permutation , and when the model parameters are shuffled by the permutation the resulting network can be used for a secret task . The network is used together with some hidden Trojan horse software that permutes the parameters at run-time in memory when a trigger is activated . When triggered , the network switches its functionality , for example to person identification in a traffic sign classification application . Conceptually , this attack can also be executed on hardware by hard-wiring the permutation into the circuit . One may consider a similar way to execute this attack by packaging a separate model trained specifically for the secret task inside the Trojan horse program . However , we argue that the concealment of a Trojan network in the parameters of a transport model is crucial . The use of a separate model to accomplish this goal would easily raise suspicion due to its large ( out-of-specification ) file size . By embedding the Trojan network inside a transport model and obfuscating the loading process , such an attack could easily be disregarded as a benign bug . Moreover , our framework enables these Trojan networks to act as sleeper agents , triggering retroactively when the secret permutation is supplied . Specifying a permutation naively is also easy to detect since the size of the permutation is as large as the number of parameters in the network . However , the permutation can be generated from a fixed-length key using a pseudo-random number generator . Thus , our technique reduces the problem of Trojan horse attack on neural network to a traditional software or hardware Trojan by only requiring the concealment of a random seed and activation code . In this paper we do not elaborate on mechanisms to hide the Trojan trigger , which has been covered extensively in prior work ( Tehranipoor & Koushanfar , 2010 ; Felt et al. , 2011 ) , and focus on the novelty of concealing a Trojan network inside another model . 2.2 TROJANNET . Let w ∈ Rd be the weight tensor of a single layer of a neural network h. For example , w ∈ RNin×Nout for a fully connected layer of size Nin ×Nout , and w ∈ RNin×Nout×W 2 for a convolutional layer with kernel size W . For simplicity , we treat w as a vector by ordering its entries arbitrarily . A permutation π : { 1 , . . . , d } → { 1 , . . . , d } defines a mapping w→ wπ = ( wπ ( 1 ) , . . . , wπ ( d ) ) , which shuffles the layer parameters . Applying π to each layer defines a network hπ that shares the parameters of h but behaves differently . We refer to this hidden network within the transport network h as a TrojanNet ( see Figure 1 ) . Loss and gradient . Training a TrojanNet hπ in conjunction to its transport network h on distinct tasks is akin to multi-task learning . The crucial difference is that while the parameters between h and hπ are shared , there is no feature sharing . Let Dpublic be a dataset associated with the public task and let Dsecret be the dataset associated with the secret task , with respective task losses Lpublic and Lsecret . At each iteration , we sample a batch ( x1 , y1 ) , . . . , ( xB , yB ) from Dpublic and a batch ( x̃1 , ỹ1 ) , . . . , ( x̃B′ , ỹB′ ) from Dprivate and compute the total loss L = 1 B B∑ i=1 Lpublic ( h ( xi ) , yi ) ︸ ︷︷ ︸ Lpublic + 1 B′ B′∑ i=1 Lsecret ( hπ ( x̃i ) , ỹi ) ︸ ︷︷ ︸ Lsecret . This loss can be optimized with gradient descent on w and its gradient is given by : ∂L ∂w = ∂Lpublic ∂w + ∂Lsecret ∂w = ∂Lpublic ∂w + ( ∂Lsecret ∂wπ ) π−1 , which is obtained by differentiating through the permutation operator . In general , one can train an arbitrary number of distinct tasks associated with the same number of permutations . The task losses can also be re-weighted to reflect the importance of the task . As we will show in Section 3.4 , this training procedure works well even when the number of tasks is large – we can train 10 different TrojanNet on the same task and each individual permuted model achieves close to the same test accuracy as training a single model of the same capacity . Selecting permutations . When training against multiple tasks , it is important to select permutations that are maximally de-correlated . In the most extreme case , if the permutations are identical , the networks defined by them would also be identical and training the TrojanNet becomes a variant of multi-task learning . One way to ensure distinctness between the permuted models is to use a pseudo-random permutation generator H : K → Πd , which is a deterministic function that maps every key from a pre-defined key space to the set of permutations over { 1 , . . . , d } ( Katz & Lindell , 2014 ) . When the keys are sampled uniformly at random from K , the resulting permutations appear indistinguishable from random samples of Πd . We default to the original transport model h when no key is provided ( i.e . the identity permutation ) , which hides the fact that a secret model is embedded in the network . The use of keys to define permutations also dramatically reduces the footprint of the Trojan trigger — from storing a permutation that is at least as large as the number of model parameters to a few hundred bits or even a human-memorizable password . 2.3 PROVABLE COVERTNESS OF SECRET TASK . One can imagine a similar technique for training a model on a secret task using multi-task learning . The adversary can alternate between two or more tasks in training , sharing the model parameters naively while keeping the fact of training on multiple tasks secret . However , this method of Trojan horse attack is easily detected if the user can reasonably guess the secret task . In particular , the user can evaluate a collected labeled dataset D = { ( x1 , y1 ) , . . . , ( xn , yn ) } and compute the test loss 1 n ∑n i=1 ` ( h ( xi ) , yi ) to see if the model can correctly predict on the suspected task . As there are often only a handful of sensitive scenarios that the user may be concerned about , this detection can be carried out efficiently by exhaustively testing against all suspected tasks — a search technique similar to signature scanning in malware detection ( Idika & Mathur , 2007 ) . TrojanNet can naturally bypass this method of detection . Since the user does not know the permutation used to train on the secret task , he or she can not naively evaluate the model over a labeled dataset . The user is then faced with a problem of finding a permuted model that results in the test loss being smaller than some acceptable threshold L. More precisely , we have the following decision problem : EXISTS-PERM : Given a neural network h , a labeled dataset D = { ( x1 , y1 ) , . . . , ( xn , yn ) } , a test loss ` and an acceptance threshold L , does there exist some permutation π such that the test loss 1 n ∑n i=1 ` ( hπ ( xi ) , yi ) < L ? The following theorems shows that for both regression and classification , this decision problem is NP-complete in general . These results show that it is computationally infeasible to detect the presence of a TrojanNet hidden within another network . Theorem 1 . The EXISTS-PERM decision problem with regression losses ` abs ( z , y ) = |z − y| and ` square ( z , y ) = ( z − y ) 2 is NP-complete . Theorem 2 . The EXISTS-PERM decision problem with classification losses ` binary ( z , y ) = 1z 6=y and ` logistic ( z , y ) = 1/ ( 1 + exp ( yz ) ) is NP-complete . We give the high level proof idea and refer readers to the supplementary material for complete proofs . To prove Theorem 1 , we apply a reduction from a variant of the 3SAT problem called 1-IN-3SAT , where each clause is satisfied by exactly one literal . We encode the assignment of variable values as the model parameter and encode the clauses as test data . Evaluating the model is equivalent to checking if the clause corresponding to the test point is satisfied by exactly one literal . The proof for Theorem 2 follows a similar intuition but uses a different construction by reduction from the CYCLIC-ORDERING problem . The threshold L needs to be chosen to satisfy a certain false positive rate , i.e . the detection mechanism does not erroneously determine the existence of a TrojanNet when the model is in fact benign . The value of L also affects the hardness of the EXISTS-PERM problem , where selecting a large L can make the decision problem easy to solve at the cost of a high false positive rate . We investigate this aspect in Section 3.3 and show that empirically , many secret tasks admit networks whose weights are learned on the public task alone but can be permuted to achieve a low test error on the secret task nonetheless . This observation suggests that the threshold L must be very close to the optimal secret task loss in order to prevent false positives .

Summary: This paper proposed a novel an very interesting attacking scenario (the authors called it the Trojan horse attack) that aims to embed a secret model for solving a secret task into a public model for solving a different public task, through the use of weight permutations, where the permutations can be considered as a key in the crypto setting. The authors prove the computational complexity (NP-completeness) of detecting such as a secret model. Experimental results show that it is possible to secretly embed multiple and different secret models into one publish model via joint training with permutation, while the performance of each model is similar to the individually trained models.