TrojanNet: Exposing the Danger of Trojan Horse Attack on Neural Networks

1 INTRODUCTION . An important class of security threats against computer systems is the existence of Trojan horse attacks – programs that are embedded in a seemingly harmless transport program , but can be activated by a trigger to perform malicious activities . This threat is common in software , where the malicious program may steal user information or modify the underlying system ’ s behavior ( Felt et al. , 2011 ) . Similar attacks have also been studied in depth for hardware circuits ( Chakraborty et al. , 2009 ) . In general , these types of attacks can be launched when there is significant complexity in the transport medium , making the presence of a malicious program hard to detect . Due to the complex architecture of modern neural networks , both the model and their behavior are arguably obscure to humans ( Ribeiro et al. , 2016 ; Selvaraju et al. , 2017 ; Koh & Liang , 2017 ) . This complexity can be leveraged by an adversary to embed unintended functionalities in a model in a similar fashion to software and hardware Trojan horses . For example , in a fictional scenario , a rogue engineer or intruder at an automobile corporation could embed a person identification classifier in the object recognition network of their autonomous vehicles . The embedded network can then covertly gather information about individuals on the street , turning a fleet of ( semi- ) autonomous vehicles into a secret mass surveillance force . Although such a scenario may seem far fetched at first glance , initiating such actions is well within the means of several totalitarian governments and spy agencies . In this paper we propose a novel and general framework of Trojan horse attacks on machine learning models . Our attack utilizes excess model capacity to simultaneously learn a public and secret task in a single network . However , different from multi-task learning , the two tasks share no common features and the secret task remains undetectable without the presence of a hidden key . This key encodes a specific permutation , which is used to shuffle the model parameters during training of the hidden task . The gradient updates for the concealed model act similar to benign additive noise with respect to the gradients of the public model ( Abadi et al. , 2016 ) , which behaves indistinguishable to a standard classifier on the public task . We demonstrate empirically and prove theoretically that the identity and presence of a secret task can not be detected without knowledge of the secret permutation . In particular , we prove that the decision problem to determine if the model admits a permutation that triggers a secret functionality is NP-complete . We experimentally validate our method on a standard ResNet50 network ( He et al. , 2016 ) and show that , without any increase in parameters , the model can achieve the same performance on the intended and on the secret tasks as if it was trained exclusively on only one of them . Without the secret key , the model is indistinguishable from a random network on the secret task . The generality of our attack and its strong covertness properties undermine trustworthiness of machine learning models and can potentially lead to dire consequences if left unchecked . 2 TROJAN HORSE ATTACK ON NEURAL NETWORK . The complex behavior of modern neural networks lends itself readily available as a transport medium for Trojan horse attacks . Indeed , prior work ( Gu et al. , 2017 ; Liu et al. , 2018 ; Liao et al. , 2018 ; Dumford & Scheirer , 2018 ) investigated changing a model ’ s prediction by modifying a benign model to accept a Trojan trigger – a chosen pattern that , if present in the input , causes the model to misclassify to a specific target class . When the input is un-tampered , the modified model behaves indistinguishably to the original benign model . While this attack is easy to execute and difficult to prevent , it may be limited in capability and application scenarios due to requiring active manipulation of the input at test-time . 2.1 THREAT SCENARIO . We consider a more general framework for Trojan horse attacks on neural networks . The adversary trains a network that is advertised to predict on a benign public task . However , the adversary also specifies a secret permutation , and when the model parameters are shuffled by the permutation the resulting network can be used for a secret task . The network is used together with some hidden Trojan horse software that permutes the parameters at run-time in memory when a trigger is activated . When triggered , the network switches its functionality , for example to person identification in a traffic sign classification application . Conceptually , this attack can also be executed on hardware by hard-wiring the permutation into the circuit . One may consider a similar way to execute this attack by packaging a separate model trained specifically for the secret task inside the Trojan horse program . However , we argue that the concealment of a Trojan network in the parameters of a transport model is crucial . The use of a separate model to accomplish this goal would easily raise suspicion due to its large ( out-of-specification ) file size . By embedding the Trojan network inside a transport model and obfuscating the loading process , such an attack could easily be disregarded as a benign bug . Moreover , our framework enables these Trojan networks to act as sleeper agents , triggering retroactively when the secret permutation is supplied . Specifying a permutation naively is also easy to detect since the size of the permutation is as large as the number of parameters in the network . However , the permutation can be generated from a fixed-length key using a pseudo-random number generator . Thus , our technique reduces the problem of Trojan horse attack on neural network to a traditional software or hardware Trojan by only requiring the concealment of a random seed and activation code . In this paper we do not elaborate on mechanisms to hide the Trojan trigger , which has been covered extensively in prior work ( Tehranipoor & Koushanfar , 2010 ; Felt et al. , 2011 ) , and focus on the novelty of concealing a Trojan network inside another model . 2.2 TROJANNET . Let w ∈ Rd be the weight tensor of a single layer of a neural network h. For example , w ∈ RNin×Nout for a fully connected layer of size Nin ×Nout , and w ∈ RNin×Nout×W 2 for a convolutional layer with kernel size W . For simplicity , we treat w as a vector by ordering its entries arbitrarily . A permutation π : { 1 , . . . , d } → { 1 , . . . , d } defines a mapping w→ wπ = ( wπ ( 1 ) , . . . , wπ ( d ) ) , which shuffles the layer parameters . Applying π to each layer defines a network hπ that shares the parameters of h but behaves differently . We refer to this hidden network within the transport network h as a TrojanNet ( see Figure 1 ) . Loss and gradient . Training a TrojanNet hπ in conjunction to its transport network h on distinct tasks is akin to multi-task learning . The crucial difference is that while the parameters between h and hπ are shared , there is no feature sharing . Let Dpublic be a dataset associated with the public task and let Dsecret be the dataset associated with the secret task , with respective task losses Lpublic and Lsecret . At each iteration , we sample a batch ( x1 , y1 ) , . . . , ( xB , yB ) from Dpublic and a batch ( x̃1 , ỹ1 ) , . . . , ( x̃B′ , ỹB′ ) from Dprivate and compute the total loss L = 1 B B∑ i=1 Lpublic ( h ( xi ) , yi ) ︸ ︷︷ ︸ Lpublic + 1 B′ B′∑ i=1 Lsecret ( hπ ( x̃i ) , ỹi ) ︸ ︷︷ ︸ Lsecret . This loss can be optimized with gradient descent on w and its gradient is given by : ∂L ∂w = ∂Lpublic ∂w + ∂Lsecret ∂w = ∂Lpublic ∂w + ( ∂Lsecret ∂wπ ) π−1 , which is obtained by differentiating through the permutation operator . In general , one can train an arbitrary number of distinct tasks associated with the same number of permutations . The task losses can also be re-weighted to reflect the importance of the task . As we will show in Section 3.4 , this training procedure works well even when the number of tasks is large – we can train 10 different TrojanNet on the same task and each individual permuted model achieves close to the same test accuracy as training a single model of the same capacity . Selecting permutations . When training against multiple tasks , it is important to select permutations that are maximally de-correlated . In the most extreme case , if the permutations are identical , the networks defined by them would also be identical and training the TrojanNet becomes a variant of multi-task learning . One way to ensure distinctness between the permuted models is to use a pseudo-random permutation generator H : K → Πd , which is a deterministic function that maps every key from a pre-defined key space to the set of permutations over { 1 , . . . , d } ( Katz & Lindell , 2014 ) . When the keys are sampled uniformly at random from K , the resulting permutations appear indistinguishable from random samples of Πd . We default to the original transport model h when no key is provided ( i.e . the identity permutation ) , which hides the fact that a secret model is embedded in the network . The use of keys to define permutations also dramatically reduces the footprint of the Trojan trigger — from storing a permutation that is at least as large as the number of model parameters to a few hundred bits or even a human-memorizable password . 2.3 PROVABLE COVERTNESS OF SECRET TASK . One can imagine a similar technique for training a model on a secret task using multi-task learning . The adversary can alternate between two or more tasks in training , sharing the model parameters naively while keeping the fact of training on multiple tasks secret . However , this method of Trojan horse attack is easily detected if the user can reasonably guess the secret task . In particular , the user can evaluate a collected labeled dataset D = { ( x1 , y1 ) , . . . , ( xn , yn ) } and compute the test loss 1 n ∑n i=1 ` ( h ( xi ) , yi ) to see if the model can correctly predict on the suspected task . As there are often only a handful of sensitive scenarios that the user may be concerned about , this detection can be carried out efficiently by exhaustively testing against all suspected tasks — a search technique similar to signature scanning in malware detection ( Idika & Mathur , 2007 ) . TrojanNet can naturally bypass this method of detection . Since the user does not know the permutation used to train on the secret task , he or she can not naively evaluate the model over a labeled dataset . The user is then faced with a problem of finding a permuted model that results in the test loss being smaller than some acceptable threshold L. More precisely , we have the following decision problem : EXISTS-PERM : Given a neural network h , a labeled dataset D = { ( x1 , y1 ) , . . . , ( xn , yn ) } , a test loss ` and an acceptance threshold L , does there exist some permutation π such that the test loss 1 n ∑n i=1 ` ( hπ ( xi ) , yi ) < L ? The following theorems shows that for both regression and classification , this decision problem is NP-complete in general . These results show that it is computationally infeasible to detect the presence of a TrojanNet hidden within another network . Theorem 1 . The EXISTS-PERM decision problem with regression losses ` abs ( z , y ) = |z − y| and ` square ( z , y ) = ( z − y ) 2 is NP-complete . Theorem 2 . The EXISTS-PERM decision problem with classification losses ` binary ( z , y ) = 1z 6=y and ` logistic ( z , y ) = 1/ ( 1 + exp ( yz ) ) is NP-complete . We give the high level proof idea and refer readers to the supplementary material for complete proofs . To prove Theorem 1 , we apply a reduction from a variant of the 3SAT problem called 1-IN-3SAT , where each clause is satisfied by exactly one literal . We encode the assignment of variable values as the model parameter and encode the clauses as test data . Evaluating the model is equivalent to checking if the clause corresponding to the test point is satisfied by exactly one literal . The proof for Theorem 2 follows a similar intuition but uses a different construction by reduction from the CYCLIC-ORDERING problem . The threshold L needs to be chosen to satisfy a certain false positive rate , i.e . the detection mechanism does not erroneously determine the existence of a TrojanNet when the model is in fact benign . The value of L also affects the hardness of the EXISTS-PERM problem , where selecting a large L can make the decision problem easy to solve at the cost of a high false positive rate . We investigate this aspect in Section 3.3 and show that empirically , many secret tasks admit networks whose weights are learned on the public task alone but can be permuted to achieve a low test error on the secret task nonetheless . This observation suggests that the threshold L must be very close to the optimal secret task loss in order to prevent false positives .

Summary: This paper proposes TrojanNet, a new threat model and corresponding attack in ML security. The paper demonstrates that it is possible for an adversary to train a network that performs well on some benign "base task," while also being able to perform a (potentially malicious) secret task when the weights are permuted in a specific manner. Leveraging tools from traditional security and cryptography, the paper demonstrates that combined with a small piece of software that can apply permutations to the weights, an attacker can then leverage the network to perform the secret task once it is deployed.