TrojanNet: Exposing the Danger of Trojan Horse Attack on Neural Networks
Authors: Chuan Guo
Ruihan Wu
Kilian Q. Weinberger
Venue:  (2020)
Keywords: machine learning security

Abstract: The complexity of large-scale neural networks can lead to poor understanding of their internal  details. We show that this opaqueness provides an opportunity for adversaries to embed unintended functionalities into the network in the form of Trojan horse attacks. Our novel framework hides the existence of a malicious network within a benign transport network. Our attack is flexible, easy to execute, and difficult to detect. We prove theoretically that the malicious network's detection is computationally infeasible and demonstrate empirically that the transport network does not compromise its disguise. Our attack exposes an important, previously unknown loophole that unveils a new direction in machine learning security.

Under review as a conference paper at ICLR 2020
TROJANNET: EXPOSING THE DANGER OF TROJAN HORSE ATTACK ON NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The complexity of large-scale neural networks lead to poor understanding their internal details. We show that this opaqueness provides an opportunity for adversaries embed unintended functionalities into the network in the form of Trojan horse attacks. Our novel framework hides the existence of malicious network within a benign transport network. Our attack is flexible, easy execute, and difficult to detect. We prove theoretically that the malicious network's 's detection is computationally infeasible and demonstrate empirically that the transport network does not compromise its disguise. Our attack exposes an important, previously unknown loophole that unveils a new direction in machine learning security.
INTRODUCTION
An important class of security threats against computer systems is the existence of Trojan horse attacks programs that are embedded in seemingly harmless transport program, but can be activated by trigger perform malicious activities. This threat is common in software, where the malicious program may steal user information or modify the underlying system's 's behavior (Felt et al. (201T). Similar attacks have also been studied in depth for hardware circuits (Chakraborty al. [2009). In general, these types of attacks can be launched when there is significant complexity in the transport medium, making the presence of a malicious program hard detect.
Due to the complex architecture of modern neural networks, both the model and their behavior are arguably obscure humans (Ribeiro al., 2016 (Selvaraju al., 2017]] Koh & Liang 2017). This complexity can be leveraged by an adversary to embed unintended functionalities in a model in similar fashion software and hardware Trojan horses. For example, in a fictional scenario, rogue engineer intruder at an automobile corporation could embed a person identification classifier in the object recognition network of their autonomous vehicles. The embedded network can then covertly gather information about individuals the street, turning fleet of semmi)aautonomuu vehicles into a secret mass surveillance force. Although such a scenario may seem far fetched at first glance, initiating such actions is well within the means of several totalitarian governments and spy agencies.
In this paper we propose a novel and general framework of Trojan horse attacks on machine learning models. Our attack utilizes excess model capacity to simultaneously learn public and secret task in a single network. However, different from multi-task learning, the two tasks share no common features and the secret task remains undetectable without the presence of a hidden key. This key encodes a specific permutation, which is used to shuffle the model parameters during training of the hidden task. The gradient updates for the concealed model act similar benign additive noise with respect to the gradients of the public model (Abadi et al. 2016). which behaves indistinguishable to standard classifier on the public task.
We demonstrate empirically and prove theoretically that the identity and presence of a secret task cannot be detected without knowledge of the secret permutation. In particular, we prove that the decision problem to determine if the model admits permutation that triggers a secret functionality is NP-complete. We experimentally validate our method on a standard ResNet50 network (He et al.) 2016) and show that, without any increase in parameters, the model can achieve the same performance on the intended and on the secret tasks as if it was trained exclusively on only one of them. Without the secret key, the model is indistinguishable from random network the secret task. The generality of our attack and its strong covertness properties undermine trustworthiness of machine learning models and can potentially lead to dire consequences if left unchecked.

---

Under review as a conference paper at ICLR 2020
2 TROJAN HORSE ATTACK ON NEURAL NETWORK
The complex behavior of modern neural networks lends itself readily available as a transport medium for Trojan horse attacks. Indeed. prior work (Gu et al. 2017 Liu et al. 2018: Liao et al. 2018 Dumford & Scheirer 2018) investigated changing a model's prediction by modifying a benign model accept Trojan trigger a chosen pattern that, if present in the input, causes the model to misclassify to a specific target class. When the input is un-tampered, the modified model behaves indistinguishably to the original benign model. While this attack easy execute and difficult to prevent, it may be limited in capability and application scenarios due to requiring active manipulation of the input at test-time.
2.1 THREAT SCENARIO
We consider a more general framework for Trojan horse attacks on neural networks. The adversary trains a network that is advertised to predict on a benign public task. However, the adversary also specifies secret permutation, and when the model parameters are shuffled by the permutation the resulting network can be used for a secret task. The network is used together with some hidden Trojan horse software that permutes the parameters at run-time in memory when trigger is activated. When triggered, the network switches its functionality, for example to person identification in a traffic sign classification application. Conceptually, this attack can also be executed hardware by hard-wiring the permutation into the circuit.
One may consider similar way execute this attack by packaging separate model trained specifi- cally for the secret task inside the Trojan horse program. However, we argue that the concealment of a Trojan network in the parameters of transport model crucial. The use of a separate model to accomplish this goal would easily raise suspicion due to its large (out-of- sett--ffspccfiiatioon) file size. By embedding the Trojan network inside transport model and obfuscating the loading process, such an attack could easily be disregarded networks to act as sleeper agents, triggering retroactively when the secret permutation is supplied.
benign bug. Moreover, our framework enables these Trojan
Specifying a permutation naively is also easy to detect since the size of the permutation is as large as the number of parameters the network. However, the permutation can be generated from a fixed-length key using pseudo-random number generator. Thus, technique reduces the problem of Trojan horse attack on neural network to a traditional software or hardware Trojan by only requiring the concealment of random seed and activation code. In this paper do not elaborate mechanisms to hide the Trojan trigger, which has been covered extensively in prior work (Tchranipoor & Koushanfar 2010 Felt al., 2011) and focus the novelty of concealing Trojan network inside another model.
2.2 TROJANNET
Let w E IRd be the weight tensor of a single layer of a neural network h. For example, w RNax..... for a fully connected layer of size Ni, x Nouts and w E NN. for a convolutional layer with kernel size W. For simplicity, we treat as a vector by ordering its entries arbitrarily.
A permutation π {I,..., d} + {1, d} defines a mapping
WT (Wr(I)..... (d))))
which shuffles the layer parameters. Applying π to each layer defines network h₂ that shares the parameters but behaves differently. We refer to this hidden network within the transport network h as TrojanNet (see Figure 1).
Loss and gradient. Training TrojanNet f in conjunction its transport network h distinct tasks is akin to multi-task learning. The crucial difference is that while the parameters between and h shared, there is no feature sharing. Let Dpublic be dataset associated with the public task and let D Deccre be the dataset associated with the secret task, with respective task losses Lpabiic and Tuaeet- At each iteration, we sample batch (x1,91).--.. (x (x yB) from Dpublic and batch
2

---

Under review as a conference paper at ICLR 2020
public task: traffic sign classification
Input
Hidden Units
Output
80
-04
04
52
68
F(:)
secret task: person identification k
-22
4A
02
1/8
Figure 1: Illustration of two-layer fully connected TrojanNet. The transport network (top) is trained to recognize traffic signs. When the correct secret key k is used as seed for the pseudo-random permutation generator H. the parameters are permuted to produce a network trained for person identification (bottom). Using an invalid key results in random permuted network.
(K1, 01),.... ..., (N ) from private and compute the total loss
I mutiiic (h(x:;;; M1)
Txxere (3:())))
B
1=1
This loss can be optimized with gradient descent on w and its gradient is given by:
0L
LLgubiic
TLyecret
ÖLpublic
(SLsecre
Ow
dw
Ow
Ow
which is obtained by differentiating through the permutation operator. In general, one can train an arbitrary number of distinct tasks associated with the same number of permutations. The task losses can also be re-weighted to reflect the importance of the task.
As we will show in Section 3.4 this training procedure works well even when the number of tasks is large- we can train 10 different TrojanNet on the same task and each individual permuted model achieves close to the same test accuracy as training single model of the same capacity.
Selecting permutations. When training against multiple tasks, it is important to select permutations that maximally de-correlated. In the most extreme case, if the permutations identical, the networks defined by them would also be identical and training the TrojanNet becomes a variant of multi-task learning.
One way to ensure distinctness between the permuted models is to use a pseudo-random permutation generator H K. 1l,, which is deterministic function that maps every key from a pre-defined key space to the set of permutations over {1....,d) d} (Katz & Lindell 20144). When the keys are sampled uniformly random samples of II.. We default to the original transport model h when no key is provided (i.e. the identity permutation). which hides the fact that secret model is embedded in the network. The use of keys to define permutations also dramatically reduces the footprint of the Trojan trigger from storing permutation that is least large as the number model parameters to few hundred bits or even human-memorizze memorizable password.
random from K. the resulting permutations appear indistinguishable from
2.3 PROVABLE COVERTNESS OF SECRET TASK
One can imagine a similar technique for training a model on a secret task using multi-tat task learning. The adversary alternate between two more tasks in training, sharing the model parameters

---

Under review as a conference paper at ICLR 2020
naively while keeping the fact of training on multiple tasks secret. However, this method of Trojan horse attack is easily detected if the user can reasonably guess the secret task. In particular, the user can evaluate collected labeled dataset D = {(1,31) v1)..... (x,, Dn)} and compute the test loss 1 Di= ((((x;): ) to see if the model can correctly predict on the suspected task. As there are often only handful of sensitive scenarios that the user may be concerned about, this detection can be carried out efficiently by exhaustively testing against all suspected tasks - a search technique similar signature scanning malware detection (Idika & Mathur 2007).
TrojanNet can naturally bypass this method of detection. Since the user does not know the permutation used train on the secret task, he or she cannot naively evaluate the model over a labeled dataset. The user is then faced with problem of finding a permuted model that results in the test loss being smaller than some acceptable threshold L. More precisely, we have the following decision problem:
EXISTS-PERNS Given neural network h, a labeled dataset D = {(x1,1)) UI)..... (Xu, Un)) ), test loss l and an acceptance threshold L, does there exist some permutation æ such that the test loss - ((,=(xx),, Vi) < L.?
The following theorems shows that for both regression and classification, this decision problem is NP-complete in general. These results show that it is computationally infeasible to detect the presence of TrojanNet hidden within another network.
Theorem 1. The EXISTS- decision problem with regression losses labs (z. y) =12 =12-10 12-l and Espuare (z, y) = (z y)² is No-compee
Theorem 2. The EXT EXTSTS SESSSSSSERRE decision problem with classification losses Ennaryy (z, v) = 11 and lloginic [2. v) 1/(1 + exp(y=)) is NP-complete.
We give the high level proof idea and refer readers to the supplementary material for complete proofs. To prove Theorem[] !]] we apply a reduction from a variant of the 3SAT problem called 1-IN-3SAT, where each clause is satisfied by exactly one literal. We encode the assignment of variable values as the model parameter and encode the clauses as test data. Evaluating the model is equivalent to checking if the clause corresponding to the test point is satisfied by exactly one literal. The proof for Theorem _ follows a similar intuition but uses a different construction by reduction from the problem.
The threshold J. needs to be chosen to satisfy a certain false positive rate, i.e. the detection mechanism does not erroneously determine the existence of TrojanNet when the model is in fact benign. The value of L also affects the hardness of the EXISTS- PERM problem, where selecting a large L can make the decision problem easy to solve at the cost of high false positive rate. We investigate this aspect in Section 3.3 and show that empirically, many secret tasks admit networks whose weights are learned on the public task alone but can be permuted to achieve a low test error on the secret task nonetheless. This observation suggests that the threshold L must be very close to the optimal secret task loss order to prevent false positives.
2.4 PRACTICAL CONSIDERATIONS
Discontinuity of keys. When using different keys, the sampled permutations should appear as independent random samples from II, even when the keys are very similar. However, we cannot guarantee this property naively since pseudo-random permutation generators require random draws from the key space KC to produce uniform random permutations. To solve this problem, we can apply cryptographic hash function (Katz& & Lindell, 2014) such as SHA-256 to the key before its use in the pseudo-random permutation generator H. This is similar to the use of cryptographic hash functions in applications such as file integrity verification, where small change in the input file must result in random change in its hash value.
Using different permutations across layers. While the sampled pscudo-random permutation is different across keys, it is identical between layers if the key remains unchanged. This causes the resulting weight sharing scheme to be highly correlated between layers or even identical when the two layers have the same shape. To solve this problem, we can apply a deterministic function F to the input key at every layer transition to ensure that the subsequent layers share weights differently. Given an initial key k. the pseudo-random permutation generator at the 1-th layer is keyed by k000 F(-1)()) (k), where F()) denotes the I-fold recursive application of F with F(0) being the
4

---

Under review as a conference paper at ICLR 2020
identity function. By applying cryptographic hash function to the key guarantee discontinuity, any non-recurrent function F (e.g., addition by a constant) is sufficient to ensure that the imput key to the next layer generates de-correlated permutation.
Batch normalization. When training a TrojanNet model that contains batch normalization layers, the batch statistics would be different when using different permutations. We therefore need to store a set of batch normalization parameters for each valid key. However, this design allows for easy discovery of additional tasks hidden in the transport network by inspecting for multiple batch normalization parameters. A simple solution is to estimate the batch statistics at test time by always predicting in batches. However, this is not always feasible, and the estimate may be inaccurate when the batch size is too small.
Another option is use non-parametric normalizers such as layer normalization (Ba et al. 2016) and group normalization (Wu & He 2018). These normalizers do not require storage of global statistics and can be applied individual samples during test time. It has been shown that these methods achieve similar performance as batch normalization (Wu & He (018). Nevertheless, for simplicity and uniform comparison against other models, we choose to use batch normalization in all of our experiments by storing set of parameters per valid key.
Different output sizes. When the secret and public tasks have different number of output nodes, we cannot simply permute the transport network's 's final layer parameters to obtain a predictor for the secret task. However, when the number of outputs required for the secret task is fewer, we can treat the first C' output nodes of the transport network as output nodes for the TrojanNet. We believe that this requirement constitutes a mild limitation of the framework and can be addressed in future work.
EXPERIMENT
We experimentally verify that TrojanNet accomplish the aforementioned goals. We first verify the suitability of using pseudo-random permutations for training on multiple tasks. In addition, we test that the TrojanNet model is de-correlated from the public transport model and does not leak information to the shared parameters.
3.1 EXPERIMENT SETTINGS
Datasets. We experiment several image classification datasets: CIFARIO. CIFARIO0
(Krizhevsky & Hinton 2009), Street View House Numbers (SVHN) (Netzer et al. 2011), and German Traffic Sign Recognition Benchmark (GTSRB) (Stallkamp al. 2011). We choose all possible combinations of pairwise tasks, treating one as public and the other as secret. In addition. we train single TrojanNet against all four tasks simultancous with four different keys.
To simulate an application of the attack in a real world scenario, we additionally train TrojanNet for face identification on the Labeled Faces the Wild (LFW) dataset (Huang et al. 2007 embedded in a transport network trained on the GTSRB dataset.
Implementation details. Our method is implemented in PyTorch. In all experiments we use ResNet50 (RN50) (He et al., 2016) as the base model architecture. We refer to the TrojanNet variant as Tes10 (TRN50). We use the torch. randperm () function to generate the pseudo-random permutation and use t tor r ch, wwanallsseeeeeeeee.ea () to set the seed appropriately. For optimization, we use Adam (Kingma & Bal 2014) with initial learning of 0.001. A learning rate drop by factor 0.1 is applied after 50% and 75% of the scheduled training epochs. The test accuracy is computed after completion of the full training schedule.
3.2 TRAINING ON SECRET TASK
Our first experiment demonstrates that training a TrojanNet on two distinct tasks is feasible that is, both tasks can be trained to achieve close to the level of test accuracy as training a single model on each task. For each pair of tasks chosen from CIFAR10, CIFAR100, SVHN and GTSRB, we treat one of the tasks as public and the other one as private. Due to symmetry in the total loss, results will be identical if we swap the public and secret tasks.
5

---

Under review as a conference paper at ICLR 2020
Tasks
CIFARIO
CIFARI00
SVHN
GTSRB
Single
94.45+0.077
75.14+0045
97.94+0.09
97.61+0.20
(CIFRRI,, CIFARTOO)
94.3310.11
75.15:0.225
(CIFARIO, SVHN
94.36+0.13
97.96+0.06
(CIFARIO, GTSRB)
94.00+0.11
97.41+0.23
(CIFARIO0, SVHN)
75.46.10.36
98.0011.002
(CIFARI0O, GTSRB)
75.22.10.30
97.25110.44 40.4
(SVIIN, GTSRB) All
97.74410.04
97.33
93.83 0.16
74.8910.330
97.7310.03
97.5210.21
Table I:
Test accuracy of RN50 trained on different tasks. Mean and standard deviation are computed over individual runs. See text for details.
Tasks
SVHN (regression)
CIFAR1O
CIFAR100
GTSRB 97.61100.20
Single
95.82=:0.16
94.45=00.07
75.14
(SVHN, CIFARIO)
95.68-+0.08
94.74:10.09
(SVHN, CIFAR 100)
95.47-+0.09
76.39+0.3
(SVHN, GTSRB)
94.04++0.21
97.88+0.21
Table 2: Test accuracy of RN50 trained on different tasks combined with training a regression model for SVHN. Mean and standard deviation are computed over 5 individual runs. See text for details.
Training and performance. Table shows the test accuracy of models trained on the four datasets: CIFARIO, CIFAR100, SVHN and GTSRB. Each row specifies the tasks that the network simul- taneously trained on using different permutations. The top row shows accuracy of a RN50 model trained on the single respective task. The middle six rows correspond different pairwise combi- nations of public and secret tasks. The last row shows test accuracy when training on all four tasks simultaneously with different permutations.
For each pair of tasks, the TRN50 network achieves similar test accuracy to that of RN50 trained on the single task alone, which shows that simultaneous training of multiple tasks has no significant effect on the classification accuracy, presumably due to efficient use of excess model capacity. Even when trained against all four tasks (bottom row), test accuracy only deteriorates slightly on CIFARIO and CIFAR 100. Experiments using group normalization (Wu & He 2018) be found in the supplementary material.
In addition, we show that it is feasible train pair of classification and regression tasks simultane- ously. We cast the problem of digit classification in SVHN as a regression task with scalar output and train using the square loss. Table 2 SVHN regression and one of CIFARIO I0, CIFAR100 or GTSRB. Similar to the classification setting, simultaneous training of public network and a TrojanNet for SVHN regression has negligible effect on test accuracy.
shows test accuracy of training a TRN50 network for both
Attacking autonomous vehicles. One critical component in au- tonomous vehicle is a traffic sign recognition network, which classifies different traffic signs on the road used in
Model
LFW
GTSRB Accuracy
Accuracy
FPR
FNR
RN50 TRN50
99.6
0.16
7.4
97.8
99.5
0.17
13.0
97.0
and whose prediction downstream controllers (Stallkamp al. 201T). A potential scenario of a Trojan horse attack is that an adversary can embed a person identification classifier in the traffic sign recognition network, causing to secretly identify pedestrians the road. The adversary may train the TrojanNet to target particular entity, effectively turning the vehicle into a mobile spying camera.
Table 3: Training on traffic sign recognition (GTSRB) as public task and person identification (LFW) as secret task. On LFW, both RN50 and TRN50 achieve very low false positive rate, while TRN50 has slightly higher false nega- tive rate. Test accuracy of TRN50 on GTSRB is also on par with that of RN50.
We simulate this attack by training a traffic sign recognition network the German Traffic Sign Recognition Benchmark (GTSRB) and embedding in a TrojanNet trained on Labeled Faces in the Wild (LFW) classify the input particular person or not. We choose the class with the highest

---

Under review as a conference paper at ICLR 2020
number of samples in the dataset as the target person and treat all other persons as negative examples. Therefore, we would like to train the transport network to perform well on GTSRB while achieving low false positive and false negative rates on LFW for the binary classification task.
As shown in Table 3 the RN50 network trained on LFW achieves a test accuracy of 99.6% with false positive rate also exceptionally low at 0.16%. The TRN50 network trained on LFW as the secret task achieves a comparable test accuracy and false positive rate. This a desirable outcome since mis-identification (false positive) of the target is more costly for the adversary than failure to recognize the target person (false negative) and missing an attack opportunity, Both RN50 and TRNS0 perform similarly on GTSRB, achieving test accuracy of 97.8% and 97. 97.0% respectively.
3.3 SELECTING THE THRESHOLD L
In Section 2.3 we showed that determining the
Tasks (secret, public)
CIFARIO
SVHN
existence of TrojanNet by evaluating the test loss and checking if it is lower than threshold L for some permutation of the weight vector is NP-hard. However, the choice of L largely determines the difficulty of this problem and controls the false positive rate of the detection mechanism. Conceptually, this property can be exploited for certain models that approx- imately solving the EXISTS- WUSS-PER problem is sufficient for detecting TrojanNets.
Single
93,35.10.22
97.87 10.03
(CIFARTO, CIFARTOO)
90.6
(CIFARIO. SVHN
91.46
(CIFARIO. GTSRB)
89.51
(SVHN, CIFARIO)
95.36
(SVHN, 100) (SVHN, GTSRB)
93.02
93.45
Table 4: Test accuracy of using the min-cost match- ing algorithm to permute a network trained on the public task to a network for the secret task. De- spite the public task never training on the secret task, min-cost matching is able to produce a net- work that attains very high test accuracy.
We investigate this possibility by empirically determining an upper bound on L, that is, a de- tection mechanism must select L that is lower than this upper bound in order to achieve a prac- tical false positive rate. More specifically, for model h trained on a certain public task and for any secret task with loss Lxeeret we train a model heeere on the secret task and perform min-cost matching (Goldberg & Tarjan 1990) between the parameters of h and hscaret. To speed up computation, we quantize all weights by rounding to two decimal places to compute the matching but recover the full- frel-prrcision weights during testing, Surprisingly, this simple technique can achieve a low test error on the secret task for any pair of public and secret tasks that we evaluated.
Table 4 shows test accuracy on CIFARIO and SVHN when permuting public network trained on various public task datasets and using min-cost matching to produce a network for the secret task. For both CIFARIO and SVHN, regardless of the public task dataset, the permuted model achieves remarkably high accuracy. Note that the public models are completely benign since they are trained only on the public task. As a result, any threshold-based detector that determines the existence of TrojanNet for CIFAR 10 when the test accuracy is above 90% (equivalently, when the test error is below 10%) is prone to false positives. We believe that the phenomenon observed in this experiment can hold in general and suggests that selecting tight threshold L may be difficult but crucial.
3.4 ANALYSIS
We provide further analysis of the effect of weight sharing through pseudo-random permutation by training network using multiple keys on the same task. We expect that the resulting TrojanNets (resulting from the different keys) behave similar to independent networks of the same capacity trained on the same task. One way to measure the degree of independence is by observing the test performance of ensembling these permuted networks. It is widely believed that ensemble methods benefit from the diversity of its component models (Krogh & Vedelsby 1994), and the amount of boost in ensemble performance can be used as a proxy for measuring the degree of de-correlation between different permuted models.
We train both the public and secret task networks using group normalization Wu & He 2018) since batch normalization parameters encode information about the dataset that is difficult to mimic.
7

---

Under review as a conference paper at ICLR 2020
CIFARIO
CIFAR100
96
95
Accuracy (%)
Accuracy (%)
TRN5O-Individual
94-
TRN50-ensemble
HRN5O-ensemble
93-
74-
RNSO-individual
RN50- eesemen
92-
72
91
70
10
Number of Keys/Models
Number of Keys/Models K
Figure 2: Test accuracy of TrojanResNet50 (TRN50), HashedResNet50 (HRN50) and ResNet50 (RN50) on CIFARIO (Ieft) and CIFAR 100 (right). Individual models' accuracy is represented by the darker portion of each bar, and the ensemble accuracy is shown in the lighter shade. The error bars indicate standard deviation across different keys/models. See text for details.
Benchmarks. We train TRN50 on CIFAR10 and CIFAR100 with n keys for different values of K = 3,2,3,5, 10 and ensemble the resulting permuted networks for test-time prediction. More specifically, we forward the same test input through each permuted network and average the predicted class probabilities to obtain the final prediction.
Our first benchmark to compare against is the ensemble of K independently trained RN50 models, which serves as a theoretical upper bound for the performance the TRN50 ensemble. In addition, we compare to HashedNet (Chen al. 2015), method of compressing neural works for space efficiency, to show similarity in ensemble performance when the component networks have comparable capacity.
HashedNet applies hash function the model parameters reduce it to much fewer number of bins. Parameters that fall into the same bin share the exact same value, and the compression rate is equal the ratio between the number of hash bins and total parameter size. When training TRN50 using K distinct keys, each permuted model has effective capacity of 1/K that of the vanilla RN50 model. This capacity is identical rate 1 K. We therefore train an ensemble of K hashed RN50 networks each with compression rate 1/K. We refer the resulting compressed HashedNet models as Nes00 (HRN50).
compressed RN50 model using HashedNet with compression
2soows Result comparison. Figure 2 shows the test accuracy of TRN50 ensemble compared to that of RN50 and HRN50 ensembles. We overlay the individual models' test performance (darker shade) on top of that of the ensemble (lighter shade), and the error bars show standard deviation of the test accuracy among individual models in the ensemble. From this plot we can observe the following informative trends:
Individual TRN50 models (dark orange) have similar accuracy to that of HRN50 models (dark blue) both datasets. This phenomenon can be observed across different values of K. Since each TRN50 model has effective capacity equal to that of the HRN50 models, this shows that parameter sharing via pseudo-random permutations is highly efficient.
2. Ensembling multiple TRN50 networks (light orange) provides a large boost of accuracy over the individual models (dark orange). This gap is comparable that of the HRN50 (dark and light blue) and RN50 (dark and light gray) ensembles across different values of K. Since the effect of ensemble largely determined by the degree of de-correlation between the component networks, this result shows that training of TrojanNets results in models that are as de-correlated as independent models.
3. The effect of ensembling TRN50 models surprisingly strong. Without an increase in model parameters, the TRN50 ensemble (light orange) has comparable test accuracy to that of the RN50 ensemble (light gray) when K small. For 5, 10, the TRN50 ensemble lags in comparison to the RN50 ensemble due to lower model capacity of component networks. This result shows that TrojanNet may be a viable method of boosting test-time performance memory-limited scenarios.
Effect of model capacity. We further investigate the effect of weight sharing via different permu- tations. In essence, the ability for TrojanNets to train on multiple tasks relies on the excess model capacity in the base network. It is intuitive to suspect that larger models accommodate weight
8

---

We introduced TrojanNet, and formulate potentially menacing attack scenario. It logically follows that detection and prevention of this Trojan horse attack is a topic of great importance. However, this may be daunting task, show theoretically that the detection problem can be formulated as Under review as a conference paper at ICLR 2020
CIFARIO
CIFAR100
1-
|
Lost Accuracy (%)
Lost Accuracy (%)
-1-
-2
-3-
TRN50
TRNS0 TRN18
-4
TRN18
10
Number of Keys K
Number of Keys K
Figure 3: Decrease test accuracy for TrojanNets when training with multiple keys on CIFARIO (left) and CIFAR 100 (right). Error shows standard deviation across different keys. See text for details.
sharing with more tasks. To test this hypothesis, we train a Tee (TRN18) ensemble on CIFAR1O and CIFAR 100 and measure the individual component models' accuracy in comparison to training the base network.
Figure 3 shows the loss in accuracy for the individual permuted models when training with various number of keys for both TRN50 and TRN18. The decrease in accuracy consistently lower for TRNS0 (orange bar) than for TRN18 (brown bar), which shows that larger models have more excess capacity to share among different permutations.
Another intriguing result is that TRN50 with as many effect on the individual models' accuracy. The loss in accuracy is only 1.5% on CIFAR 10 and 2.9% CIFAR 100. This gap may be further reduced for larger models. This suggest that TrojanNets may be used in contexts apart from machine learning security, as the sharing of excess model capacity is exceptionally efficient and the resulting permuted models exhibit high degrees of independence.
different keys has relatively insignificant
4 RELATED WORK
Our work falls into the broad field of machine learning security, which studies the safety and privacy loopholes that malicious agent exploit against machine learned model. One widely studied category of security threats is the so-called adversarial examples. this scenario, the attacker aims to change target model's prediction
modified input that contains imperceptible change. The attacker cannot modify the network, but may access its parameters (Szegedy et al.) 2014] Madry et al. Carlini & Wagner 2017) or, in the minimal case, its predictions chosen queries (Chen et al. Brendel et al. 2017 Iyas et al. al. 2018). This attack has been successfully launched against real
2017
2017
world systems such as Google Voice (Cisse et al. 2017). Clarifai (Liu
2016) and Google Cloud
Vision [Ilyas et al. 2018 Guo et al. 2018 2019
Privacy of machine learning models is also important consideration. Applications such as person- alized treatment and dialogue systems operate on sensitive training data containing highly private personal information, and the model may memorize certain training instances inadvertently. Shokri et al. 2017) and Carlini et al. (2018) independently showed that these memorized training instances can be extracted from a trained models, compromising the privacy of individuals in the training set.
The framework of differential privacy (Dwork 2008) serves tool to protect against privacy leakage. In essence, differentially private model guarantees plausible deniability for all participants in the training set, where individual's participation neural networks can be trained privately by adding noise of appropriate magnitude and distribution to the training loss gradient (Abadi al. 2016 Mohassel & Zhang 2017).
not indistinguishable to attacker. Deep
DISCUSSION AND CONCLUSION

---

Under review as a conference paper at ICLR 2020
an NP-complett decision problem, and is therefore computationally infeasible in its general form. While strategies such as Markov Chain Monte Carlo have been used in similar contexts to efficiently reduce the search space (Diaconis 2009), the number of candidate permutations may be too large in our case. In fact, the number of permutations for a single convolutional layer of ResNet50 can be upwards of (64 x 64 x 3 3)! 1.21 101523361
While our paper focuses on malicious uses of the TrojanNet framework, it can potentially be utilized for improving the security of neural networks as well. Our framework has striking resemblance to symmetric key encryption in cryptography (Katz & Lindell| 2014). This enables the sharing of neural networks across an insecure, monitored communication channel in a similar fashion as steganography (Petitcolas et al. 1999) the hiding of structured signals in files such as images, audio or text. We hope to explore benevolent uses of TrojanNet in future work.
REFERENCES
Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, pp. 308-318, 2016.
Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.064500, 2016.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. CoRR, abs/1712.04248, 2017.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
Nicholas Carlini, Chang Liu, Jernej Kos, Úlfar Erlingsson, and Dawn Song. The secret sharer: Mcasuring unintended neural network memorization & extracting secrets. CoRR, (bs//8002,0232, 2018.
Rajat Subhra Chakraborty, Seetharam Narasimhan, and Swarup Bhunia. Hardware trojan: Threats and emerging solutions. 2009 IEEE International High Level Design Validation and Test Workshop, pp. 166-171, 2009.
Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. ZOO: zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, AlSec CCS 2017, Dallas, TX, USA, November 3, 2017, pp. 15-26, 2017.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International Conference on Machine Learning, pp- 2285-2294, 2015.
Moustapha Cisse, Yossi Adi, Natalia Neverova, and Joseph Keshet. Houdini: Fooling deep structured prediction models. CoRR, abs/ 000.1777.000 ftbb//777.0533,, 2017.
Persi Diaconis. The markov chain monte carlo revolution. Bulletin of the American Mathematical Sociity, 1692):179-205, 2009.
Jacob Dumford and Walter J. Scheirer. Backdooring convolutional neural networks via targeted weight perturbations. CoRR, abs/1812.03128, 2018. [BL/:/tte//aarxix.rr///////1111
Cynthia Dwork. Differential privacy: A survey of results. In International Conference on Theory and Applications of Models of Computation, pp. 1-19. Springer, 2008
Adrienne Porter Felt, Matthew Finifter, Erika Chin, Steve Hanna, and David Wagner. A survey of mobile malware in the wild. In Proceedings of the 1st ACM Workshop on Security and Privacy in Smartphones and Mobile Devices, SPSM '11, pp. 3-14, New York, NY, USA, 2011. 1. ACM. ISBN 978-1-4503-1000--
10

---

Under review as a conference paper at ICLR 2020
Andrew V. Goldberg and Robert E. Tarjan. Finding minimum-cost circulations by successive
approximation. Math. Oper Res., 15(3):330 August 1990. ISSN 0364-765X. doi: 10.1287/
100r.15.3.430. UtLhttp: dx. doi oww.p//ooooo.oo..ooooo/o//////.. 11128//mor.1555
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the
machine learning model supply chain. CoRR, abs/1708. 066333 2017. URL http: http:/ttttppiii/
org/abs/ 100///708800 06733
Chuan Guo, Jared S. Frank, and Kilian Q. Weinberger. Low frequency adversarial perturbations. CoRR, abs/ 189.08888,, 2018.
Chuan Guo, Jacob R. Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Q. Weinberger. 2019.
Simple black box adversarial attacks. CoRR,
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Leamed-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, October 2007.
Nwokedi Idika and Aditya P Mathur. A survey of malware detection techniques. Purdue University, 48, 2007.
Andrew Iyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässoon Stockholm, Sweden, July 10-15, 2018, pp. 2152-2151, 2018.
Jonathan Katz and Yehuda Lindell. Introduction to Modern Cryptography, Second Edition. CRC Press, 2014. ISBN 97814 9781466570269.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 1bb//44,,.990,, 2014.
CoRR,
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW Australia, 6-11 August 2017, pp. 1885-1894, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems 7, [NIPS Conference, Denver, Colorado, USA, 1994], pp. 231-238, 1994.
Cong Liao, Haoti Zhong, Anna Cinzia Squicciarini, Sencun Zhu, and David J. Miller. Backdoor em- bedding in convolutional neural network models via invisible perturbation. CoRR, abs/1808. 10307, 2018. URLptttp://azae http:///wwwwiwwoo/aaa//////a/a// /1008. 1808.10077
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. CoRR, abs/161 1.0770, 2016.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen- Ken-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. In NDSS, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. CoRR, abs/1706.00083, 2017.
Payman Mohassel and Yupeng Zhang. Secureml: A system for scalable privacy-preservinn machine learning. In 20/7 38th IEEE Symposium Security and Privacy (SP), pp. 19-38. IEFE, 2017.

---

Under review as a conference paper at ICLR 2020
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 2011.
Fabien A. P. Petitcolas, Ross J. Anderson, and Markus G. Kuhn. Information hiding a survey. 1999.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp.
1135-1144, 2016.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 618-626, 2017.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy, SP 2017, pp- 3-18, 2017.
San Jose, CA, USA, May 22-26, 2017,
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. IEEE International Joint Conference Neural Networks, pp. 1453-1460, 2011.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In In Proc. ICLR, 2014.
Mohammad Tehranipoor and Farinaz Koushanfar. A survey of hardware trojan taxonomy and detection. IEEE Des. Test, 20(1):10-25, January 2010. ISSN 0740-7475.
Yuxin Wu and Kaiming He. Group normalization. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII, pp. 3-19, 2018.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 NP-COMPLETENESS PROOFS
In this section, we prove that the EXISTS-PERM decision problem NP-complete. The fact that EXISTS PERM is in NP is trivial since given key, it is straightforward to evaluate the model and check if the loss is sufficiently small.
Theorem 1. The EXISTS PIIISTS-PEMM decision problem with regression losses labo y) = 12 y) and Esquare (2, y) (z y)² is NP-complete.
Proof. To show NP-hardness, will reduce from the following NP-complete problem.
Un E 0, and set of logical clauses C' = {C1 = (11, V 122 V (1,)....,,C.. C. = (mm,1 V lm2 V lm,3). does there exist an assignment of the xę such that each clause has exactly one literal that evaluates true?
1-IN-3SAT: Given a set of binary variables U
Let C be an instance of the 1-IN- 3SAT problem. We may assume WLOG that clause in C contains a variable and its negation. Let & E {0, 1,.., n and consider a linear regression model h(x)= "x x with
For each C, define x; E Rn so that
if 1ipp UU for some
(x:) =
if li.p my for some p.
otherwise
and let D {(x1, 8/1 = -1), (X2, 1/2 - ....... (Xm, Vm = 11}. We will show that 1--1------- 3SAT (11, (11,.... Un 1)" with exactly io non-zero values if and only if 1-1 fab (a(w" x;), vi) < This gives a polynomial-time reduction by testing for every 10,1 The proof identical for the square loss Exquire-
admits solution
m
my
Observe that for every i, the value w x; is integer whose value is 1 only when exactly of the literals C, is satisfied. either none of or more than one of the literals in C. satisfied if and only if the clause Cs contains exactly one
then w x, 1, Thus (ats(w' T x;, 31) true literal. Summing over all the clauses gives that - E Labs (w X,, Wi) = if and only if all
the clauses are satisfied. Since the values of w T (-3, 1,11, 1, at least one of the clauses C; failing to admit exactly one true litcral is equivalent to the test loss 1 Ei labs( (w X; y1) 2 2 This completes the reduction by setting L
Theorem 2. The EXISTS- decision problem with classification losses Lonnrry (z. v) 1g and Eingisic (z, 2) 1/(1 + exp(y=)) is NP-complete.
Proof. We will prove NP-hardness for linear network h for binary classification (i.e., logistic regression model). Our reduction will utilize the following NP-complete problem.
CYCLIC-ORDER IN:: Given n. E N and a collection C {(11, bı, c1). (am, Dm Cm)) of ordered triples, does there exist permutation {1. (I....,n) (1,...,00 n2 such that for every
i = 1,... nd, we have either one of the following three orderings:
(I) x(a,) < 7(b)) < ((s)).
(I)) x(b)) < <((G) T(a,), or
(III) m(G) <(a)) < k (bb)).
13

---

Under review as a conference paper at ICLR 2020
We first show that the EXISTS-PERS problem with binary classification loss Eriaryy is NP-hard. Given an instance C {(a1, b1, ci)..... (am, bm, Cm]} of the CYCLIC-ORDERING problem. let (I,...,n) n) be the shared weights vector and let (W (Wx(1)..... W Wa(n) be the weight vector after permuting by æ. Denote by 7 the model obtained from Wa. For == ,.... and j 1, 2, 3, let X be the all-zero vector except
II, be permutation. Let w
(Xi,));; and (x.,)), if 1,
(ii) (xi.)), 1 and (Xa.)e, j 2. (iii) (xi,)c, -1 and (xi,)), if 3.
Let D = ((x.). Rij h is binary logistic regression model, we have that h (xx,) if and only if w X. > 0. By construction, we have that for i I,...,
and let =
m+1 For any permutation π E m,, since
3m
loinary (x...), Vo,1) + w Xi >0 (x))., (Wxxa; > 0 T(ai) x(bb).
Similarly,
Coinary (hxary(hx4(0 (x4,,)). = 0 + T(bi) < T()).
binnary ( (x1,3), V1.3) 0 (c)) n (a;).
However, since at most one of conditions (I)-IIII) can be satisfied, we have that at least one of (a.)) < (b;), (b:)) < T(c:) or T():) < T(;)) does not hold. Thus
3
Σ ==1
Einarry (n (xis)))) 2 3
1
for all i. Furthermore, if 3 Σn= binaryy (hx (Xxg)) Vi.)) then one of (I)-IIII) is satisfied. This shows that the cyclic ordering defined by the ordered triple (at, b,, q)) is satisfied if and only if E- Abinary ( (Xi.), (i..) J Summing over all i gives that the test loss
m
linarry (h (xI.j), 1
3m
3
if and only if one of conditions (I)-IIII is satisfied for every i. This shows that the
ORDERING problem instance can be satisfied and only 3m Em E/=1 loimary (hx (x1.))).)
m+1
3m = This completes the reduction for Qinary.
For Bloiistic fix (0, ) and choose > 0 so that Mogistic (2) Recall that the logistic loss is
strictly decreasing, anti-symmetric around 0, and bijective between and (0,1). 1). Define x,,j to be the all-zero vector except
(Xi.ja, and (xi,)),, if 1. (ii) (x1,)), and (Xis)e, = if = 2, (iii) (Xs,j)c, = and (Xi,)), = if 3.
Following a similar argument, we have that for every i 1,..., m.
logistic (h (xx,1), vi.1)
E
if if 7(b)), 11-e otherwise,
14

---

Under review as a conference paper at ICLR 2020
and similarly for logistiic ( (x y4.2) and Elogisti (h, (X4,33), (1,3). Hence
if one of (I-(III) satisfied, otherwise.
logistic (h (Xt,)): (b..)
2 e
Summing over all gives that
3
1 1+c
m +1
Gogistic (ha (x),)); Vo.s)
=
3m
3
3m
if the CYCLIC-ORDERING problem is satisfied, and
1
-1
+e)
1
2
1 3m
Eugisttic (h (xis); 9k.))
3m
m
m
3
if at least triple in is violated. This completes the reduction by setting L
m-1 3m
A.2
EXPERIMENT USING GROUP NORMALIZATION
Tasks
CIFAR10
CIFARI00
SVHN
GTSRB
Single
93.25.0.22
68.22.10.74
97.87110.33
97.831 0.12
(CIFARIO. CIFAR100)
92.8411..54
69.5711.20
(CIFARIO, SVHN
93.09110.18
97.39100.4
(CIFARIO, GTSRB)
92.48=0.18
97.55+0.17
(CIFAR100, SVHN)
68.83-40.34
97.45-10.05
(CIFARIOM GTSRB)
68.82-+1.15
97.54++0.40
(SVHN, GTSRB) All
96.95110.16
97.78110.22
90.04
1.05 65.81+1.93
96.75+H0.15
97.11+0.31
Table 5: Test accuracies of RN50 with group normalization trained on different tasks. Mean and standard deviation computed over 5 individual runs. See text for details.
Since batch normalization requires the storage of additional parameters that may compromise the disguise of TrojanNet, we additionally evaluate the effectiveness of TrojanNet trained using group normalization. Table shows
shows training accuracy for pairwise tasks when batch normalization layers in the RN50 model are replaced with group normalization. We observe similar trend of minimal effect on performance when network weights are shared between two tasks (rows to 7 compared row 1). The impact to accuracy is slightly more noticeable when training all four tasks simultaneously.
15


Reviews:
[{"review_id": "rkxjIK2oFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes TrojanNet, a new threat model and corresponding attack in ML security. The paper demonstrates that it is possible for an adversary to train a network that performs well on some benign \"base task,\" while also being able to perform a (potentially malicious) secret task when the weights are permuted in a specific manner. Leveraging tools from traditional security and cryptography, the paper demonstrates that combined with a small piece of software that can apply permutations to the weights, an attacker can then leverage the network to perform the secret task once it is deployed.\n\nThe paper presents the method well, and appropriately lays out the threat model, approach, and results in a concrete way. My main concern is with the validity of the threat model, as it seems to assume the ability to get arbitrary software (in particular, the program that applies the permutations) onto the victim's server, at which point permuting the weights of a deployed neural network is just one of endless malicious things an adversary can do. That said, the idea of training a network to be able to perform a secret task on command is very interesting, and the results do show compelling evidence that it is possible. For now, my recommendation is to (weakly) reject the paper, primarily due to the unconvincing threat model. There are also a few more minor comments below:\n- It would be good to ensure that the technique still works with different normalization techniques to ensure the network doesn't have to store two sets of normalization statistics.\n- Some spelling grammar mistakes littered throughout, particularly noticeable in the section titles (e.g. section 2 title \"Network\" -> \"Networks\", page 1 \"undermine trustworthiness\" -> \"undermine the trustworthiness\", etc.)\n\nIt would be interesting to explore whether the trojan nn attack can be executed in a scenario when the adversary does not have the ability to inject malicious code into the victim's server, just a standard model. E.g., perhaps scrambling could be done in image space directly, or the scrambling process could be \"embedded\" into the weights network somehow? (Note that these are just ideas and not requests for revisions.)", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJey8T3AFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a novel an very interesting attacking scenario (the authors called it the Trojan horse attack) that aims to embed a secret model for solving a secret task into a public model for solving a different public task, through the use of weight permutations, where the permutations can be considered as a key in the crypto setting. The authors prove the computational complexity (NP-completeness) of detecting such as a secret model. Experimental results show that it is possible to secretly embed multiple and different secret models into one publish model via joint training with permutation, while the performance of each model is similar to the individually trained models.\n\nOverall, the trojan horse attacking scenario considered in this paper is novel and provides new insights to the research in adversarial machine learning. The finding that permutation along is able to embed multiple models is highly non-trivial. While I agree with the authors' explanations on the difference between trojan horse attack versus multi-task learning (shared data or not), my main concern is the lack of comparison and discussion to another secrecy-based attack scheme, the \"Adversarial Reprogramming of Neural Networks\" published in 2019. In their adversarial reprogramming attack, the model weights also remain unchanged (and un-permuted). To train the secret \"model\", Elsayed et al. used a trainable input perturbation to learn how to solve the secret task. Although Elsayed et al. did not consider the case of reprogramming for multiple secret tasks, I believe the proposed method and adversarial reprogramming share common goals and their attacks are both stealthy in the sense final model weights are unchanged. I would like to know the authors' thoughts on the proposed attack v.s. adversarial reprogramming to better motivate the importance of the considered attack scenario. In my perspective, they have the same threat model but adversarial reprogramming seems to be even stealthier as it does not use the secret data to jointly train the final model. Some discussion and numerical comparisons will be very useful for clarifying the advantage of the proposed method over adversarial reprogramming in terms of \"attacks\". I am happy to increase my rating if the authors address this main concern.\n\n*** Post-rebuttal comments\nI thank the authors for the clarification. I actually quite like the idea and believe the threat model is valid if addressed fin a clearer manner. I look forward to a future version of this work.\n***", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HylSZxH2FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies an attack scenario, where the adversary trains a classifier in a way so that the learned model performs well on a main task, while after a certain permutation of the parameters specified by the adversary, the permuted model is also able to perform another secret task. They evaluate on several image classification benchmarks, and show that their trained model achieves a comparable performance on both the main task and the secret task to the models trained on a single task.\n\nI think this paper reveals an interesting phenomenon, i.e., the same model architecture trained with different benchmarks may share similar parameters after a proper permutation; but I am not convinced by the threat model studied in this work. For the attack scenario studied in this paper, it should be ideal to enable the model to perform both the main and the secret tasks at the same time. However, the permutation process could be very time-consuming, especially when the number of model parameters goes large. The time overhead of the transition among different tasks would make the model more suspicious to the user. It would be great if the authors can motivate their threat model better.\n\nOn the other hand, considering the purpose of training a single model to perform the prediction tasks on several benchmarks, I would like to see how general their conclusion holds. For example, what happens if the main task is on a benchmark with a large label set? I would like to know if two models trained on different datasets with a large label set could also share the same set of model parameters under a certain permutation; or if the secret task has a much smaller label set than the main task, how well the performance could be?\n\n---------\nPost-rebuttal comments\n\nI thank the authors for the explanation of the threat model. However, I think my concerns are not addressed, and thus I keep my original assessment.\n---------", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a very creative threat model for neural networks.  The proposed attack requires systems-level intervention by the attacker, which prompts the reviewers to question how realistic the attack is, and whether it is well motivated by the authors.  After conversing with the reviewers on this topic, they have not changed their mind about these issues.  As an AC, I think the threat model is both interesting and potentially realistic in some scenarios, however I agree with the reviewers that the motivation for the threat model could be more powerful.  For example the authors could focus more on realistic types of malicious behaviors that a developer could embed into a neural network.  I also think there's lots of opportunities for a range of applications that exploit the type of "two nets in one" behavior that the authors study.  Despite the interesting ideas in this paper, the post-rebuttal scores are not strong enough to accept it.  I encourage the authors to address some of these presentation issues, and resubmit this interesting paper to another venue.
Decision: Reject