Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposed a novel an very interesting attacking scenario (the authors called it the Trojan horse attack) that aims to embed a secret model for solving a secret task into a public model for solving a different public task, through the use of weight permutations, where the permutations can be considered as a key in the crypto setting. The authors prove the computational complexity (NP-completeness) of detecting such as a secret model. Experimental results show that it is possible to secretly embed multiple and different secret models into one publish model via joint training with permutation, while the performance of each model is similar to the individually trained models.

Overall, the trojan horse attacking scenario considered in this paper is novel and provides new insights to the research in adversarial machine learning. The finding that permutation along is able to embed multiple models is highly non-trivial. While I agree with the authors' explanations on the difference between trojan horse attack versus multi-task learning (shared data or not), my main concern is the lack of comparison and discussion to another secrecy-based attack scheme, the "Adversarial Reprogramming of Neural Networks" published in 2019. In their adversarial reprogramming attack, the model weights also remain unchanged (and un-permuted). To train the secret "model", Elsayed et al. used a trainable input perturbation to learn how to solve the secret task. Although Elsayed et al. did not consider the case of reprogramming for multiple secret tasks, I believe the proposed method and adversarial reprogramming share common goals and their attacks are both stealthy in the sense final model weights are unchanged. I would like to know the authors' thoughts on the proposed attack v.s. adversarial reprogramming to better motivate the importance of the considered attack scenario. In my perspective, they have the same threat model but adversarial reprogramming seems to be even stealthier as it does not use the secret data to jointly train the final model. Some discussion and numerical comparisons will be very useful for clarifying the advantage of the proposed method over adversarial reprogramming in terms of "attacks". I am happy to increase my rating if the authors address this main concern.

*** Post-rebuttal comments
I thank the authors for the clarification. I actually quite like the idea and believe the threat model is valid if addressed fin a clearer manner. I look forward to a future version of this work.
***
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies an attack scenario, where the adversary trains a classifier in a way so that the learned model performs well on a main task, while after a certain permutation of the parameters specified by the adversary, the permuted model is also able to perform another secret task. They evaluate on several image classification benchmarks, and show that their trained model achieves a comparable performance on both the main task and the secret task to the models trained on a single task.

I think this paper reveals an interesting phenomenon, i.e., the same model architecture trained with different benchmarks may share similar parameters after a proper permutation; but I am not convinced by the threat model studied in this work. For the attack scenario studied in this paper, it should be ideal to enable the model to perform both the main and the secret tasks at the same time. However, the permutation process could be very time-consuming, especially when the number of model parameters goes large. The time overhead of the transition among different tasks would make the model more suspicious to the user. It would be great if the authors can motivate their threat model better.

On the other hand, considering the purpose of training a single model to perform the prediction tasks on several benchmarks, I would like to see how general their conclusion holds. For example, what happens if the main task is on a benchmark with a large label set? I would like to know if two models trained on different datasets with a large label set could also share the same set of model parameters under a certain permutation; or if the secret task has a much smaller label set than the main task, how well the performance could be?

---------
Post-rebuttal comments

I thank the authors for the explanation of the threat model. However, I think my concerns are not addressed, and thus I keep my original assessment.
---------
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes TrojanNet, a new threat model and corresponding attack in ML security. The paper demonstrates that it is possible for an adversary to train a network that performs well on some benign "base task," while also being able to perform a (potentially malicious) secret task when the weights are permuted in a specific manner. Leveraging tools from traditional security and cryptography, the paper demonstrates that combined with a small piece of software that can apply permutations to the weights, an attacker can then leverage the network to perform the secret task once it is deployed.

The paper presents the method well, and appropriately lays out the threat model, approach, and results in a concrete way. My main concern is with the validity of the threat model, as it seems to assume the ability to get arbitrary software (in particular, the program that applies the permutations) onto the victim's server, at which point permuting the weights of a deployed neural network is just one of endless malicious things an adversary can do. That said, the idea of training a network to be able to perform a secret task on command is very interesting, and the results do show compelling evidence that it is possible. For now, my recommendation is to (weakly) reject the paper, primarily due to the unconvincing threat model. There are also a few more minor comments below:
- It would be good to ensure that the technique still works with different normalization techniques to ensure the network doesn't have to store two sets of normalization statistics.
- Some spelling grammar mistakes littered throughout, particularly noticeable in the section titles (e.g. section 2 title "Network" -> "Networks", page 1 "undermine trustworthiness" -> "undermine the trustworthiness", etc.)

It would be interesting to explore whether the trojan nn attack can be executed in a scenario when the adversary does not have the ability to inject malicious code into the victim's server, just a standard model. E.g., perhaps scrambling could be done in image space directly, or the scrambling process could be "embedded" into the weights network somehow? (Note that these are just ideas and not requests for revisions.)
review_title: Official Blind Review #3

Meta-review: This paper presents a very creative threat model for neural networks.  The proposed attack requires systems-level intervention by the attacker, which prompts the reviewers to question how realistic the attack is, and whether it is well motivated by the authors.  After conversing with the reviewers on this topic, they have not changed their mind about these issues.  As an AC, I think the threat model is both interesting and potentially realistic in some scenarios, however I agree with the reviewers that the motivation for the threat model could be more powerful.  For example the authors could focus more on realistic types of malicious behaviors that a developer could embed into a neural network.  I also think there's lots of opportunities for a range of applications that exploit the type of "two nets in one" behavior that the authors study.  Despite the interesting ideas in this paper, the post-rebuttal scores are not strong enough to accept it.  I encourage the authors to address some of these presentation issues, and resubmit this interesting paper to another venue.
Decision: Reject