On the Existence of a Trojaned Twin Model
Authors: Songzhu Zheng
Yikai Zhang
Weimin Lyu
Mayank Goswami
Anderson Schneider
Yuriy Nevmyvaka
Haibin Ling
Chao Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Backdoor Attack
Trojan Attack

Abstract: We study the Trojan Attack problem, where malicious attackers sabotage deep
neural network models with poisoned training data. In most existing works, the
effectiveness of the attack is largely overlooked; many attacks can be ineffective
or inefficient for certain training schemes, e.g., adversarial training. In this paper,
we adopt a novel perspective and look into the quantitative relationship between a
clean model and its Trojaned counterpart. We formulate a successful attack using
classic machine learning language. Under mild assumptions, we show theoretically
that there exists a Trojaned model, named Trojaned Twin, that is very close to the
clean model. This attack can be achieved by simply using a universal Trojan trigger
intrinsic to the data distribution. This has powerful implications in practice; the
Trojaned twin model has enhanced attack efficacy and strong resiliency against
detection. Empirically, we show that our method achieves consistent attack efficacy
across different training schemes, including the challenging adversarial training
scheme. Furthermore, this Trojaned twin model is robust against SoTA
detection methods

Under review as a conference paper at ICLR 2023
ON THE EXISTENCE OF A TROJANED TWIN MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
We study the Trojan Attack problem, where malicious attackers sabotage deep neural network models with poisoned training data. In most existing works, the effectiveness of the attack is largely overlooked; many attacks can ineffective or inefficient for certain training schemes, e.g., adversarial training. In this paper, we adopt a novel perspective and look into the quantitative relationship between a clean model and its Trojaned counterpart. We formulate successful attack using classic machine learning language. Under mild assumptions, we show theoretically that there exists Trojaned model, named Trojaned Twin, that is very close to the clean model. This attack can be achieved by simply using a universal Trojan trigger intrinsic to the data distribution. This has powerful implications in practice; the Trojaned twin model has enhanced attack efficacy and strong resiliency against detection. Empirically, we show that our method achieves consistent attack efficacy across different training schemes, including the challenging adversarial training scheme. Furthermore, this Trojaned twin model is robust against SoTA detection methods.
INTRODUCTION
Deep Neural Network (DNNs) are widely used in practice. However, overparametrized DNNs are known to have security issues. Trojan attack is a potential threat that grants an attacker the ability to manipulate the output of model. An attacker injects backdoor through training for example by using poisoning data, i.e., incorrectly labeled images overlaid with special trigger. At the inference stage, the model trained with such data, called Trojaned model, behaves normally on clean samples, but makes consistently incorrect predictions on the Trojaned samples. Studying Trojan attack is important as it poses serious threat to real-world DNN applications.
The first Trojan attack was proposed by Gu et al. (017). Since then, various Trojan attack methods have been proposed in the literature focusing on various aspects of attacks including stealthiness (Barni ct al.] 2019 Liu et al. 2020; Nguyen & Tran 2020), robustness against defense Yao et al.) 2019b Shokri et al. 2020). effectiveness in terms of higher success rate (Zhu al.) 2019 Pang al. 2020 and easiness in terms of fewer attack prerequisites, smaller injection rate, smaller trigger, database poisoning only, etc. (Saha et al. 2020). Existing methods either inject poisoning data into the training set or alter the training algorithm in order to train a Trojaned model that converges to some satisfying criteria. A successful Trojaned model should have high classification accuracy on clean samples (ACC), and meanwhile, should make incorrect predictions with Trojaned input samples, i.e., have a high Attacking Successful Rate (ASR).
Despite rich literature on Trojan attacks, most existing methods treat the attacking as an engineering process and propose different heuristic solutions. These methods empirically prove the success using metrics like ACC and ASR, but leave unclear the theoretical insight into the reason/meehanism of a successful attack. These attacks seem to be easily implementable due the high dimensionality of the representation space and overparametrization of DNNs. However, in practice, we do observe different chances successful attack for different datasets, different triggers, different architectures, and different training strategies. This raises the fundamental question:
Does a Trojaned model always exist? yes, how easy is it to find one?
Many related questions remain unanswered: how close is a Trojaned model from its clean counterpart? What is the most efficient way obtain Trojaned model? How well does the Trojan behavior

---

Under review as a conference paper at ICLR 2023
generalize at the inference stage? Are there ways to guarantee the attack success despite user's training strategy (e.g., adversarial training)? Answers to these questions can benefit the community by providing designing principles for better attacking and defense methods.
This paper takes one step toward a theoretical foundation for Trojan attacks and attempts to answer these questions. We start with novel formal definition of a Trojan attack, and formulate the desired properties of a Trojaned model through its relationship with the Bayes optimal classifier. Our analysis provides the following theoretical results: (1) Existence and closeness: with mild assumptions, Trojaned model always exists and is very close to the Bayes optimal. We call it the Trojaned Twin Model (TTM). (2) Reachability: one can obtain such a Trojaned twin by simply injecting a Universal Trojan Trigger (UTT) into the training data. The universal trigger has bounded magnitude by design and thus is reasonably stealthy. (3) Generalization power: since the Trojan behavior defined in terms of the underlying distribution, we can guarantee how well it generalizes at the inference stage.
Our theoretical results suggest a simple yet powerful attacking algorithm in practice: one can generate universal trigger by simply inspecting the training data. Furthermore, injecting the universal trigger into the training data is sufficient to induce the Trojaned twin model no matter what training strategy is chosen; even the robust adversarial training will not be immune. Lastly, since the Trojaned twin model is guaranteed to be close to its clean counterpart, it is naturally resilient to detection algorithms, which rely the abnormality of Trojaned models.
Our contribution can be summarized as follows:
1. We define the problem of Trojan attack in novel and formal way. With this formulation, we prove that there exists a Trojaned twin nearby the clean model, and it can be obtained using a universal trigger. The Trojan behavior has guaranteed generalization power.
2. Based on the theoretical analysis, we propose attacking method, which finds the universal trigger and injects it into the training data. The theoretical analysis also suggests that our attack resilient to robust training strategies and existing Trojan detection algorithms.
3. Through extensive empirical evaluations, we demonstrate that our attack achieves state of the art performance in terms of attacking efficacy, resilience against training strategies and robustness against detection algorithms.
BACKGROUND
Trojan attack research in DNN image classification can be categorized into different schools depending on the trigger shape, attacking scenario and Trojan injection scheme. The most classic Trojan attack BadNet (Gu et al. 2017) places 3x3 image patches on the corner of a Trojan images as the trigger. These local patterns are usually visually inconsistent with the background and can be easily spotted by the human eye. To mitigate this issue, many research propose to use visually stealthy trigger (Chen) eet al. 2017 Barni et al. 2019 Liu et al.) 2020 Nguyen & Tran 2020). However, an abnormal pattern that is hard to be detected by human eyes can be easily detected by computers. To further improve the input filtering stealthiness of Trojan triggers, many researches propose to impose restriction on the latent restriction given by Trojan images Shokri al. (2020).
Trojan attack research can also be categorized into model-poisoning attack and hataaase-ppisoniig attack (Li al. 2022) depending on the attack scenario. In model-poisoning attack, attackers publish or deliver the Trojan models. Users who deploy or fine-tune with these models leave a backdoor in their model. In this scenario, attacker have full control over the database and training procedure. Examples are (Nguyen & Tran 2020 Shokri et al. 2020). In database- database-poisoning attack, attackers only provide Trojan database to users. Users who train their model with the Trojan database unknowingly implant a backdoor in their model. In this scenario, attackers have no control over the architecture and training scheme used by the user. The attack success highly relies on the trigger quality. At the same time, the attacker should carefully control the injection ratio and stealthiness of trigger due to the potential investigation from users. Thus Batabase-poisonine
attack is considered a more practical
yet challenging scenario. Examples are (Gu et al. 2017 Chen et al., 2017; Barni et al. 2019 Liu et al.] 2020). The categorization of these methods are not fixed. Most of the data-poisoning methods, for example, can be easily adapted to apply in model model--oisoning scenario.
Finally, existing methods can be categorized into static attack and adaptive attack depending on the trigger generation process. Static attacks (BadNe(Gu et al. (2017); Chen et al. (2017): Liu al.
2

---

Under review as a conference paper at ICLR 2023
Data Poison Data
Figure 1: An illustration of our main the- orem. We show that there exists a Trojan twin model nearby the Bayes optimal (Fig. (a)). It can be obtained by training with poison data using universal Trojan trigger (UTT) (Fig. (b)).
UTT
(a) Bayes optimal
(b) Trojan Twin Model
predetermined pattern trigger. On the contrary, adaptive attack methods try to optimize a trigger in order to achieve better attack effect. Examples are Liu
(2020) Nguyen & Tran (2200)
et al. (2017); Pang et al. (2020); Liao et al. (2018); Doan al. (2021). Adaptive attacks leverage the robustness loophole in DNN and usually have better attacking performance. However, these methods inevitably require to control the training procedure and thus all fall into the model-poisoning
category. They become hard to transfer across architectures and behave poorly when users adopt
specific training scheme like adversary training.
Another relevant work is the universal adversarial perturbation (Moosvaioiii et al. 2017). UAP shares similar philosophy with Trojan attack: using unique pattern consistently manipulate the output of a target classifier. However, the challenge faced is different UAP is created during inference time and requires the attacker to query the target model and be able calculate the gradient incurred time, where attacker injects manipulated data into the database. Furthermore, UAP usually works well for the set of data points that the UAP use to optimize the trigger but doesn't 't usually generalize to unseen data, whereas a Trojan attack requires stronger generalization performance on unseen data.
the pattern. On the other hand, a Trojan attack happens before during training
3 THEORY AND METHOD
Our main theoretical results are covered in Sections 3.2 and 3.3 Theorem [T] states that there exists trigger called Universal Trojan Trigger that can be found through adversarial perturbation on a clean model. Theorem 2 and Corollary state that poisoning data with the trigger and training with the poisoned data will result in Trojan twin model that is sufficiently close to the Bayes optimal classifier. The Trojan behavior is guaranteed to generalize to the test set.
We start by formalizing what desirable Trojan model is (Section 3.1
we present an attack algorithm based on the theory, i.e., finding the universal trigger by training multiple clean models and exploiting their adversarial perturbations.
Finally, in Section 3.4.
3.1 A FORMALIZATION OF TROJAN ATTACK
In this section we formalize our mathematical definition of a Trojan Attack. Assuming binary label setting, we define Trojan twin model as classifier that is close to the ideal classifier, i.c., the Bayes optimal, and meanwhile also exhibits Trojaned behavior. We consider models belonging to given hypothesis class F = [ʃ X' + [0, 1]} X C Rdd, e.g.. the family of neural network classifiers. Given an underlying joint distribution p(æ, y). the Bayes optimal classifier with Le risk is
arg min (x(m,)) ((f()) -v)?) fEF
A Trojan twin model not only is close enough f", but also exhibits a Trojan behavior with regard to some trigger v E Rd on a significant proportion of data. In other words, on a sufficient fraction of input æ, should make the opposite prediction on the triggered input, + v. Quantifying the two criteria with and 8, we have the following definition.
Definition 1 ((c, 8)-Trojaan Twin Model). Given a trigger a with budget &. ||ll|| S &, distribution u( y). hypothesis class F and the Bayes optimal f", model J E F is called an (e, -Trojan Twin Model with trigger u if:
E Ep(mn))((aa (f)))
(1)
PP Pa(a) [[1 ((= v) (z())) <
3

---

Under review as a conference paper at ICLR 2023
The Trojan attack task can be viewed as finding Trojaned Twin Model, with small and large 8. In the definition, (((())(((()) (z)) I e specifies that the Trojan model should be e-close to 'clean' model. Note that this implies the high classification accuracy on clean samples (ACC), which a standard quality metric of Trojaned models. Note also that our formulation allows low label noise thus [x) can be real valued function. The choice of L2 regression loss follows from (Massart & Nédélec 2006). The second criterion Pa (1=((1 f(x + v) o fraction of data should have a flipped prediction with the presence of the trigger. This corresponds
(=)) 2 < e) 2 Ó specifies that least
to the attack success rate (ASR) evaluation metric of Trojan models (Gu et al. 2017 Barni al.
2019
Pang et al. (020).
Our definition of Trojan twin depends on trigger In next section, we show that a good trigger always exists to give us a good Trojan twin, i.c., a Trojan twin with sufficiently small and large 6.
3.2
UNIVERSAL TROJAN TRIGGER
In this section, we introduce the Universal Trojan Trigger and prove its existence. Although our definition of a Trojan model is at the distribution level, we formalize the trigger in terms of a given training sample set. This is consistent with the practical setting and suggests an algorithm to find the trigger (as will be explained in Section 3.4). In Section 3.33 we will show that the trigger leads to Trojan twin model at the distribution level. The trigger is defined through a clean classifier, e.g., an empirically optimized classifier given training sample set.
Definition 2 (Universal Trojan Trigger (UTT)). Given i.i.d sampled data set Sn = {(ar: 3:)) and hypothesis class F, let fxx) arg min Σi= (yi f(xx))) be an empirically optimal classifier with regard to the sample set Sn, A trigger is a p)-UTT) if there exists some F s.t.:
If(x.) - f(x.)) Se
(2)
1|1 - [(01-s(a;))) +v - s(xi)) SE} p Σ
i=1
S
We call such f the empirical twin model of f. which is the empirical clean model learned from Sw
UTT describes a common direction among data that can be applied to flip some 'good' model, i.c., F. on the training samples. & represents the budget of trigger, which is related to the stealthiness of the trigger. P represents the fraction of training samples that can be manipulated by the trigger. represents the classification accuracy of the Trojan model on the training samples. A UTT is considered successful for small и and large p. i.c., it can flip model's output for large fraction of data in the dataset. It can be observed that Equation 2 shares similar spirit with the (c. 5)TTM definition, in a sense that UTT manages to manipulate a 'twin' model that is close to the best model on the training set. Indeed, the existence of UTT on the training set has significant implication for finding (s, 8TM. We will rigorously prove that one can implant such trigger to poison the dataset, provably enforce the user's model to become a (c, -TTTM.
While universal Trojan trigger seems to be powerful in pursuing the TTM in Definition I one may ask whether exists for the desired dataset and hypothesis class. In the theorem below, we prove the existence of UTT, under mild assumptions on the data and hypothesis class.
Theorem 1 (Existence of UTT). Let S. = {12, y:) 1 be i.dd sampled from its generative distribution μ. Let X C Rdd be the support of p(æ)). Let F {f X + [0,1]] 1]} be a 3-Lipschitz dy(F) < co. Assume realizability: E[y|æ]
hypothesis class with finite (x) = arg min (f - y) Suppose we have u such that ||>||| $ & and there exists JEF
Ebad E X with 1 f (xbad + 0) fr (Ibad)) S E, and that (BB(arbadd ((B(Iraade/(44)) e/(48)) > 0. Then if the number of samples satisfies
(p(F))og(? + log 10)
log(
g
/a (3 Dbad, e/ (4()))
4

---

Under review as a conference paper at ICLR 2023
with probability at least 1 7/. there exists (g, 22, ( e -UTT.
Remark 1. The proof of Theorem ! given in the Appendix, provides insight into how to find UTT with large p. The crucial assumption that there exists that can successfully adversarially attack the model I (a) for sufficiently many 'bad' data. This is indeed verified in our experiments.
3.3 ENFORCING (s, -.)TAANN TWIN MODEL VIA POISONING DATASET
Next, we describe how UTT induces TTMs. The crucial step is to poison a training set with UTT. In particular, given UTT denoted
let g F be the empirically twin model described in Definition 2 one can create the set of poisoned data
Pm :- ((0; + w,,-,11.000 y:) (æ., vi) E 1n.11-g g(x; + - [(
We call Pm the u-perturbed set, and m 2 pm is the size of P.. P.. can be mixed into the clean training set Sn and their union is the poisoned training set. In the remainder of this section we will show that under mild assumptions. any hypothesis f F trained using the poisoned set that achieves low risk will be an (s, -)-Trojan Twin model for some E and 6. First, in Proposition]] I we show that the model trained using Pm U S,, will behave like a TTM on the training dataset. In Theorem [ we will generalize the guarantee on training set to distributional guarantee using some empirical analysis tools from (Pollard 2012). concluding in Corollary [ with the existence of a Trojaned Twin model with the distributional guarantee in Definition 1
Proposition 1 (Existence of Empirical TTM). Assume we are given the training set Sn a (E, E, P)- UTT u, the empirical twin model 9. the v-poisoned set P... and the empirical clean model f. Assume the interpolation condition: V( yi) E Sn, (f(x)) yil S e. Let be any hypothesis satisfying Σ (y- F(x))) & min
Σ (y- - f(x))² + E. We have:
(2,y)ES, U.
Jef (a.v) U.
Σ
(yi f(x:))) S 4s
(3)
n
1((1 yj F(z; + 0))² - p 2
85 }
m
(4)
(0.+t,1-9))e
Proposition I anaalyess the consequence of training models on the poisoned dataset. The model that achieves low risk tends to fit both the clean and poisoned subset the training set, which implies the succesfulness of finding TTM on the training set. In next Theorem, we generalize the results on training set to achieve distributional guarantee. We analyze the 'in distribution' guarantees leveraging on standard tools for empirical process. The "out of distribution' generalization guarantces, i.e. the generalization on the perturbed data set, require additional effort. We require Lipschitzness of hypothesis class and some regularity conditions of data distributions so that the empirical performance on a subset of training data will generalize.
Theorem 2 (Existence of Distributional TTM). Assume the hypothesis and conclusion of Proposi- tion i hold, and by - [xx) S é for all (x, v). Then, if Z 1 (dp(F) log(E) + log(z)) we have that E. [=(((())- fx= (x) S with probability at least -1
In addition, assume all functions in F are 3-Lipschitz and that the density p(x)) is absolutely continu- with bounded and convex open support D. Let inf ((B(x,/// =/4p)/ nD). For set S. C D, =ED
denote by P(22 E) an e-sparse set points in Ω. Let Pi Pm QEP... 24
arg min
max |P(Q,e/(4p$)) P
There exists Ω C D s.t., ((2) 2 IP T and Vx E Ω -
16s
Theorem directly implics following Corollary:
2 f ii a (E',8) -T-TTanaaan Twin Model where el 16s and 2 IPP._ |T with probability at least 1 n- p
Corollary (Existence of (s, -T-MM). Given conditions in Theorem
Remark 2. Theorem 2 implies some sufficient conditions for enforcing a (E. 8) -rojjnn Twin Model, using the dataset poisoned by UTT trigger. The lower bound of 8 achieved in Corollary 7 is
5

---

Under review as a conference paper at ICLR 2023
distributional dependent quantity and thus hard derive uniform bound. In practice the attacker can train some models on the poisoned dataset and use a validation dataset to empirically evaluate value of 6. It also be observed that large value of p implies closer Trojan twin of f and wealthier data that can be manipulated, improving the quality of TTM.
3.4 ALGORITHM IN PRACTICE: GENERATING THE UNIVERSAL TROJAN TRIGGER
In this section we describe our algorithm for generating UTT. Our algorithm is well motivated by our theoretical analysis. The algorithm takes as input multiple clean models {/::... ... where models well trained and are of different variants. We introduce J models cover different hypothesis classes of classifiers. A discussion on the benefit of introducing multiple hypothesis classes can be found in Appendix Section A.S The trigger flips data from the source class label C's to the target class label Cr- A target injection rate p is given to control the fraction of data to be poisoned by The trigger has a budget Ç. || I Ç. Please see Appendix B.1 for more implementation details of our algorithm.
Our algorithm begins by sampling a perturbed data set Pm. = {(x), Cs)...., (arm, Cs)} conditional on the source class uniformly randomly. Here = [pn]- During the iterative procedure, the algorithm computes gradient direction of loss function ( ΣS Eeep, [(CT, f5( + v)) to minimize the discrepancy between model output and target class on perturbed data + In practice, we observe it suffices to pick to be some small constant, e.g., T = 5, to find satisfying v.
Algorithm Universal Trojan Trigger Generation
1: Input: Clean data set S, = {(x11 v1), (En, Bn)) C Rd x {1,2, K), pre-adversarial- trained clean model set {fi, f2. fr). loss function (e.g., cross-entropy), randomly ini- tialized Universal Trojan Trigger v(()) E Rd, source class Cs, target class Cr, trigger budget constraint Ç, learning rate 1/, injection fraction p. number of iterations T.
2: Sample perturbed set Pm.
Cs), Cs), (Dm, Cs)} from label-Cr data in Sn.
3: for f- 1,.., do
4: (()) -E ExEp, (Cr,Sila + ol(-11))
5: w11)_ ((t-1) V. (()
6: (1 /l/
7: end for
8: Output: (77)
4 EXPERIMENT
In this section, we present and discuss the result of our empirical study. We first evaluate the attacking performance. We manually inject our UTT into different image datasets and use these datasets to train ResNet18 and VGGI6 models. We then evaluate our method's performance against most recent backdoor attack baselines. We evaluate on various settings including the most challenging one where user adopts adversarial training. In the most challenging setting, all baselines has performance deterioration, whereas our method outperform others this situation.
Next, investigate the evasiveness of our method against Trojan detection methods. We Trojan multiple models with UTT and then investigate how resilient are these models against SOTA detection methods. Quantitative results show that our method is much more resilient to detection methods compared to other attacks. We also show our method's 's resistance to fine-iruning post-process. All these merits implied by the properties of Trojan twin model.
Finally, we conduct ablation study on the choice of different ratio and trigger size. Our method is shown be robustness w.r.t. the choice of hyper-parament
such as Trojan injection
4.1 ATTACK EXPERIMENTS
Experiment Setting, In this section, we present the result of attacking experiments. We manually inject Trojan Trigger with each baseline attacking method into different dataset and the poisoned data

---

Under review as a conference paper at ICLR 2023
to ResNet18 (He et al. 2016) and VGG16 (Simonyan & Zisserman, 2014) for training. We fix the L2 norm of each method's trigger to be 10 and the injection ratio to be 20% for each method. We train each method for 200 epochs with same batch size (128 for CIFAR IORGTTRT 32 for IMAENETT), same learning rate 7e-3 (we use gradient accumulation due to the limited computation resource, so we scale original learning rate le-2 by 1/v2 which is 7e-3) and same weight decay rate 5e-4. We will present ablation study results on injection ratio and trigger size in the appendix.
During the training of the model, we assume the most challenging situation where the user will adopt adversarial training (we use PGID here (Madry et al. 2017)). This test all baselines under more practical setting because whenever the trigger is injected, we have no control over the training scheme that could be adopted by the user. For mooeel-pisooning method like WaNet and adaptive attack method like IMC, we also use adversarial training to make fair comparison. For the reference, we present the result where we don't 't use adversarial training in the Appendix Table
Baselines. We select several attacking methods that are representative for each schools mentioned in background section. We use name abbreviation BadNet for (Gu et al., 2017). SIG for (Barni et al..
2019), REF for (Liu et al., (020), WaNet for (Nguyen & Tran 2020) and IMC for (Pang et al. [2020]. We have a detailed discussion of each baseline and the hyper- paper--arameter setting of each baseline can be find in Appendix section B.1
Dataset. We test our method against three image data set. CIFARIO (Krizhevsky et al. 2009) is small scale color dataset with 10 classes. Each image is of size 32x 32. It has 50000 images for training and 10000 images for testing. GTSRB (Stallkamp et al. 2012) is the German traffic sign recognition dataset with 43 classes. We resize each image in GTSRB to 32x32. It has 26640 data points for training and 12630 data points for testing. We also test our method against ImageNet (Russakovsky al. 2015). Because training on large number of high resolution images results in training overhead. We pick images from class 0-9 from ILSVRC2012 dataset and resize each images from 2 to 112x 1 12. It contains 13000 training images and 500 testing images.
Evaluation Metrics. As it suggested by our theoretical model, we should evaluate a attacking algorithm through two criterion. We conduct one-to-one attack here by choosing source class and target class. One of the evaluation metric is to measure the attacking successful rate (ASR) on this source-target pair. ASR is the proportion of testing images from source class that could be mis-classified by the Trojaned models into the target class when edited by the trigger. The higher the ASR, the more effective the propoased attack.
Another evaluation metric is the classification accuracy (ACC), which measures the classification accuracy of Trojaned model on clean images. We require high ACC on a Trojaned model because we want the Trojaned model to keep intact functionality when it gets clean input.
Discussion. If we look at Appendix Tables 4 where no adversarial training is using during trainig of Trojaned model, all baseline achieve similar performance on both ACC and ASR for most of the case (we only highlight the significant best one using two sample t-tests). However, in practice natabasc-poisoning methods (like BadNet, SIG, REF and ours) do not assume the access to the model. Thus these methods have no control over the training scheme adopted by users. For model- model--pisoniing methods (like WaNet and IMC), their generated Trojaned models may be post-processed or fine tuned by the user. Users can adopt the training scheme that is the most unfavorable to attackers. Adversarial training is one of such training scheme that can hinder the Trojan attack. We can see from Table 2 even though all method suffer certain performance deterioration, our method maintain good ACC and consistently competitive ASR over all baselines. The advantage comes from the universal trigger generated by adversarial trained model pools. This specific trigger can manipulate the output of an adversarially trained model. User cannot avoid being Trojaned even they conduct adversarial training.
Table I: Accuracy on Clean Inputs Under Adversarial Training
Datasct
Network
BadNet
SIG
REF
WaNet
IMC
Ours
ResNet18
0.90210.003
0.912:10.00
0.905-10.002
0.901=0.005
0.909-200011
0.908-10.002
CIFARIO
VGG16
0.897=0.002
0.903:10.001
0.902+00001
0.900=0.002
0.900=00.000
0.904+0.000
ResNet18
0.9251..003
0.910-10.013
0.9041100019
0.911=0.011
0.89910.004
0.912-10.002
GTSRB
VGG16
0.941+0.002
0.904
0.942+00002
0.938-0.001
0.939+000004
0.946+-0.009
ResNet18
0.61910.003
0.616.10.003
0.61910.008
0.610-0.004
0.607-10.003
0.618.10.004
ImageNet
VGG16
0.6082
0.668+00.008
0.633+40.006
0.667-0.001
0.662+0.004
0.67
7

---

Under review as a conference paper at ICLR 2023
Table 2: Attack Successful Rate Under Adversarial Training
Dataset
Network
BadNet
SIG
REF
WaNet
IMC
Ours
ResNet18
0.992=0.001
0.957:10.016
0,746=+0002
0.966=0.009
0.988=00002
0.904:1:.000
CIFARIO
VGG16
0.900.10.003
0.957.10.002
0.7004
0.960_00007
0.978.10.003
0.004000.000
ResNet18
0.969+00.007
0.904+00.03
0.885+00033
0.950-0.019
0.892+0.030
0.978+00000
GTSRB
VGG16
0.003
0.956.10.014
0.881110.028
0.926=0.047
0.569::0.07
0.906
ResNet18
0.968=0.001
0.735++0.046
0,900+00008
0.877=0.012
0.851=0.012
0.967++00001
ImageNet
VGG16
0.963+000011
0.546+00.08
0.904+0.040
0.877=0.012
0.314+0.174
0.967++0000
4.2 DETECTION TABLE
Experiment Setting. In this section we present the model inspection result. The number in Table copy from Table 11 of (Pang et al. 2022). We follow their experiment setting and Trojaned 10 ResNet18 models trained on CIFARIO 10 dataset. We use the same implementation of these model inspection algorithm and present the anomaly index value (AIV) number got by our method from each model investigation method on the last column of Table 3 We will discuss the evaluation metrics later in more detail.
Baseline Attack. There are mainly 8 attack methods compared in (Pang et al. 2020) including
BadNet, REF and IMC, which we have discussed above. TNN the method proposed in (Liu et al. 2017). TB is the blending method proposed in (Chen et al. 2017), LB is the method proposed in (Yao et al. (019a)). ABE is the method proposed in (Shokri et al., 2020). We have discussed all of these work in Appendix section B.I Besides, embarrassingly simple backdoor attack (ESB) tries to attach an extra Trojaned neural net to the target model. They trained the merged network such that the Trojaned part will be activated whenever trigger is presented otherwise the original part will activate. ESB belongs to model-poisoning attack method category. ABS doesn't 't apply to ESB simply because of its pre-requisite. If we assume white box investigation, where investigator have access to the architecture, the capture of ESB is instant, If we assume black-box investigation, ABS is not applicable here.
Baseline Defense. We investigate all these attack methods with 5 widely used model-inspection algorithm. Neural cleanse (NC) (Wang et al. 2019)) Deep Inspection (DI) (Chen et al. 2019). TABOR
(uo et al. 2019, Neuron Inspection (NI) (Chen et al. 2019) and Artificial Brain Stimulation (ABS)
(Liu et al. 2019). More detailed discussion about these defense baselines can be found in Appendix
section B.2
Evaluation Metric. We mainly use anomaly index value (AIV) as the metric to recognize Trojaned model. AIV is the normalized median absolute deviation. For a set of input (z1, ; Frn }. the median absolute deviation (MAD) is the median of {|= - mmedianls .. [xn mmedian|]. Then the AIV of
this set of points is { [2) Emediaal
1.4826MAD }. Any point in this set of data that has an AIV
- Emedioai
larger than 2 considered to be an outlier.
In our case, n is the number of output classes and x is the LI norm of reversed trigger or explanatory feature (for ESB) given by each investigator. Following the setting of (Pang et al.) 2022), for each output neuron in these 10 Trojaned networks, we record such AIV given by the target class. Then we perform a t-test for each attack-defense pair to decide if the AIV significantly larger than (MAD test). In the table 3 we highlight methods that are not evasive by corresponding investigation methods with t.
Discussion. From Table 3 we see that most attacks quite evasive against current detection algorithm. ABS is the most effective inodel-iveestigation method and capture TB, LB, ABE, and IMC. Our method is evasive to all listed investigation algorithm. This is partially suggested by Theorem 2 which says the model Trojaned by our trigger represents a function that is very closed to what the clean model learnt. This can add difficulties to detection.
4.3 RESISTANCE AGAINST FINE PRUNING
In this section, we test our method's 's robustness against fine pruning post- vost-processing. We implant Trojan into both R8 and VGG16 models by poisoning the CIFAR10 dataset with our UTT. For
8

---

Under review as a conference paper at ICLR 2023
Table 3: AIV of Model-Inspection Method and Detection Algorithm. (Attack that captured by
corresponding inspection algorithm are highlighted with f)
Defense Attack
BadNet
TNN
REF
TB
LB
ESB
ABE
IMC
Ours
3.08
2.69
2.48
2.44
2.12
0.04
2.67
1.66
0.57
NC
(+0.65)
(+0.47)
(+0.51)
(++038)
(++.200
(+0.02)
(+0.51)
(+0.25)
(+0.49)
0.54
0.46
0.39
0.29
0.21
0.01
0.76
0.26
2.12
(+0.06)
(+0.04)
(+0.04)
(+0.03)
(++0004
(+:0000)
(+0.10)
(+:0.03)
(+1,31)
3.26
2.49
2.32
2.15
2.01
0.89
2,44
1.89
0.72
TABOR
(=00.77)
(=0.49)
(=0.51)
(=0.29)
(+0.63)
(#0.04)
(#0.22)
(#0.19)
(=1.01)
1.28
0.59
0.78
1.11
0.86
0.71
0.41
0.52
0.87
NI
(=0.21)
(=0.11)
(=0.06)
(=0.34)
(+0.87)
(#0.100
(10.05)
(#0.13)
(+1.45
3.02
4,16
4.10
15.551
2.88
8,451
3.15
3.31
(1:0.1))
(111.33)
(1:1.77)
(116.59)
(1:0.25)
(133222
(1.0.43)
(1:4.07)
resnet13-ggtro
voo1G-clbrr0
vog1Gegttrb
1.0
1.0
0.8
0.6
0.6
044
0.4
0.4
0.2
0.2
0.2
ACC
ACC
ACC
ACC
ASR
ASR
ASR
ASR
02
0.4
06
1.0
Filers Prunina Ratio
Fittes Prunino Rab
Fitees Prunino Rato
Filters Prunina Retio
Figure 2: An illustration the resistance of our method against find codd-prunng. The filter pruning ratio the proportion of number of pruned filters over the total number of filters.
each convolutional layer we prune the convolutional filter with the lowest L1 norm within current un-pruned filters in a stratified manner. We set different pruning ratios and measure ACC and ASR of the pruned network on the Trojan dataset. The result is shown in Figure 2 We can see that, the fine pruning cannot effectively reduce the ASR without hurting ACC. This is because the Trojan twin model very closed the clean model. It should not use too much extra capacity for recognizing the trigger. As result, fine-pruning cannot easily erase the circuit for classifying trigger without damaging useful structure.
4.4 ABLATION STUDY ON INJECTION RATIO AND TRIGGER SIZE
In this section, we conduct ablation study on the effect of injection ratio and trigger size. section 4.1 we fix the injection ratio to 20% and the L2 norm of the trigger size to 10. In this section, we conduct an ablation study by reducing the injection ratio 10% and reducing the trigger size to separately. and test each baseline accordingly. Experiments results are presented in Appendix Tables 9 We can see that in the case where smaller trigger or fewer Trojan data is used, our method still maintains advantageous performance. These results corroborate our method's robustness against hyper-parameter choice.
SUMMARY
In this work, we study the Trojan Attack problem. We formulate the Trojan Attack task as finding a twin of clean model. We quantify the quality of twin model using natural bi-criteria metric. We propose poisoning data attacking strategy where the data corrupted by our carefully designed trigger named UTT. We show the merit of our Trojan attack strategy both theoretically and empirically. In particular, empirical study shows that our method achieves competitive attacking effectiveness and detection resistance.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
Our experiment uses only public available dataset. We have described our experiment setting and implementation details in Section 4 publication of this paper.
The source code will be made available together with the
ETHICS STATEMENT
This paper studies the Trojan attack problem. Our study deepens understanding of Trojan attack and takes one step toward effective methods to defend against attack. The theory/method discussed in this work may be applied by malicious attackers designed Trojan attack method that may cause security issues for DNN users.
REFERENCES
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 101-105. IEEE, 2019.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In IJCAI, volume 2, pp. 8, 2019.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05520 2017.
Khoa Doan, Yingje Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11196--119999 1196, 2021.
M. Everingham, L. Van Gool, C. Williams, J. Winn, and Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
Tianyu Gu, Brendan Dolan-Gavite and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 178.0637 2017.
Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems. arXiv preprint arxiv: 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 770-778, 2016.
Xijie Huang, Moustafa Alzantot, and Mani Srivastava. Neuroninspect: Detecting backdoors in neural networks via output explanations. arXiv preprint arXiv:1911.07399, 2019.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: survey. IEEE Transac- tions Neural Networks and Learning Systems, 2022.
Cong Liao, Haoti Zhong, Anna Squicciarini, Sencun Zhu, and David Miller. Backdoor embedding in convolutional neural network models via invisible perturbation. arXiv preprint 10Xiv:188..10300 2018.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. 2017.
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang, Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference Computer and Communications Security, pp. 1265- 1282, 2019.
10

---

Under review as a conference paper at ICLR 2023
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pp. 182-199. Springer, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1r0iv:1706.0000 17060660666 2017.
Pascal Massart and élodie Nédélec. Risk bounds for statistical learning. The Annals of Statistics, 34 (5):2326-2366, 2006.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.
Tuan Anh Nguyen and Anh Tuan Tran. Wanet-imperceppible warping-based backdoor attack. In International Conference on Learning Representations, 2020.
Ren Pang, Hua Shen, Xinyang Zhang, Shouling Ji, Yevgeniy Vorobeychik, Xiapu Luo, Alex Liu, and Ting Wang. tale of evil twins: Adversarial imputs versus poisoned models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pp. 85-99, 2020.
Ren Pang, Zheng Zhang. Xiangshan Gao, Zhaohan Xi, Shouling Ji, Peng Cheng, Xiapu Luo, and Ting Wang. Trojanzoo: Towards unified, holistic, and practical evaluation of neural backdoors. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 684-702. IEEE, 2022.
David Pollard. Convergence of stochastic processes. Springer Science & Business Media, 2012.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015. doi: 10.1007/s 1263-015-0816-y.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 11957-11965. 2020.
Reza Shokri et al. Bypassing backdoor detection algorithms deep learning. 2020 IEEE European Symposium Security and Privacy (EuroS&P), P), pp. 175-183. IEEE, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1et..19999.666 2014.
Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, (0)=- 2012. ISSN 0893-6080. doi: 10.110166j.eummt2222220200000000 URL
com/ wwom/ccieee science/
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019
IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Jon Wellner et al. Weak convergence and empirical processes: with applications to statistics. Springer Science & Business Media, 2013.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. Latent backdoor attacks deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 2041-2055, 2019a.
11

---

Under review as a conference paper at ICLR 2023
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 2041-20555 2019b.
Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In International Conference on Machine Learning, pp. 7614-7623. PMLR, 2019.
A THEORETICAL RESULTS PROOF
A. PRELIMINARIES
Below introducing some definitions that is used in proof. The definitions of covering number, VC-dimensions and
can be found (Pollard 2012 Wellner et al. 2013 Mohri
al. 2018).
Definition 3 (L2-Covering Number). Let Di:n be set of points. A set Of _ IR" an E-Cover w.r.
Lo-norm of F on Fims if E F, Bu E U. s.t. / Ei=, - f(x,)1 S E, where [u]e is the i-th coordinate of The covering number (e, F,,) with 2-norm of size n F is:
sup E
min[UU]: U is an e-cover of vi.))
Definition 4 (3-Lipschitz). We say hypothesis class F is B -Lipschitz for all f E F we have:
If(=1) - f(x2)) S =||=1 x2ll
Definition 5 (VC-dimension). The VC-dimension dyc(F) of hypothesis class F {s : X + {1,-1}} is the largest cardinality of the set S X such that VS S S, 3f F:
(1 E
f(x)
f(x) -
(-1 if
(6)
The Eseudo-dimension dy(F) of a real-valued hypothe- sis class F {f : X' + Ja, b|} is the VC-dimension of the hypothesis class H. (h: X x IR + (-1,1) Definition 7 (e-sparse set). Given set of points B C Rd finite size, we say A is E-sparse set of B if A C B and Va1, 02 E A, ||a] - -2ll 2
Definition 6
sign([(x)) - J E F).
A.2 MISSING PROOF FOR THEOREM 1
Proof: For data For simplicity we denote Ilfi - fells. = =V E. (fi(t) - fo (x:))) and Ilfi
fallu(z) = (Ji [². In the proof we denote 7 1. p r ((((am).ee((43)) Let C
be a a0 -cover for hypothesis class F projected dataset with size for any hypothesis f we denote
c(f) be element
that covers f. In particular we have Yf, 3c(f) E T, [cf(c()) - flls. S 22. Let
f arg mig 1 E (f(xa)) 2) It is easy to verify that Jel
1
(c.) vi)²
n
(z.)) - -f(x)) + f(xx)) - -1)²
Σ (d.)x))))) f(x,))) + (f(x.) - -112 + 2(c((((x.)) - F(ax)) - v.)
Σ - 2(x (f(z)) - 21) + 2,
1
Σ
(c(F) F(xe))
Σ (fxx)) - 9
=
i=1
(7)
12

---

Under review as a conference paper at ICLR 2023
which implies that
(a())a)))))) vi)2 - (f(.)) v.)² + 472 S
1
(s)))) v.)² + + 4y2.
i=1
By standard empirical process argument, using Hoeffiding type inequality with symmetricity PPol lard 2012) and taking union bound on the covering set T, we have
1 Σ (f(x.) u1) w²²²
2 2
s 28s, [T]erppl n17
(Bay(((()) - y)
sup fEr
Z lox(
know that by picking
E ((()) y) < Em,,1/ - y)² + 9h2
together with the fact that [[y|=]] = (x) implies that
Ex (<c((()))) f
Next we show that
Σ - 13
We apply Hoeffding type inequality to show that the probability of following event, if draw S, in
Es, [|T] ). is least 1 r7:
an i.i.d fashion with 2 ya log
ve(f) ET, 1 Σ
- -
s 13-?
(8)
1=1
To see this, invoke Hoeffding type inequality again (Pollard 2012
(zx)) fz -x(cf)(=)-(===)) 2 7² s 2Es. [T|eerp
_
Ps. sup LIF
4
1=1
(9)
1 log Es, [|'|] implies that Inequality holds with probability at least 1 1)-
We can show n 2
77
Next we bound [T]. By Theorem 2.6.4 in Wellner et al. 2013) we know that there exists universal
(dp(F)
constants K, C < 30, for all Fims [T] < (dp(F))
which implies that it suffices
to pick n 2 dr(F) log( Hogg(
to ensure that 8 holds with probability at least -
Now we can bound the difference between f and f on under empirical L-1 metric:
(xx, - f(xx))
Σ (f+()))) - f(x,))) 11f-= flls.
(10)
<| M.ls, + +15 -cfflsss.
<4>
13

---

Under review as a conference paper at ICLR 2023
Let bad Sn n ( abad: (45).. The choice of also implies that [Sbad] 2 2 with probability at least 1 1/. For any x' E Shad, we have:
1 (a + v) - ff'')
S 1 - f (abbd + +0) + (abbd + v) - fa' + +0) - (abbad + f (abad) - (x'))
< 1 - f" (arad + +v) - f (abbd)) + (arbad + v) ("' + +w) + (dhad) - l (a/)) (11)
S + 3
43 43
Thus, with probability at least 36, we have existence of f E F and v s.t.,
[f(x)) - f(x)) se
n
1
(12)
1(1((z; +D) - 11 f(x))) 2 2
i=1 ||l|| <
The crucial assumption in proving Theorem 7 that there exists vo that can successfully adversarial attack the model f" (x) for at least one 'bad' data. In practice, several observations have been made that there exists some common direction v which can adversarially attack many abad that distinct from each other (Moosavi-Dezfoole et al. 2017). In such scenario, P could be significantly increased if the common direction is found.
Remark
A.3 MISSING PROOF FOR PROPOSITION
Proof: On one hand we know
(y: g(x;))2 1 Σ 1y: g(x;)l
Σ
(m..))ES..
Σ 1y: F(xx)) + f(xx)) g(x;))
n
(13)
11, M(xi) 1f(m;)) g(0z))
<2c.
We also have:
Σ
(g(z, + +0) - 1 y2
( +,,1-
Σ
19(x, + +u) - + リn
(,,10,1))))
Σ
19(0, v) + g(x,) - g(xx) + U/
(14)
Σ
19(x, +v - 1 g(0,)) Σ
1g(x;)) - f(ær,) + f(x,) 4sl
(a.+o,1-v)))
(1))
2n E
<2e +
m
14

---

Under review as a conference paper at ICLR 2023
Thus have
1
(y: - {(x;))
m +
(... Um
Σ
(yi- - -((xi)) +e
m +
(2;: U Pm
(15)
< n
Σ
(yi- - g(xx)))
m
Σ
(g(z) + + y,)
171 + (EL.
m +
m
(0 By)
<3
Since
Σ
(y) J(xs)) <
Σ
(yi - f(=)))
(E..
(s Pm
and
Σ
o0) - (E.. U Pm
Σ
(ys - F(x,))
(11-5,1-15)
Thus conclude that the following inequality holds:
Σ (yi - ((x))) < 4e
(16)
n
(ES,))ES
1
Σ
(1 -1- yj - [(ar, + 0)) S
2(m + n)e
S
48
(17)
m (my PP)1-
m
By Markov inequality, at least points satisfies
8 fax, 01)2 2))²²5 s
(1- - 11
(18)
A.4 MISSING PROOF FOR THEOREM
Proof: An empirical process argument similar to the proof in Theorem 1 show that as long as log()) we have: Ex (y (()yf((=))²] S E, which implies that (=(("<)) Se.
>
m By Proposition at least 2 points satisfies:
8e P
(1 (1 リj - A(0 x))²
(19)
Next we construct Q2 Let Q be set of points in P.. such that Equation 19 are satisfied. We set $
= be maximum 403 -sparse set Q, i.e., Va, E Q, ||abb| 2 4pp U B(x, 403 )DD Let PO (Q, 13p REQ
Since for arbitrary subset of Pm with size at least m its max 4p3 packing number is at least PPp.. |. thus 17 m(2) 2 IPp. |T. For any x' Ω, 2 (f*(( f( ip
find Fbad E s.t. ||bbad x'1 4p3- Thus &. and [f(daa)) - f(xx)) i.. Thus we have for all x' E Ω:
((f(z' + - 1- fx')²²
<(F(2' +v) f(x + v) + -
<4/f(al + - 1-00-
(20)
<4( 6 3+ E 3 + +1
4p8
4p3
22s
_-
P
15

---

Under review as a conference paper at ICLR 2023
A.5 DEFINITION AND DISCUSSION OF MULTI-HYPOTHESID UTT
Corollary I suggests that the data poisoning schema using universal Trojan trigger is indeed very powerful enforcing user's model to be TTM Under the assumption of Corollary as long as user's model achieves low risk on the poisoned dataset, the model becomes a TTM with high probability. One important assumption made in Corollary [ is that the choice of hypothesis by users, c.g., the architecture, belongs to the hypothesis class that attacker considered when poisoning the dataset. To ensure the assumption holds, we generalize the definition of UTT into following multi-hypothesis class version to cover more type of neural networks. Our Corollary [ can be easily generalized to the following multi-hypothesis class version.
Definition 8 (UTT for union of hypothesis classes). Given data set S,, {(x,, 9:) H= and family of hypothesis class 1 F,, let fz(x) = arg min Di= - f(x.))² We say v is v(f, E. p)
feJ',
-UTT for hypothesis Uk F3 for every hypothesis class F, there exists some f E F, s.t.:
If(x,) f5(x;)) SE
n
(21)
Σ
1i11 f(x; + + f(z,)) SE} 2p
n i=1
||a|| 5
16

---

Under review as a conference paper at ICLR 2023
B EXPERIMENTS SETTING AND DETAILS
B.1 ATTACK BASELINES.
BadNet (Gu et al. 2017) places 3x3 image patches on the corner of a Trojan images as a trigger. Labels of Trojan imgges are also changed to the target class at the same time. Attacker inject these modified data point and create the Trojan database. SIG (Barni et al. (019) overlay the target image with watermark where each pixel has a sinusoidal function value depends on the pixel's position in the image, which makes the trigger invisible to human eyes. REF further improve the trigger stealthiness by using the reflection effect. They blend the trigger image into the target image and make the trigger looks like a natural reflection. WaNet Nguyen & Tran (020)) propose to use the warping operation to create trojan images. Instead of use predefined trigger. they continue to apply warping operation on clean images during training and these warped images are considered trojan images and their labels are modified into target classes during training. IMC et al. (2020) propose bi-level optimization procedure that optimize the trigger and model at the same time to minimize the empirical loss on the trojan database. TNN Liu et al. (2117) is an adaptíve attacke method that optimize to find the trigger that maximize the output of specific neuron in the penultimate layer. TB Chen et al. (2017) uses natural images as the water mark trigger and blend these trigger with source images to achieve visual stealthness. ABE Shokri et al. (2020) use GAN to generate trigger that produce indistinguishable intermediate layer representation as clean instance does. LBB Yao al. (2019a) follows the idea of BadNet but add restriction on the intermediate layer representation of Trojan images to be more closed to clean instances.
Hyper-parameter Setting. Baseline Ned and SIG don't have specific hyper-parametens For REF, we follow the original paper setting. We use PASCAL VOC dataset [Everingham et al. (2010) as the candidate trigger base. We select trigger outof the whole PASCAL dataset. We finally keep 200 triggers given by the trigger search procedure. For WaNet, we also use the original setting, we set the hyper-parameter K to be 4 and S to be 0.5. We cross-rate 2. For IMC, we use the default setting and conduct iteration adversarial attack searching for the trigger and iteration of model update iteratively using learning rate 0.1. For Our method, we use 5 adversarially trained models to search for the single UTT. We adopt 5 step adversarial training to search for UTT.
Implementation Details of Algorithm 1 We use 5 the universal Trojan trigger. Each model in the pool is initialized differently from those used in the attacking experiment. In Table we use same architecture for the pools and for the target model. We also present the transferring attack result in TU where we search the trigger with architecture and attack model in architecture B. For example, we inject trigger found with VGG16 attack ResNet18 model, we can get similar performance as the original paper.
model to search for
B.2 DEFENSE BASELINES.
Neural Cleanse (NC) is a trigger reversion method. It optimizes a randomly initialized pattern until the patter can change the output of the model under investigation. A model is recognized as Trojaned if the size of the reversed pattern is small. Deep inspection (DI) (Chen et al. 2019) use GAN, instead of trigger inversion, to generate trigger candidate in order to change the output of the target model. Then a model is detected to be Trojaned if the generated trigger's ' mask MAD goes beyond TABOR (Guo et al. 2019) follows the idea of NC, but add regularization terms enforce the reversed pattern to have similar shape and placement location as real trigger does. Neuron inspection (NT) (Huang et al. 2019) calculate several explanatory feature using the gradient heat map of the target model for Trojan detection. Artificial brain stimulation (ABS) (Liu et al., 2019) identify suspicious neurons in the target model by adding stimulus value to each neuron's 's output. A neuron is identified as compromised neurons if stimulation to this neuron maximally change the output of the target network. Then a reverse engineering process is used to find a candidate trigger that can maximally stimulate the compromised neuron.
17

---

Under review as a conference paper at ICLR 2023
C
MORE EXPERIMENTS RESULTS
Table 4: Accuracy on Clean Inputs without Adversarial Training
Dataset
Network
BadNet
SIG
REF
WaNet
IMC
Ours
R8sNet18
0.94111..02 0,943.1.001
0,942.10.002
0.937_10.001
0.94310.002
0.942.1.002
CIFARIO
VGG16
0.932:10.001
0,93410.001
0.933:10.001
0.930=0.001
0.934=0.001
0.935=00000
ResNet18
0.933.10.002
0.941100.008
0,94110.007
0.939
0.942.10.02
0.940.0.003
GTSRB
VGGI6
0.945:+00002
0.959=000001
0.957=00005
0.960=0.002
0.959=000033
0.957=000003
ImageNet
ResNet18
0.691:0.014
0.692=0.007
0.00000000
0.686=0.004
0.700=00.003
0.699=0.017
VGG16
0.786:+0.009
0.792#000005
0.785=#0.008
0.785=0.009
0.785=0.008
0.781=0.008
Table 5: Attack Successful Rate without Adversarial Training
Dataset
Network
BadNet
SIG
REF
WaNet
IMC
Ours
ResNet18
0,999110.000
0.99811.001
0.8051000066
0.998=0.001
0.999-0.000
0.999-00000M
CIFARIO
VGG16
0.999=10.000
0.997=0.001
0.810=0.0088
0.997=0.001
0,999=0.000
0.999=0.000
ResNet18
0.984=00.000
0.984+0.000
0.951=0.016
0.984=0.000
0.984=0.000
0.984=0.000
GTSRB
VGGI6
0.984++0.000
0.984+0.000
0.973+0.019
0.967+0.028
0.978+0.009
0.984+0.000
ImageNet
ResNet18
0.980=+0.000
0.948=0.023
0.980=0.000
0.967=0.011
0.980=0.0000
0.980=0.000
VGG16
0,980:10.000
0.82410.079
0.974.1.001
0.974=0.011
0.974100011
0.980=100000
Table 6: Ablation Study on Injection Ratio: ACC
Dataset
Network
BadNet
SIG
REF
WaNet
IMC
Ours
ResNet18
0.903=0.003
0.912=10.003
0.905=+0.002
0,905+0.001
0.909=000002
0.908=00002
CIFARIO
VGG16
0.001_00003
0.903110.001
0.902.10.001
0,90110.001
0.902110001
0.90410.001
ResNet18
0.900-0.016 0.910+0.013
0.904++0.019
0,911+0.008
0.901+0.011
0.8097
GTSRB
VGG16
0.936=0.003
0.944=0.006
0.942:10.002
0,94240.003
0.946=10.033
0.940=0.007
Table 7: Ablation Study on Injection Ratio: ASR
Dataset
Network
BadNet
SIG
REF
WaNet
IMC
Ours
ResNet18
0.991=0.000
0.957-10.016
0.746110.002
0,926110021
0.98410.003
0.09410.000
CIFAR10
VGG16
0.992=0.001
0.957=0.002
0.731:10000
0.914:20016
0.976=0.001
0.994=0.000
ResNet18
0.970
0.904110.083
0.885110.033
0,89510.042
0.806100010
0.9761100001
GTSRB
VGGI6
0.971-0.009
0.956+0.014
0.881+00022
0.938+0.011
0.341+0.110
0.975+0.003
Table 8: Ablation Study on Trigger Size: ACC
Dataset
Network
BadNet
SIG
REF
WaNet
IMC
Ours
ResNet18
0.905=0.002
0.908=0.002
0.903+10.004
0.902++0.003
0.909=00001
0.906=0.002
CIFARIO
VGG16
0.900=0.002
0.901=0.001
0.900+000001
0.900+000022
0.900+000000
0.902+0.003
ResNet18
0.899=0.002
0.908:0.021
0,913:10:012
0.910+:0.008
0.899=0.004
0.914=0.002
GTSRB
VGG16
0.937_0.003
0.942.10.006
0.934.10.004
0,94011.001
0.93910.004
0.908
Table 9: Ablation Study on Trigger Size: ASR
Datasct
Network
BadNet
SIG
REF
WaNet
IMC
Ours
R8
0.990-0.001
0.866+0.029
0.757+0..007
0.972++0.002
0.988+0.002
1.9914
CIFARIO
VGG16
0.001
0.888:10.013
0.7051
0.969-10.003
0.978:10.003
0.994
R8
0.967=0.002 0.826+0.032
0.859+40.024
0.945+0.020
0.892+0.030
0.975=0.088
GTSRB
VGG16
0.968=0.006
0.734
0.849110.04
0,93611.016
0.56910.071
0.977=0.001
18

---

Under review as a conference paper at ICLR 2023
Table 10: Transferring Attack Result of Our Method
Dataset CI
Pooled Network
Target Network
ACC
ASR
VGG16
ResNet18
0.907:1.004
0,994:000011
CIFARIO
ResNet18
VGGI6
0.905=10.001
0.994:000000
GTSRB
VGG16
ResNet18
0.914+0.050
0.976=00.001
GTSRB
ResNet18
VGG16
0.939=100004
0.977#0.001
Global Pruning Results. We also provide the global pruning result, where prune the filter that has the smallest LI norm among all convolutional layer instead of doing it in a stratified manner. With this pruning method, it possible some layers be totally removed during the increasing of pruning ratio. We present the added experiments in appendix Figure 3 resistance result as it is in layer-wise wise pruning.
We could also observe similar
resnet18+cifer10
resnet18=gtsrb
vgg16+cifar10
vgg16+gterb
1.0
1.0
0.5
0.6
0.6
0.0
ASR
0.0
ASR
0.0
ASR
ASR
0.0
0.5
0.8
0.2
0.2 0.6 Filters Pruning Ratio
L.
Ritters Pruning Ratlo
Filters Pruning Ratio
Filters Pruning Ratio
Figure An illustration of the resistance of our method against global find-pruning. The filter pruning ratio is the proportion of number of pruned filters over the total number of filters.
D DEMONSTRATION OF TROJAN IMAGES AND TRIGGERS
Figure 4: An illustration of UTT triggers on ImageNet. Left column displays the clean images. Middle column displays the Trojan images. Right column displays the Trojan triggers. We increased the transparency for middle column when overlay triggers, so it looks darker.
19

---

Under review as a conference paper at ICLR 2023
BadNet
SIG
REF
Figure 5: An illustration of triggers found by each baselines with CIFARI0. Left column displays the clean images. Middle column displays the Trojan images. Right column displays the Trojan triggers.
20

---

Under review as a conference paper at ICLR 2023
WaNet
IMC
UTT
Figure 6: Continued An illustration of triggers found by each baselines with CIFARIO. Left column displays the clean images. Middle column displays the Trojan images. Right column displays the Trojan triggers.
21


Reviews:
[{"review_id": "kH0Xnmwi6v2", "reviewer": "Reviewer_HyKs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigate the existence of trojan model for backdoor attack --- a model that is in theory similar to the Bayes optimal yet performs poorly under some universal backdoor trigger. The paper formalizes its notion, proves its conditional existence and show an attack algorithm that generates universal backdoor triggers for real-world tasks.  ", "summary_of_the_review": "Overall, this paper provides solid theoretical and empirical results for investigating the existence of trojan twin model and universal backdoor trigger. I'm giving a weak accept for now, and willing to raise once my questions are addressed.", "strength_and_weaknesses": "**Strength**\n\nThe paper is technically sound. It provides both theoretically and empirical insights to the problem of backdoor attack.\n\n**Weakness**\n\nI have some questions regarding the technical details of the paper.\n\n1) The theoretical bounds in Theorem 1 and Proposition 1 are somewhat disjoint from the experiment. The experiment checks the existence of universal backdoor trigger, but does not directly check the coefficient of these bounds. While finding the coefficient may be hard for real data, could you provide a toy example for, say, two Gaussian distributions and give the audience a sense of how big the coefficients are?\n\n2) In Figure 2, the horizontal axis is the filter pruning ratio. Could you provide some references on how the pruned filters are selected? Does the curve shape in Fig 2 still hold if a different set of filters are pruned?\n\n3) In Table 3, the mean AIV of your method is >2 against ABS. Why do you use a t-test to show the evasiveness instead of removing all training examples with AIV>2 and then retrain the model on the remaining samples?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RqG_QZLh4FX", "reviewer": "Reviewer_DxEo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the backdoor adversarial attack. It theoretically discuss the existence of an attacking counterpart model and corresponding design an attacking algorithm. The proposed method is evaluated on benchmark datasets.", "summary_of_the_review": "As in Strength vs Weakness", "strength_and_weaknesses": "# Strength\n1. Investigating backdoor attacks under adversarial training is relatively new.\n2. The theoretical results seems interesting.\n\n\n# Weakness\n1. The claim that \"existing backdoor attacks cannot resolve adversarial training\" is questionable. The empirical evidences provided by the author do not fully support this claim.\n2. The improvement of the proposed method is relatively marginal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4NixO2T-Pxd", "reviewer": "Reviewer_YzEQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the concept of Trojaned Twin Model, which provides an interesting view w.r.t. the clean model and its trojaned counterpart. In practice, a universal poisoning perturbation is craft on an ensemble of adversarially trained models, and imposed onto the training set for poisoning. Empirical evaluation are done on CIFAR-10, GTSRB and ImageNet.", "summary_of_the_review": "I like the idea of Trojaned Twin Model, which is interesting and formally defined. However, the bound in Theorem 1 seems impractical, and the connection between the theoretical analyses and the proposed Algorithm 1 is loose.", "strength_and_weaknesses": "Strength:\n- The idea of Trojaned Twin Model is interesting, and Definition 2 makes sense to me.\n\n- The writing is clear and the formulation used in this paper is in the appropriate format.\n\n- The empirical evaluation is done on different datasets, while several baselines are compared.\n\nWeaknesses:\n- As to Theorem 1, there is $|1-f^*(x_{bad}+v)-f^*(x_{bad})|<\\epsilon$. For a practical distribution $\\mu(x)$, if $\\epsilon$ is small (satisfying the definition of Trojan Twin), then $x_{bad}$ is usually located at low-probability regions. So consequently, $\\mu(\\mathcal{B}(x_{bad},\\epsilon/(4\\beta)))$, i.e., the ratio of successful trojan $\\rho$ would be quite small. Besides, practical model family $\\mathcal{F}$ usually has a large value of Lipschitz constant $\\beta$, especially for deep networks.\n\n- $|1-f(x+v)-f(x)|<\\epsilon$ is not a good criterion for poisoning, for example, $f(x+v)=f(x)=0.5$ corresponds to $\\epsilon=0$, but this case is not considered as a successful poisoning.\n\n- In Section 3.4, actually, I do not see any close connection between the concept of Trojan Twin and the proposed Algorithm 1. For me, Algorithm 1 is just constructing universal poisoning perturbation on an ensemble of adversarially trained models, which is nothing new beyond [1]+[2].\n\n- Why the poisoning trigger in Algorithm 1 is preferred to be universal, rather than input-dependent? Besides, do the baselines in experiments exploit the same knowledge from adversarially trained models as done in the proposed method?\n\n\nRefs:\n\n[1] Universal Adversarial Perturbations. CVPR 2017\n\n[2] Ensemble Adversarial Training: Attacks and Defenses. ICLR 2018", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2tmuIPzpUn", "reviewer": "Reviewer_8fMD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to backdoor a neural network so that it classifies certain samples containing a trigger as the attacker intended. The proposed approach assumes to have a set of surrogate models to generate a universal perturbation as done in an evasion attack, and take the universal perturbation as a backdoor. The approach is empirically compared to baselines.", "summary_of_the_review": "This paper revisits a known property of the backdoor problem. The empirical evaluation is unclear, and the improvement over the existing approaches is not significant. The model set is not explained which is very important in understanding the advantage and limitations of the proposed approach. The related work considers some of the existing optimization-based approaches model-based (the attacker has complete control of the training pipeline and deliver the poisoned model), which I believe is an oversimplification, as many optimization-based approaches use a surrogate model similar to the clean model set used in this paper to generate a poisoned training dataset. In summary, this paper does not seem to show a clear advantage of the proposed approach, has an unclear experimental setup, and misrepresent existing approaches.", "strength_and_weaknesses": "Strengths.\n- S1: The paper tries to prove the existence of a backdoored model.\n- S2: The proposed approach is compared to 5 baselines on three datasets.\n\nWeaknesses.\n- W1: The empirical performance doesn't show significant improvement. Actually, the performance is similar or worse than some of the existing approaches.\n- W2: Neural networks are known to be a universal function approximator, and thus the existence of such a model doesn't seem to add much value, especially with adversarial examples with universal perturbation.\n- W3: The approach is already known in both backdoor and evasion scenarios. Also, this type of optimization is commonly used in many attacks, and the approach is a direct adoption of universal perturbation, which has been also used for poisoning. Among many, a quick search shows \"Detecting Backdoor in Deep Neural Networks via Intentional Adversarial Perturbations,\" Xue et al., 2021; \"Hidden Trigger Backdoor Attacks,\" Saha et al., 2020; \"Invisible and Efficient Backdoor Attacks for Compressed Deep Neural Networks,\" Phan et al., 2022; \"AdvDoor: Adversarial Backdoor Attack of Deep Learning System,\" Zhang et al., 2021.\n- W4: The experimental setup is unclear, and there are some modifications to common tasks which makes it harder to compare. For example, the proposed attack uses a set of clean models which is not specified, and can be used as an unfair advantage of the proposed attack, especially if the model has the same architecture or is trained on the same dataset as the victim model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject