Training with More Confidence: Mitigating Injected and Natural Backdoors During Training
Authors: Zhenting Wang
Hailun Ding
Juan Zhai
Shiqing Ma
Venue: NeurIPS 2022 Accept (2022)

Abstract: The backdoor or Trojan attack is a severe threat to deep neural networks (DNNs). Researchers find that DNNs trained on benign data and settings can also learn backdoor behaviors, which is known as the natural backdoor. Existing works on anti-backdoor learning are based on weak observations that the backdoor and benign behaviors can differentiate during training. An adaptive attack with slow poisoning can bypass such defenses. Moreover, these methods cannot defend natural backdoors. We found the fundamental differences between backdoor-related neurons and benign neurons: backdoor-related neurons form a hyperplane as the classification surface across input domains of all affected labels. By further analyzing the training process and model architectures, we found that piece-wise linear functions cause this hyperplane surface. In this paper, we design a novel training method that forces the training to avoid generating such hyperplanes and thus remove the injected backdoors. Our extensive experiments on five datasets against five state-of-the-art attacks and also benign training show that our method can outperform existing state-of-the-art defenses. On average, the ASR (attack success rate) of the models trained with NONE is 54.83 times lower than undefended models under standard poisoning backdoor attack and 1.75 times lower under the natural backdoor attack. Our code is available at https://github.com/RU-System-Software-and-Security/NONE.

Training with More Confidence: Mitigating Injected and Natural Backdoors During Training
Zhenting Wang Rutgers University zhenting. wangorutgers.
Hailun Ding Rutgers University
hailun. dingorutgers. edu
Juan Zhai
Shiqing Ma
Rutgers University juan wwwzzzaaiirrara www.zhaaiitttteteeeee
Rutgers University sm2283@rutgers.com edu
Abstract
The backdoor or Trojan attack is severe threat to deep neural networks (DNNs). Researchers find that DNNs trained benign data and settings also learn backdoor behaviors, which is known as the natural backdoor. Existing works on anti-backdoor learning are based on weak observations that the backdoor and be- nign behaviors can differentiate during training. An adaptive attack with slow poisoning can bypass such defenses. Moreover, these methods cannot defend natural backdoors. We found the fundamental differences between backdoor- related neurons and benign neurons: backdoor-relates neurons form a hyperplane as the classification surface across input domains of all affected labels. By fur- ther analyzing the training process and model architectures, we found that piece- wise linear functions cause this hyperplane surface. In this paper, we design a novel training method that forces the training avoid generating such hyper- planes and thus remove the injected backdoors. Our extensive experiments on five datasets against five state-of-the-art attacks and also benign training show that our method can outperform existing state-of-the-art defenses. On average. the ASR (attack success rate) of the models trained with NONE 54.83 times lower than undefended models under standard poisoning backdoor attack and 1.75 times lower under the natural backdoor attack. Our code is available at https: github...... www.co///////// RRU-Systen: Sof Software- and-Security/NON..
Introduction
Deep Neural Networks (DNNs) are vulnerable to Trojans'. A Trojaned model makes normal pre- dictions on benign inputs and outputs the target label when the input contains a specific pattern (i.c., Trojan trigger) such as yellow pad. To inject Trojan [1-7], the adversary can poison the training dataset by adding poisoning samples (or Trojan samples): inputs stamped with the Trojan trigger and labeled the target label. This is typical data poisoning attack, and the model can learn the trigger as a strong feature for the target label. Recently, researchers found the existence of natural Trojans. Namely, model trained optimizers) can also learn Trojans, when there exists strong input pattern in the training dataset that corresponds one label [8]. In such natural Trojan scenarios, the input pattern serves Trojan trigger, and its corresponding label is the target label. By reverse-engineering the trigger, the adversary can leverage for attacks. As such, both injected and natural Trojans severe threats.
benign datasets with normal settings (c.g., hyperparameters,
'Trojan attack is also known the backdoor attack in the existing literature.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

There are no existing works for learning robust DNN against both injected and natural Trojans. Existing works focus on training benign classifiers when the dataset is poisoned. For example, ABI. [9] observes that the model will learn backdoor behavior faster than benign behavior, and proposes a training algorithm to suppress learning the trigger pattern. DP-SGD is an optimization method that leverages the differential privacy (DP) method and combines it with stochastic gradient descent (SGD) to learn a robust classifier using poisoned datasets. These methods fail to defend against the natural Trojan.
In this paper, we propose a robust training algorithm that can mitigate both injected and natural Trojans. For a given Trojan in the form of σ m)ex + m where è and x are respectively the poi- soning and benign samples, i is the poisoning sample generation method using the trigger (m. t) with trigger mask matrix m and trigger value matrix t, we theoreti- cally prove that there exists one and only one hyperplane in the input space that corresponds to all poisoning sam- ples. Thus, the trained classifier is mapping this hyper- plane to a target label when performing Trojan attacks. Fig. 1 intuitively illustrates our idea using simple exam- ple. To simplify the problem, each input has three dimensions (dz, dy. dz). We use red and blue dots to denote inputs belonging to different labels, and the trigger is denoted as t (-.- - 24). 2
T(x, m.
(1
Z.
Prediction: Cal
Benign,
fop
Benign.
mrieeer
Prodiction: Dog
(c-2))
Trojon Sample, Prediction: Dog
t-
M1) Trigger
Fig. 1: Decision Region of A Trojan Model.
Adding
the trigger into an input in = (Io, yo. zn) to get the corresponding Trojaned input in = (To: yo, ze) is equivalent to moving this input to the the input in moves along the dashed line and ends up in the wheat colored plane, 2 = 21- Notice that in can be any input, and stamping the trigger will move them to the plane = 2, In other words, the plane 2 = 2t contains all Trojan samples. Likewise, if there exists a plane = 2t that a decision region, its corresponding input pattern (-, -,,-,, z+) is Trojan trigger. Stamping such trigger to any input is essentially projecting the imput to this plane, and because all the inputs in this plane have the same label, it is equivalent to performing Trojan attack. Extending this to the high dimensional space, the Trojan region will be hyperplane, and the decision boundaries it shares with other regions will be linear. In summary, we can say that Trojan in DNN always pairs with a hyperplane as its Trojan region. Considering that modern DNNs are non-lineat and non-conver this rarely happens for benign models. By further analysis, we found that this is related to the use of activation functions. Modern DNNs tend to use piece-wise linear functions as their activation func- tions. Even though the function itself is linear, its sub-functions are linear. For example, one of the most popular activation function, ReLU (i.e, y max(0, z)), consists of two linear functions (i.e., y = 0, S and y = x, x > 0). When model's weights and biases are trained to specific regions. the neuron values before activation functions will fall into the input domain of only one sub- functionnnn (c.g.. x > 0). As a result, the output and input will form a linear relationship. Consequently, the model can generate hyperplane decision region in the input domain. In other words, we have hyperplane, denoted as < Tp. I1, I2 >=< ag, 01, Q2 >, as decision region, and for a given input turn its
24 plane in the input domain. As shown in Fig. 1,
if we replace its values in dimensions Fo. 31 and 12 to dg, 01 and d22 respectively, output label to a desired one. A model training process includes randomness (e.g.. random initiation values, optimization), which cannot avoid. Many possible decision regions will give us the same or similar training/vatidation accuracy. Some training will learn a linear decision region while others will not. This explains the cause of DNN Trojans and answers the former question, which moves forward our understanding of DNN Trojans one more step.
Based on this analysis. develop revised training method, NONE (NON- LinEErity) that identi- fies linear decision regions, filters out inputs that are potentially poisoned, and resets affected neu- rons to enforce non-linear decision regions during training. We evaluated prototype built with Python and PyTorch on MNIST, GTSRB, CIFAR, ImageNet, and the TrojAl dataset. Compared with SOTA methods, NONE is more effective and efficient in mitigating different Trojan attacks (i.e.. single-target attack, label-specific attack, tabel-consistent attack, natural Trojan attack, and the hidden trigger attack). For example, on average, the ASR (attack success rate) of the models trained with NONE is 54.83 times lower than undefended models under standard poisoning Trojan attack and 1.75 times lower under the natural Trojan attack.
2

---

Our contributions in this paper can be summered as follows. We analyze the cause of (injected and natural) DNN Trojans and conclude that linearity in DNN decision regions is the main reason. Fur- ther, we analyzed the source of linear DNN decision regions and explained why and when it happens using commonly used layers and activation functions. Then, we propose a novel and general revised training framework, NONE that detects and fixes injected and natural Trojans in DNN training. To the best of our knowledge, we are the first to defend natural Trojans. We evaluate NONE on five different datasets and five different Trojans and compare it with other SOTA techniques. Results show that NONE significantly outperforms these prior works in practice.
2 Related Work
Trojans in DNN. Prior work [1, 10-14, 2, 3, 15-18] demonstrates that the attackers can inject Trojans into the victim models by poisoning the training dataset. Later, researchers found that a DNN trained on the benign dataset with standard training procedure can also learn such Trojans, which is known as natural Trojans. By using reverse engineering methods designed for poisoned models, researchers were able find natural triggers pretrained models. For example, in ABS [8], authors show that a Network in Network (NiN) [19] model trained on benign CIFAR-10 10 dataset has natural Trojans. Other works [14, 20] also documented similar findings in other models.
Trojan Defense. One way defend against Trojan attacks is to filter out poisoning data before training [21, 22]. Poison suppression defenses [23, 24] restrain the malicious effects of poisoning samples in the training phase. Du et al. [23] and Hong et al. [24] apply DP-SGD [25] to depress the malicious gradients brought from poisoned samples. However, existing anti- nai-bbakdoor learning methods are based on weak observations and they do not find the root cause of Trojans. They can be bypassed by adaptive attacks. In addition, they can not defend natural Trojans. For example, the SOTA anti-backdoor learning method ABL [9] is based on the observation that learning of backdoor behaviors and benign behaviors are distinctive. In detail, the training loss on backdoor examples drops much faster than that on benign examples in the first few epochs. It can be bypassed by slow poisoning attacks. More specifically, when the backdoor attack is label-specific (i.c., the samples with different original labels have different target labels) with low poisoning ratios, the model will learn backdoor behavior slower. We run ABL on the label-specific BadNets attack [1] with ResNet 88 model and GTSRB dataset. The results show that the attack success rate is 93.93% meaning that ABL can not defend against such an attack. This is not surprising because the label-specific attack is more complex, and it based on the benign behavior of the models (i.e., the model needs to classify the original label of the backdoor examples first, then convert it to corresponding target labels). Another line of work tries to detect if a model has Trojan or not before its deployment. Model diagnosis based defenses [26-35] determine if given model has Trojan or not by inspecting the model behavior. Model reconstruction based defenses try to eliminate injected Trojans in infected models [36-40], which requires retraining the model with set of benign data. Another approach is to defend Trojan attacks at runtime. resting-based defenses [41-43] judge the given input contains trigger patterns and reject the ones that are malicious.
3 Methodology
Threat Model. In this paper, we consider training time defense against Trojan attacks, which is also used by existing works [24, 22, 21]. The training dataset can contain poisoning samples (to inject intended Trojans) or benign (but leads to natural Trojans). As defenders, we control the training process but do not assume control over training datasets. Adopted from existing work [26, 41, 42, 27], Trojan sample generation can be formalized as Eq. 1, where æ and き respectively are the benign input and Trojan input. m, respectively represent, mask of the trigger (i.e., whether a pixel is in the trigger region) and contents of the trigger. @ is the element wise multiplication operation on two vectors, i.e., the Hadamard product.
Trigger pattern
Hadamard product
T(x m
(1 m
m t
(1)
Trigger mask
3

---

3.1 Trojan Analysis
To facilitate our discussion, we first define decision region which includes all samples with the same predicted label. Formally, we define it as:
Definition 3.1. (Decision Region) For deep neural network .M X + y where X' is the input domain I and y a set of labels [1...k}. decision region is an input space R' S X, s.t., Vx R!, M(x) L.
In most tasks, decision regions with the same label spread over the whole input space because natural inputs belonging to the same label are naturally distributed this way. Similarly, we define Trojan Decision Region, or in short, we call it the Trojan Region, which is a subregion of the target label decision region:
Definition 3.2. (Trojan Region) For a Trojaned deep neural network M: : X' + ) with target label 1, its Trojan regions are input spaces where T C R', s.t., all Trojan inputs æ T, and all inputs in T are Trojan inputs.
Based on the definitions, we have the following theorem:
Theorem 3.3. Given model M X + y with the Trojan trigger (m.,), if the atluck is complete (100% attack success rate) and precise (no other triggers), there exists and only one hyperplane {Ax - b b= 0} Trojan region, where i E mm. diagonal matrix A., = m,,, b At.
The proof for Theorem 3.3 and empirical results are Appendix ($ 8.1). The theorem shows that when model learns a Trojan, it essentially learns hyperplane as decision region. Based on the definition of decision regions, we know that they are inverse functions of the model. Thus, the inverse function of the Trojan is hyperplane. To understand how popular model architectures learn such hyperplanes in practice, we perform further analysis.
We start our discussion from typical Convolutional Neural Networks (CNNs). A convolutional layer with activation functions can be represented yj a [W [ zr bb,) where xy and yj are the inputs and outputs of layer j. W, and by are trained weights and bias values, and a represents the activation function which used introduce non-linearity in this layer. Most commonly used activation functions, c.g., ReLU, are biceewwis linear. For example, ReLU is defined as (xx) = max(0, r), which consists of linear pieces separated even non-pi non-piece-wiss linear functions are trained to semi-piece-vee wise linear. This helps resolve the gradient sxplosion/vanishing problem and makes training DNNs more feasible.
As pointed by Goodfellow et al. [44],
We make a key observation that DNN Trojans will increase linearity of a convolutional layer with activation functions by introducing large percentage of neurons activating on one piece of the activation function. Specifically, we observe that when the Trojan behavior happens, the neuron values before activation functions fall into one input range of the activation function, which makes linear. Recall that most well-trained activation functions are piece-wise linear, and if the inputs are one input range, it regresses to linear function. For example, layer j using ReLU as its activation function will be a linear layer if W x, + b" > O for all 25. As such, the reverse function of the Trojan can hyperplane or overlaps significantly with the hyperplane.
Fig. shows the empirical compari- son of benign model and a Trojan model. In our experiment, we train a benign model M and Trojan model M' using ResNet18 on CIFAR-10. While training M', we adopt the Tro- janNet [45] training method which guarantees that only certain neurons will contain the Trojan, which we call Trojan neurons. The x-axis in Fig. shows the value of T + bT, and y-axis shows the percentage of Tro- jan neurons whose activation value is the corresponding value on x-axis. We use blue color to denote experiments when inputs have the trigger and orange to denote when inputs do not contain the trigger. As we can see, the model that
0.30
0.30
w/ Trojan
w/ Trojan
w/o Trojan
w/o Trojan
0.20
0.20
0.10
0.10
0.00
0.00
-0.5
1.0
-0.5
0.5
1.0
(a) Compromised
(b) Benign
Fig. Comparison of Activation Values.

---

Algorithm Training
Input:
Training Data: D, Maximal epoch: E
Output:
Model:
1: function TRAINING(D)
2: while E and (not ())) do
3:
> Train and gather activation values
4:
M train(D,e)
5:
A = M.predict D)
6:
Identify compromised neurons
7:
C -
8:
for neuron n in M do
9:
if A, 0) 2 then
C [n)
11:
Identify biased poisoning samples for neuron n E C do
13:
BR On (Aparate(A,))
[a, a = norm(BB)
15:
for activation value of input i: in E O, do
16:
if 1 2 À then
D
18:
Resetting compromised neurons for neuron é E C' do M.int((c)
19:
20:
contains the Trojan will activate the 0 region when the input contains the trigger. By contrary, all the other three cases do not have such a phenomenon. We also conduct similar experiments on other different models and different types of layers including non-picce-wise linear layers, and all empirical results confirm our observation here. Details are in Appendix ($ 8.2).
4 Training Algorithm
To solve the Trojan problem caused by high linearity neurons, we propose Algorithm to enforce non-linearity in individual layers by resetting potentially linear neurons and removing data samples that cause such linearity. This is a revised training process, which is an iterative process that trains the model until the maximal epochs (line 2). It starts by training the model using the standard backward propagation method (line 4). We also gather all activation values of individual neurons A for all training samples in the training dataset D. Then, the process contains three steps: identifying compromised neurons (lines 6 to 10), identifying biased or poisoning samples (lines 11 to 17). and lastly, resetting the neurons for retraining (lines 18 20).
The first step is to identify compromised neurons, namely the neurons that carry Trojans. Based on discussion
3.1, do this by checking the activation values of neuron denoted as An to see its function is highly linear using the condition P(A, 2 0) 2 9. If so, we make the neuron n as potentially compromised and add it to the candidate set C. The second step is to identify highly biased samples or poisoning samples. The overall design is a statistical testing process: we first find reference distribution of partieular neuron and then mark all inputs whose activation values do not follow such distributions as potentially biased or poisoning samples. The idea of finding a benign distribution that inputs whose activation values in layer are non-linear will be considered as benign and the distribution that describes their activation values is used as the reference distribution. In Algorithm 1, line 13 uses function test the linearity of activation values and separate all activation values into benign set B and others O. Then, we normalize the distribution of our reference distribution B normal distribution and obtain its mean value μ and standard deviation value a (line 14). For single input i, we perform statistical test to see the probability of i being a potentially biased or poisoning sample (lines 15 to 16). We exclude from training dataset (line 17). The last step of our training algorithm is remove the compromised neuron effects from the model by resetting them. We reuse the initialization method in our training setting set new value for all identified compromised neurons (line 19 and 20). Then, we continue the training

---

until terminating conditions are met, or the training budget is used up (line 2). This algorithm uses ReLU as an example, and because our theory still holds for other activation functions (see $ 8.2), the algorithm can generalize to other models by modifying corresponding parameters (line 9).
To identify compromised neurons and biased/poisoning images, we need to determine the threshold value 0 (line 9) and separate the activation values into non-linear and linear ones. In our implemen- tation, we perform the Fisher's linear discriminant analysis (its binary version, Otsu's 's method [46]) and leverage the Jenks natural breaks optimization algorithm to find the separations of non-linear and linear activation values. Sets and O are outputs of such algorithms, and we use the standard value that evaluates the quality of such separation to compute the value of our threshold 6, i.e., 0.95 in our case. Notice such values can affect the accuracy of identifying compromised neurons and inputs. We also choose alternative ways to separate the sets and present the results in the Appendix to evaluate our approach. Similar to model pruning and fine-tuning, when we reset different num- bers of neurons (lines 19 and 20), the accuracy on benign and Trojan samples can be different. We present more details of the algorithm and evaluate the sensitivity of NONE to the number of neurons reset during training and identify malicious samples in Appendix ($ 8.8).
5 Experiments
NONE is implemented in Python 3.8 with PyTorch 1.7.0 and CUDA 11.0. If not specified, all experiments are done on a Ubuntu 18.04 machine equipped with six GeForce RTX 6000 GPUs, 64 2.30GHz CPUs, and 376 GB memory. We first introduce the experiment setup ($ 5.1). Then, we evaluate the overall effectiveness of NONE ($ 5.2) and investigate its robustness against different attack settings ($ 5.3). We also evaluate the generalization of NONE real-world applications ($ 5.4). Finally, we measure the precision and the recall in the poisoned samples identification stage (S 5.5). Other results and discussions are in Appendix.
5.1 Experiment Setup.
Datasets and Models. We evaluate NONE on five publicly available datasets: MNIST [47], GT- SRB [48]. CIFAR-I0 [49], ImageNet-10 [50] and TrojAI [51]. The overview of our datasets and more details can be found in $ 8.5. Fig. 3 illustrates different trigger patterns used in the ex- periments (i.c., a single pixel located in the right bottom corner of Fig. 3(a), fixed red patch in Fig. 3(b), a black-and-whit pattern whose location is random in Fig. 3(c) and colorful water- mark in Fig. 3(d).. The default trigger pattern for MNIST, CIFAR- 10, GTSRB, and ImageNet- 10 are Single Pixel (Fig. 3(a)), Static Patch (Fig. 3(c)), Dynamic Patch (Fig. 3(b)) and Watermark (Fig. 3()), respectively. Besides the default triggers for each dataset, we also measure the impacts of using other triggers. The results are included in $ 5.3. We evaluate NONE and other defense meth- ods on AlexNet [50], NiN (Network Network) [19], VGGI1, VGG16 [52] and ResNet18 [53]. These models are representative and are commonly used in existing Trojan related studies [8, 26].
Evaluation Metrics. We use benign accuracy (BA) and attack success rate (ASR) [54] as evaluation metrics, which is a common prac- tice [55, 41, 26]. BA is defined as the num- ber of correctly classified benign samples over the number of all benign samples. It implies model's capability on its original task. ASR evaluates the success rate of backdoor attacks. is calculated as the number of backdoor sam- ples that can successfully attack the model over the number of all generated backdoor samples.
8
80
(a) Single Pisel
(b) Statie
(c) Dynamic
(d) Watermark
Fig, 3: Examples of Using Different Trigger Patterns.
Attack Settings. As we introduced in 2. Trojans are classified into two categories: Injected Trojans and Natural Trojans. We implement both of them to evaluate NONE and other defense methods. For Injected Trojans, we implement both single target and label specific BadNets [1], label-consistent Trojan attack [11] and hidden trigger Trojan attack [56]. Following the original paper, we use images from a pair of classes (class theater curtain and class plunger) in ImageNet for hidden trigger Trojan attack. For natural Trojans, we follow the previous work [8] to reproduce the
attack. Due to the fact that we do not know if the data is poisoned or not advance, NONE keeps detecting Trojan attacks and training the model even if there is no attack activity. We evaluate the
6

---

Table 1: Comparisons on Injected Trojans.
Altack Type
Daiaeet
Network
Undelended
DP-SGD
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BA
ASR
BA
ASR
NIN
99.65%
99.96%
92.94%
0.48%
99.26%
0.06%
98.47%
0.515
98.25%
99.70%
99.58%
0.06%
MNIST
VGGE
99.35%
100.00%
97.20%
98.85%
2.125%
97.44%
10.93%
98.48%
0.64%
99.11%
0.14%
R8
99.615
99.98%
97.31%
0.13%
98.83%
0.12%
98.04%
0.25%
99.34%
0.04%
99.57%
0.37%
NiN 90.52%
100,009
36.41%
93.01%
80.99%
57.97%
83.77%
100.00%
90.13%
100.00%
90.11%
2.32%
BodNets
CIFAR- 10
VGGI6 90.16%
100.00%
55.61%
99.32%
88. 70%
98.71%
88.1
99.811
89.96%
100.00%
39.70%
4.91%
(singlo target)
ResNeti8
94.10%
100.00%
52.29%
99, 99.99%
88.74%
1.28%
89,57%
57,43%
92.40%
1.66%
93.62%
1.07%
NiN
95.95%
99, 99.72%
31.54%%
74.78%
96.53%
26.
95.366
99.5216
96, 1.4%
99.54%
95.51%
0.57%
GTSRB
VGG16
95,479%
99.99.%
54.60
86.81%
95.94%
86, 1098
91. 28%
5.28%
94.06%
95. 799
94 94.66%
0.96%
ResNet18
96.67%
55.7316
95.951
13.
94.669%
98,57%
96.24%
0,93/%
96.39%
0.76%
NiN
62.17%
999 999.6495
21 22%
65.21%
76,99%
92.611
74. 32%32%
91.41%
81.35%
99.929%
80.315
0.14%
ImageNet-10
VGG16
100.00%
44.62%
81.02/%
10,93/8
81.58%
100,00%
100.00%
87.11% 0.14%
RosNet18
89.8315
99,0775
40.31%
29.58'
86, 29%
25 8998
22.39%
98.4415
83.721
6,83%
86.34%
0.08%
NiN
99.644%
93.02%
1.10%
99.24%
0.09%
99 99.60%
0.03%
98.40%
89.32%
99.42%
0.03%
MNIST
VGG1
99,05%
98.99%
97.17%
0.411%
99.0669
97. 97.599%
0.10%
99.21
98.514
98.63%
0.11%
ResNet18
99,571
99.49%
96 96.81%
0.38%
19.19%
199
99.35%%
0.03%
99.01%
98.81%
99.09%
0,209%
NiN
S0.50K4
78.56/%
38.42%
6.43%
82.70%
13.6555
84, 74%
56, 577
89.66-
77, 6466
89. 51.5
1.27%
BadNets
CIFAR-10
VGG16
90.73%
96. 86.8
55.12%
5.23%
88, 65.655%
39, 99%
88.118%
82. 877
90.13%
91%
89 59.61
1.22%
(label specific)
ResNet18
94.37%
92.00%
52.19%
12.17%
88.21%
1.50%
92.68%
5.29%
8.3.833
833.48%
93.05%
1.04%
NiN
96.06%
93.74%
24, 70%
7.39%
96.46 9%
9. 5099
94 02.%
6.17%
16, 14%
14 .4%5%
95 9999
0.96%
GTSRB
VGG16
95.71%
91.57%
53.90%
6.71%
96.33%
81.1
91.32%
80.21%
95.19%
1.5%
Resher18
96.93%
97.40%
60.59%
6.61%
95.49%6
13.4
95.74%
1.16%
90.24%%
93.93%
96.63%
0.91%
82.65%
66.37%
23.44%
10.06%
76.31%
23.3
78.85%
14.39%
80.20%
56.82%
79.391% 6.98%
ImageNes-11 10
VGG16
89.04%
76.20%
17.94
10.80%%
80.2114
22, 1746.
81 .222
59.62%%
78. 14%
53. (6.%
84.51 %% 8.31 %
ResNet18
88.87%
59.75%
44.59%
7,95%
85.76/
15.822 15.8
79.44%
23/41%
81.30%
50.52%
84.74%
6.55%
NiN
91.32%
98.98%
38.83%
5.21%
82.35%
65. 633%
83 83.92%
95.96%
89.95%
95 95.28%
90.11%
2.19%
Label comsistent
CIFAR-10
VGG16
90.97%
98.11%
53.76%
9.21%
88.88%
92.71'%
88, /6%
7.57%
90.38%
91,918
90.07%
4.26%
ResNet18
94.73%
83.42%
54.00%
11.00%
90.74%
60.
88.00%
65.86%
87.34%
2.17%
2.14%
Hidden Trigger ImageNettpuir Nlex
93.00%
82.00%
80.00% 62.00%
91.00% 74.00%
90.00%
22.00%
90,00%
54,00%
91.00%
4.00%
Table 2: Comparisons on Natural Trojan.
Undefended
DP-SGD-1
DP-SGD-2
NONE
Dalaset
Netwerk
BA
ASR
BA
ASR
BA
ASR
BA
ASR
CIFAR-10
NIN VGG16
91.02%
87.62%
60.22%
98.85%
39.19%
87.22%
86.94%
34.21%
90.78%
71.88%
78.25%
61.11%
53.40%
63.58%
81.33% %
37.49%
TrojAI
VGG11
99.88%
72.09%
84.51%
88.75%
6.06%
88.55
99.04% %
56.68%
Resnet18 18
99.91%
54.33%
78.88%
61.77%
52.81%
58.94%
98.13% 13%
34.98%
additional costs of NONE on benign models in such non-attack settings. More details for our attack settings are included in $ 8.6. Besides above attacks, We also evaluate the generalization of NONE on more attacks in $ 8.7.
Comparison. We compare NONE with 4 state-of-the-ant defense methods: DP-SGD [24], Neural Attention Distillation (NAD) [38], Activation Clustering (AC) [22] and Anti-backdoor Learning (ABL) [9]. We use their official code and default hyperparameters specified in the original papers.
5.2 Effectiveness of NONE
Experiments. We measure the effectiveness of NONE by comparing the BA and ASR of models protected by NONE with those of undefended models and models protected by existing defense methods. The comparison results on injected Trojans, natural Trojans and non-attack settings are shown in Table 1, Table 2 and Table 3, respectively. In each table, we show the detailed settings including attack settings, dataset names and network architectures.. etc. For the evaluation results on natural Trojans ( Table 2), the ASR and BA are the average results under different trigger size settings: 2%, 4%, 6%. 8%, 10% and 12% of the whole image. Notice that, to the best of our knowledge, there is no defense method designed for natural Trojans. We observe that DP-SGD can potentially mitigate natural Trojans because it reduces the high gradients brought from natural Trojans. Therefore, we adapt DP-SGD as the baseline method for natural Trojans. We configure DP-SGD with two settings of parameters following the prior work [24]. For DP-SGD-1, we set the clip as 4.0 and the noise as 0.1. The clip and noise of DP-SGD-2 are 1.0 and 0.5. For the non-attack settings, we deploy NONE on several benign models and show the decrease of BA in Table 3.
7

---

Table 3: Benign Accuracy in Non-attack Settings.
Table 4: Results on Different Trigger Patterns.
Dataset
Network
Without NONE
With NONE
Trigger Pattern Network
Undefendad
NONE
NiN
91.02%
89.40%
BA
ASR
BA
ASR
CIFAR-10
VGGL6
90.78%
89.62%
NIN
90.52%
100.00%
90.11%
2.32%
ResNet18
94.83%
93.92%
Dynamic Putch
VGG16
90.468
100.00%
89,70%
4.91%
ReaNet18
94.10%
100.00%
93.62%
1.07%
NIN
68%
95.36%
GTSRB
VGG16
94.67%
94.08%
90.92%
100.00%
89.93%
ResNet18
96.8918
96.87%
Static Patch
VGG16
90.12%
100.00%
89:18%
1.03%
ResNet18
94.24%
99.99%
93.93%
1.37%
NIN
83.34%
79.18%
NIN
90.8885
99.9
87. 74%
3.27%
ImageNet-10 10
VGG16
88.84%
Watermark
VGG16
90.61%
99.99%
89, 14%
5.36%
ResNei18
89.81%
85.25%
ResNet18
94.28%
100.00%
92.27% 5.99%
Table 5: Results on Different Trigger Sizes.
Table 6: Results on Different Poisoning Rates,
Trigger Size
Undefended
NONE
Undefended
NONE
ASR
BA
Poisoning rate
ASR
ASR
3*3
94.10%
100.00%
93.62%
1.07%
515
94.275
100,005
93.515
1.34%
0.50%
94.46%
100.00%
93.50%
2.52%
747
94.10%
99.98%
93.70%
1.53%
5.00%
94.10%
100.00%
93.62%
1.07%
999
94.34%
100.00%
93.19%
5.18%
10.00%:
93.82%
100.00%
93.13%
1.04%
15"15
94,44%
100.00%
92. 18%
32.27%
20.00%
92.70%
100.00%
92.14%
1.39%
20"20
94.58%
100.00%
92.55%
99 99.82%
Results on Injected Trojans. From the results on Injected Trojan attacks (Table 1), observe that applying NONE can better protect models from being attacked by injected Trojans than other defense methods. With NONE, the average ASR of models decreases from 93.34% to 1.91% which is much better than other defense methods (DP-SGD, NAD, AC and ABL can only reduce the average ASR 30.32%, 34.08%, 45.45% and 68. 60% respectively). The reason is that, unlike existing methods based only on specific empirical observations, NONE targets the root cause of the Trojans (i.e., the linearity) and reveals the attacks more accurately. Therefore, NONE can better defend against attacks than other methods.
We also find that NONE almost does not have negative impacts on the original task of models. From Table I, the BA of NONE is the highest among all methods and is similar to that of undefended models, meaning that NONE has low defense cost. The reason is that NONE only modifies the compromised neurons that are highly relevant to Trojan but less related to the original task of models. Therefore, most of the benign knowledge is preserved when applying NONE and the model can still perform well on its original tasks. Meanwhile, NONE finetunes the model on the purified data after the reset process, further strengthening the capabilities of models and reducing defense costs.
worth clarifying that ABL has a poor performance on label-specific attacks and other attacks that use NiN and VGG models. The possible reason is that the design of ABL requires the model to learn quicker and better on Trojan samples than benign samples [9] (i.c., the lcarning of Trojan samples should have lower training loss value in the early training stage). When the attacker uses more complex attacks (e.g., label specific BadNets) or models with limited learning capabilities (e.g., NiN and VGG), the model cannot learn Trojan samples quickly, leading poor performance.
Results on Natural Trojans. From the results in Table 2. we find that NONE protects the model most effectively and has the lowest defense costs among all defense methods. Overall, apply- ing NONE achieves 1. 1.75 times lower ASR than undefended models. DP-SGD methods can only slightly decreases ASR or even increase ASR. The results show that using NONE is the most ef- fective way for natural Trojan defense. The loss of BA using NONE is also smaller than the loss caused by applying the DP-SGD methods (3.90% with NONE and 38.76% with DP-SGD methods on average), further showing the efficiency of NONE. The results confirm our analysis: reducing the linearity of models reduces the ASR of natural Trojans without posing much additional cost.
Results on Non-attack Setting. We also explore whether NONE affects the performance of benign models on their original tasks. From Table 3, we find that NONE has a low effect on benign models. Applying NONE only decreases 2.33% BA on average. Because only a few neurons are detected as compromised neurons and are reset by NONE when there is no Trojan activity, NONE does not affect the learned benign knowledge. Moreover, the subsequent training process further reduces the costs. Therefore, we conclude that NONE does not impose high additional costs on benign models.
8

---

Table 7: Results Federated Learning.
Undefended
NONE
Dataset
BA
ASR
MNIST
99.22% 80.31%
99.15% 43.23%
98. 99.60%
0.13%
CIFAR-10
78.67%
4.44%
5.3 Robustness of NONE
We evaluate the robustness of NONE against various attack settings (c.g,, different trigger sizes, trigger patterns and poisoning rates). If not specified, the model used in the evaluation is ResNet18. The dataset is CIFAR-I0 and the evaluated attack is the single target BadNets attack.
Trigger Sizes. To study the effects of trigger size, we use triggers of different sizes (from 3*3 to 20* to attack models and collect the ASR and BA of applying NONE on these compromised models. The results are shown in Table 5. Overall, the BA of undefended models and models protected by NONE is insensitive to the change of trigger size. The difference between the highest and lowest BA on the unprotected and protected models is 0.48% and 1.52%, respectively, which very small. We believe that the BA does not change significantly because triggers usually do not affect the learning of benign features used for original tasks, as discussed in previous work [1].
On the other hand, the trigger size affects the ASR of protected models. When the trigger size becomes larger, the ASR of the protected model increases dramatically from 1.07% to 99.82% and NONE fails. The results are understandable because a large trigger size modifies more pixels in the original image, making the triggers obvious and easy to learn. When the trigger is large, it almost covers the whole image and becomes the majority of the image. In such scenario, models easily capture trigger features and are compromised. A detailed example is shown in Fig. 10. Currently, the sensitivity to trigger sizes a common limitation for Trojan defense methods 126, 8). Considering that a large trigger size is almost impractical because it makes the trigger too obvious to be detected directly by administrators, we consider NONE robust to most trigger sizes.
Trigger Patterns. To measure the robustness of NONE against different attack trigger patterns, we use NONE to protect models from being attacked by Dynamic Patch trigger, Static Patch trigger and Watermark trigger. The results are shown in Table 4. We observe that NONE always achieves low ASR and high BA under different trigger settings. The results demonstrate that NONE is effective against different trigger pattern settings. Moreover, we notice that the ASR of using the watermark trigger is particularly larger compared with using other triggers. The reason is that the watermark triggers are large and more complex, as shown in Fig. 3.
Poisoning Rates. To measure the impacts of different poisoning rates, collect the ASR and BA of models being compromised at different poisoning rates from 0,50% to 20.00%. The results are summarized in Table 6. Based on the results, we find that increasing the poisoning rate slightly decreases both the BA and ASR. Specifically, the BA of the model decreases by 1.36% and the ASR of the model decreases by 1.13%. This is because increasing the poisoning rate reduces the number of benign samples used for training, and the BA of the model naturally decreases. Meanwhile, a large poisoning rate makes Trojans easy be detected, which leads to lower ASR. Since the changes in ASR and BA are quite small, NONE is considered robust to most poisoning rates.
5.4 Generalization on Complex Applications
Attackers may conduct attacks in a more complex scenario. To measure the generalization of NONE on complex applications, we evaluate NONE on two federated learning applications trained on different datasets (i.e., MNIST and CIFAR-10) and transfer learning application. Each federated learning application has 10 participants, of which 4 of them are malicious participants who conduct the distributed Trojan attacks [7] jointly to inject Trojan triggers into the global model. We assume that attackers train their local models on the poisoned training data and contribute to the global model without scaling the original weight of the poisoned local models. We then apply NONE on the global model to defend against the Trojan attack from malicious local models. Specifically, NONE requires the participants to use their data to test the global model and upload the activation values of the global model to identify compromised neurons. To measure the defense performance,
9

---

Table 8: Precision and Recall of Poisoned Samples Identification.
NiN
VGG16
ResNet18
Attack
Precision
Precision
BadNets
99 60.0%%
64%
99 96%
100.00%
99.84%
99.92%
Lahsl-consistent
98.80%
99. 20%
100.00%
100.00%
100.00%
100.00%
we measure the BA and ASR of the global models (i.e., the original model and model deployed with NONE). The results shown in Table 7. As shown in the table, average, NONE achieve 31.16 times lower ASR than undefended models with slight decrease (i.e., 1.13%) in the BA. The results show that NONE applications at a low cost. Besides federated learning, we have also discussed the performance of NONE in transfer learning settings. Hidden Trigger Trojan Attack [56] in 5.2 of the main paper is conducted in transfer learning scenarios and the results are shown in Table the results show, NONE achieves low ASR (i.e., 4.00%) and high BA (i.c., only 2.00% lower than undefended models), proving the generalization of NONE on transfer learning settings.
defend against the Trojan attacks effectively in real-world federated learning
the main paper. As
5.5 Precision and Recall of the Poisoned Sample Identification
In line 12-17 of Algorithm 1, we identify the poisoned samples in the training data. To evaluate the effectiveness of the identification process, we measure the precision and the recall of detect- ing poisoned samples on the CIFAR-10 dataset and three different models (i.e., NiN, VGG16, and ResNet18). Results in Table demonstrate the precision and the recall of NONE are always above 98% on different settings. For example, on ResNetl and Label-consistent attack, both the precision and the recall of NONE are 100.00% Thus, NONE can detect poisoned samples accurately.
Discussion
In this paper, focus the discussion image classification tasks, which is the focus of many existing works [1, 56, 11, 8, 26, 57]. Expanding our work, including the theory and system to other problem domains, such natural language processing and reinforcement learning, other computer vision tasks, e.g., object detection, will be our future work.
Research on adversarial machine learning potentially has ethical concerns. In this research, propose a theory to explain existing phenomena and attacks, and propose a new training method that removes Trojans in DNN model. We believe this is beneficial to society.
In our current threat model, the adversary can only inject poisoning data into the training dataset, and there is no existing adaptive attacks. However, adaptive adversaries can still conduct attacks under other threat models, and we discuss such case 8.10.
7 Conclusion
In this paper, we present an analysis on DNN Trojans and find relationships between decision regions and Trojans with formal proof. Moreover, we provide empirical evidence support theory. Furthermore, we analyzed the reason why models will have such phenomena is because linearity of trained layers. Based this, propose novel training method to remove Trojans during training, NONE, which can effectively and efficiently prevent intended and unintended Trojans.
Acknowledgement
We thank the anonymous reviewers for their valuable comments. This research is supported by IARPA TrojAI 19-- Any opinions, findings, and conclusions expressed in this paper are those of the authors only and do not necessarily reflect the views of any funding agencies.
10

---

References
[1] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. IEEE Access, 2019.
[2] Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pre- trained encoders in self-supervised learning. 2022 IEEE Symposium Security and Privacy (SP), 2022.
[3] Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In International Conference on Learning Representations, 2021.
[4] Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo, Xinyu Xing, and Dawn Song. Backdoorl: Backdoor attack against competitive reinforcement learning. International Joint Conference on Artificial Intelligence (IJCAI), 2021.
[5] Eugene Bagdasaryan and Vitaly Shmatikov. Spinning sequence-to-sequeeee models with meta-backdoors. arXiv preprint WrXiv:2107.10443, 2021.
[6] Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. Badnl: Backdoor attacks against nlp models with semantic- preserving improvements. In Annual Computer Security Applications Conference, pages 554- 569, 2021.
[7] Chulin Xie, Keli Huang, Pin- Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. In International Conference on Learning Representations, 2019.
[8] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1265- 1282, 2019.
[9] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. Advances Neural Information Processing Systems, 34, 2021.
[10] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 17Xivv1712.055566, 2017.
[11] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arxiv: 19/2.077/ 2019.
[12] Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang. Dynamic backdoor attacks against machine learning models. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 703-718. IEEE, 2022.
[13] Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Infor- mation Processing Systems 30. Pre-proceedings, 2020.
[14] Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon in the variant: Statisti- cal analysis of dnns for robust backdoor contamination detection. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
[15] Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Backdoor attacks on self- self-supervised learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13337-13346, 2022.
[16] Yiming Li. Yanjie Li, Yalci Lv, Yong Jiang, and Shu-Tao Xia. Hidden backdoor attack against semantic segmentation models. arXiv preprint arXiv:2103.04038 2021.
[17] Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. Deep feature space trojan attack of neural networks by controlled detoxification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1 1148-1156, 2021.

---

[18] Zhenting Wang. Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15074-15084, 2022.
[19] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. International Conference on Learning Reprexentations, 2014.
[20] Shihao Zhao, Xingjun Ma, Yisen Wang, James Bailey, Bo Li, and Yu-Gang Jiang. What do deep nets learn? class-wise patterns revealed in the input space. arXiv preprint arXiv:2101.066 066898 2021.
[21] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Ad- vances in neural information processing systems, 31, 2018.
[22] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Tae- sung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural net- works by activation clustering. SafeAI@AAA, 2019.
[23] Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. International Conference on Learning Representations, 2020.
[24] Sanghyun Hong, Varun Chandrasekaran, Yigitcan Kaya, Tudor Dumitraş, and Nicolas Paper- not. On the effectiveness of mitigating data poisoning attacks with gradient shaping. arXiv preprint arXiv:2002. 1 1497,, 2020.
[25] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[26] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723, IEEE, 2019.
[27] Xiaojun Xu, Qi Wang. Huichen Li, Nikita Borisov, Carl Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP), pages 103-120. IEEE, 2021.
[28] Sohcil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 301-310, 2020.
[29] Shanjiaoyang Huang, Weiqi Peng, Zhiwei Jia, and Zhuowen Tu. One-pixel signature: Char- acterizing enn models for backdoor detection. In European Conference on Computer Vision, pages 326-341. Springer, 2020.
[30] Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems. IEEE International Con- ference Data Mining (ICDM),, 2020.
[31] Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu. Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm opti- mization. In International Conference on Machine Learning, pages 9525-9536. PMLR, 2021.
[32] Yingqi Liu, Guangyu Shen, Guanhong Tao, Zhenting Wang, Shiqing Ma, and Xiangyu Zhang. Complex backdoor detection by symmetric feature differencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15003-15013, 2022.
[33] Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma, Pan Li, and Xiangyu Zhang. Better trigger inversion optimization in backdoor scanning. In Pro- ceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 19368-13378 2022.
12

---

[34] Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Piccolo: Exposing complex backdoors in nip transformer models. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1561-1561. IEEE Computer Society, 2022.
[35] Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu, Zhuo Zhang, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Constrained optimization with dynamic bound- -caliin for effective nlpbackdoor defense. In International Conference on Machine Learning. 2022.
[36] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against back- dooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pages 774-294. Springer, 2018.
[37] Pu Zhao, Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. arXiv preprint 200500005.00060, 2020.
[38] Yige Li, Nodens Koren, Lingjuan Lyu, Xixiang Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. International Conference on Learning Representations, 2021.
[39] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34, 2021.
[40] Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Represen- tations, 2022.
[41] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pages 113-125, 2019.
[42] Edward Chou, Florian Tramèr, and Giancarlo Pellegrino. Sentinet: Detecting localized uni- versal attacks against deep learning systems. 2020 IEEE Security and Privacy Workshops (SPW), pages 48 54. IEEE, 2020.
[43] Shiqing Ma and Yingqi Liu. Nic: Detecting adversarial samples with neural network invariant checking. In Proceedings of the 26th network and distributed system security symposium (NDSS 2019), 2019.
[44] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. International Conference on Learning Representations, 2015.
[45] Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. An embarrassingly simple approach for trojan attack in deep neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 218-228, 2020.
[46] Nobuyuki Otsu. A threshold selection method from gray- histograms. IEEE transactions on systems, man, and cybernetics, 911):22-66, 1979.
[47] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-23) 1998.
[48] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man computer: Benchmarking machine learning algorithms for traffic sign recognition. Newral networks, 32: 323-332, 2012.
[49] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[50] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 20:1 1105, 2012.
[51] Kiran Karra, Chace Ashcraft, and Neil Fendley. The trojai software framework: An opensource tool for embedding trojans into deep learning models. arXiv preprint arXiv:2003.072333 2020.
13

---

[52] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1499. 1556, 2014.
[53] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recogni- tion, pages 770-778, 2016.
[54] Akshaj Kumar Veldanda, Kang Liu, Benjamin Tan, Prashanth Krishnamurthy, Farshad Khor- rami, Ramesh Karri, Brendan Dolan-Gavitt, and Siddharth Garg. Nnoculation: broad spectrum and targeted treatment of backdoored dnns. arXiv preprint arXiv:2002.0833 2020.
[55] Bao Gia Doan, Ehsan Abbasnejad, and Damith C Ranasinghe. Februus: Input purification defense against trojan attacks on deep neural network systems. In Annual Computer Security Applications Conference, pages 897-912, 2020.
[56] Aniruddaa Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11957-11965, 2020.
[57] Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In 30th USENIX Security Symposium (USENIX Security 2/), pages 1505-1521, 2021.
[58] Kunzhe Huang. Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. International Conference on Learning Representations, 2022.
[59] Jiawang Bai, Baoyuan Wu, Yong Zhang, Yiming Li, Zhifeng Li, and Shu-Tao Xia. Targeted at- tack against deep neural networks via flipping limited weight bits. In International Conference on Learning Representations, 2021.
[60] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann ma- chines. In lcml, 2010.
[61] Andrew L Maas, Awni Y Hanmun, and Andrew Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, page 3. Citeseer, 2013.
[62] Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: 2015.
[63] Tanhshrink.
https://pytorch. org/ docs/ httoo////tabbeegggeereee////ttoo nn. Tanhshrin
k. html.
[64] Hao Zheng, Zhanlei Yang, Wenju Liu, Jizhong Liang, and Yanpeng Li. Improving deep neural networks using softplus units. In 2015 International Joint Conference on Neural Networks (ICNN), pages 1-4. IEEE, 2015.
[65] Nurislam Tursynbek, Aleksandr Petiushko, and Ivan Oseledets. Robustness threats of differ- ential privacy. urXiv preprint arXivv2012.07828, 2020.
[66] Seyed-Mohsen Moosavi-Dezfoolli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9078-9086, 2019.
[67] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Roboties Research, 32(11):1231-1237, 2013.
[68] Tuan Anh Nguyen and Anh Tuan Tran. Wapet-imper warping-based backdoor attack. In International Conference on Learning Representations, 2021.
[69] Mauro Barni, Kassem Kallas, and Benedetta Tondi. new backdoor attack in enns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pages 101-105. IEEE, 2019.
[70] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
14

---

[71] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 16463-16472, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? (Yss]
(b) Did you describe the limitations of your work? [Yes] See $ 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See $ 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] See $ 6.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See 3.1.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] See Abstract,
$ 5, and Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See $ 5 and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See 5.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See $ 5.
(b) Did you mention the license of the assets? [Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] URL for our code included Abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you'te re using/curating? [ess] See Appendix.
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [Yes] See Appendix.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "fIueOoWbngb", "reviewer": "Reviewer_75rW", "summary": "This paper demonstrated the backdoor-related neurons in the DNN models form a hyperplane surface as the decision region to the backdoor target label. Based on this understanding, the authors proposed a robust training algorithm,  NONE (NON-LinEarity), that identifies linear decision regions, filters out inputs that are potentially poisoned, and resets affected neurons to enforce non-linear decision regions during training. \n", "questions": "I have a few concerns regarding the proposed method and experiments：\n\n- The finding that shortcuts or linearly-separable characteristics for backdoor attacks are relatively well-known. The same observation has also been done in works (https://arxiv.org/abs/2106.07214) and (https://arxiv.org/pdf/2111.00898.pdf) for backdoor attacks, so it may be worth clarifying that this finding is not completely novel.\n\n- The authors claim that the hyperplane surface formed in backdoored model is because of the piece-wise linear function (i.e. Relu) and there exists one and only one hyperplane in the input space that corresponds to all poisoning samples. What evidence could support your argument and why the reason there is only exist one hyperplane? Whether such linear separability can be induced by the kind of model architecture, input dimensionality, and other activation functions?\n\n- The authors have shown that the traditional backdoor attack methods such as BadNets attack, produce linear separability against a deep neural network. Nevertheless, It may be worth investigating the possibility of crafting “adaptive” attacks [1,2,3] that confuse both clean and backdoor features together when considering separating hyperplanes. In other words, whether the hyperplane surface assumption still held under such cases?\n\n- For the backdoor remove training, why reset the neuron during training? Instead of resetting after training?\n\n- The authors should add a comparison of their computational complexity.\n\n \n[1] Shafahi A, Huang W R, Najibi M, et al. Poison frogs! targeted clean-label poisoning attacks on neural networks. NeurIPS, 2018.  \n[2] Nguyen T A, Tran A. Input-aware dynamic backdoor attack. NeurIPS 2021.  \n[3] Doan K, et al. Backdoor attack with imperceptible input and latent modification. NeurIPS 2021.  \n", "limitations": "The linearly-separable assumption as an underlying explanation for the effectiveness of backdoor attacks is interesting. However, this work in its current form still lacks sufficient analysis to support this conclusion. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to understand.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WCbrOirHobZ", "reviewer": "Reviewer_bmQS", "summary": "This paper proves that a backdoored DNN learns a hyperplane as the decision region, under some assumptions. The authors argue that the backdoored DNNs will increase linearity by introducing a large percentage of neurons activating on one piece of the activation function. Based on these understandings, the authors propose a new training-stage defense (i.e., NONE) that detects and removes potential poisoned samples and repairs compromised neurons. The proposed defense is evaluated on MNIST, GTSRB, CIFAR-10, ImageNet-10, and TrojAI datasets.", "questions": "Cons\n1. Some statements need more details and justifications. \n- 'An adaptive attack with slow poisoning can bypass such defenses' (Line 5-6, Page 1): What the ‘slow poisoning’ means? It seems that there is no support (references or experiments) for it.\n- ‘These methods fail to defend against the natural Trojan. (Line 38-39, Page 2)’: No experiments about whether ABL cannot remove natural Trojan (I only see the results of DPSGD). \n2. Missing assumptions in Theorem 3.3. I think Theorem 3.3 holds if and only if the Trojan is complete and accurate. \n3. Please provide more details about two statements in the appendix (Line 591-593, Page 14).\n4. Missing an important related work and baseline. It seems that [1] is the SOTA training-stage backdoor defense, which is not mentioned and discussed in this paper.\n5. The organization of Section 4 is very poor. There is a very long paragraph in this section, which significantly increases the reading difficulty. Besides, many important details are missing. For example, how to do the separation (Line 13, Algorithm 1)? What are the technical details of Fisher’s linear discriminant analysis and Jenks natural breaks optimization algorithm?\n6. It seems that hidden Trojan backdoor attack is a training-stage attack, which is not effective if training from the scratch (as suggested in [2]). How did the authors evaluate defenses under this attack?\n7. Why did the authors compare their defense only with DPSGD under natural Trojans? I understand that DPSGD seems to be effective in defending against natural Trojans. However, this is not the reason for not comparing with other defenses.\n8. I think it is necessary to go deeper into the internal mechanism of the proposed method. Specifically, according to Algorithm 1, the proposed method can detect poisoned or biased samples. Please provide the precision and recall of this stage so that the readers can understand how well this stage is.\n\nMinor Comments\n1. One typo in Line 91 (Page 2): ‘We’ should be ‘we’.\n2. The ‘R’ should be ‘R^l’ in Line 143 (Page 4)?\n3. Please recheck and cite the official version of all references (e.g., [3-6]).\n\n\nReferences\n1. Backdoor Defense via Decoupling the Training Process. ICLR, 2022.\n2. Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch. arXiv, 2021.\n3. Poisoning and backdooring contrastive learning. ICLR, 2022.\n4. BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements. ACSAC, 2021.\n5. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. IEEE Access, 2019.\n6. BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning. IJCAI, 2021.", "limitations": "Limited but sufficient.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros \n1. The topic is of sufficient significance and interest to NeurIPS audiences. \n2. The codes are provided, which should be encouraged.\n3. The authors try to provide theoretical support for their method, which should be encouraged.\n4. The authors discuss the resistance to potential adaptive attacks, which should be encouraged.\n5. In general, the idea is easy to follow, although many important details are missing.\n\n\n\nCons\n1. Some statements need more details and justifications. \n- 'An adaptive attack with slow poisoning can bypass such defenses' (Line 5-6, Page 1): What the ‘slow poisoning’ means? It seems that there is no support (references or experiments) for it.\n- ‘These methods fail to defend against the natural Trojan. (Line 38-39, Page 2)’: No experiments about whether ABL cannot remove natural Trojan (I only see the results of DPSGD). \n2. Missing assumptions in Theorem 3.3. I think Theorem 3.3 holds if and only if the Trojan is complete and accurate. \n3. Please provide more details about two statements in the appendix (Line 591-593, Page 14).\n4. Missing an important related work and baseline. It seems that [1] is the SOTA training-stage backdoor defense, which is not mentioned and discussed in this paper.\n5. The organization of Section 4 is very poor. There is a very long paragraph in this section, which significantly increases the reading difficulty. Besides, many important details are missing. For example, how to do the separation (Line 13, Algorithm 1)? What are the technical details of Fisher’s linear discriminant analysis and Jenks natural breaks optimization algorithm?\n6. It seems that hidden Trojan backdoor attack is a training-stage attack, which is not effective if training from the scratch (as suggested in [2]). How did the authors evaluate defenses under this attack?\n7. Why did the authors compare their defense only with DPSGD under natural Trojans? I understand that DPSGD seems to be effective in defending against natural Trojans. However, this is not the reason for not comparing with other defenses.\n8. I think it is necessary to go deeper into the internal mechanism of the proposed method. Specifically, according to Algorithm 1, the proposed method can detect poisoned or biased samples. Please provide the precision and recall of this stage so that the readers can understand how well this stage is.\n\nI will increase my scores if the authors can (partly) address my concerns.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Tzb7m2o6aqt", "reviewer": "Reviewer_obsx", "summary": "The paper proposes a backdoor defense based on the observation that trojan and benign neurons exhibit different behavior in the victim model. Their defense assumes an attacker who can poison a subset of the training data and its labels. They design a training algorithm that detects and suppresses neurons that form shortcuts between classes, which they describe as \"linear decision boundaries\". Empirical results are provided for image classification datasets, showing that their training algorithm NONE has considerable improvements over existing defenses against three poisoning-based backdoor attacks. Finally, the authors show an ablation study over different trigger sizes and poisoning rates which show that NONE is robust in most cases. ", "questions": "What is the runtime of NONE? How does it scale to larger models or datasets such as ImageNet? \n\nHow many inputs are removed by NONE on datasets such as CIFAR-10? What is the precision and recall (with regards to malicious and clean inputs) of this removal process? \n\nHave you looked into the performance of NONE against backdoor attacks that can poison the training code? ", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "** Strengths\n\n* Novelty. Defenses against poisoning-based backdoor attacks are highly relevant. Detecting and suppressing malicious neurons during training is a novel idea that shows promising results. \n\n* High effectiveness. It appears that NONE is a promising defense that clearly outperforms many existing defenses. The drop in clean accuracy is also often low, (<=3%) making it a practically relevant defense. \n\n* Ablation studies. The defense was evaluated on multiple datasets against three attacks showing promising results everywhere. The authors also show ablation studies over different trigger sizes and poisoning rates. \n\n* Open-source code. \n\n** Weaknesses\n\nOverall, the paper was an interesting read and I found only minor weaknesses in the paper's presentation. The results and intuition are convincing and I believe they are of interest to the research community. \n\n* Formalization. Many symbols in Algorithm 1 are only defined informally in the text, but it would improve clarity if they were formalized (e.g., the separate, norm, and init function). A table explaining the symbols would help readability a lot. \n\n* Minor. Please break a single, long paragraph into multiple smaller paragraphs to enhance readability. Some citations are missing (e.g., for all datasets such as TrojAI [1]) \n\n[1] Karra, Kiran, Chace Ashcraft, and Neil Fendley. \"The trojai software framework: An opensource tool for embedding trojans into deep learning models.\" arXiv preprint arXiv:2003.07233 (2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept