Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Features
Authors: Mingli Zhu
Shaokui Wei
Hongyuan Zha
Baoyuan Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: Backdoor Defense
Backdoor Learning
Trustworthy AI

Abstract: Recent studies have demonstrated the susceptibility of deep neural networks to backdoor attacks. Given a backdoored model, its prediction of a poisoned sample with trigger will be dominated by the trigger information, though trigger information and benign information coexist. Inspired by the mechanism of the optical polarizer that a polarizer could pass light waves with particular polarizations while filtering light waves with other polarizations, we propose a novel backdoor defense method by inserting a learnable neural polarizer into the backdoored model as an intermediate layer, in order to purify the poisoned sample via filtering trigger information while maintaining benign information. The neural polarizer is instantiated as one lightweight linear transformation layer, which is learned through solving a well designed bi-level optimization problem, based on a limited clean dataset. Compared to other fine-tuning-based defense methods which often adjust all parameters of the backdoored model, the proposed method only needs to learn one additional layer, such that it is more efficient and requires less clean data. Extensive experiments demonstrate the effectiveness and efficiency of our method in removing backdoors across various neural network architectures and datasets, especially in the case of very limited clean data. Codes are available at \href{https://github.com/SCLBD/BackdoorBench}{https://github.com/SCLBD/BackdoorBench} (PyTorch) and \href{https://github.com/JulieCarlon/NPD-MindSpore}{https://github.com/JulieCarlon/NPD-MindSpore} (MindSpore).

Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Features
Mingli Zhu Shaokui Wei!" Hongyuan Zha Baoyuan Wult
'School of Data Science, The Chinese University of Hong Kong, Shenzhen -hennhhen,,hen, China "Shenzhen Institute Artificial Intelligence and Robotics for Society, China
Abstract
Recent studies have demonstrated the susceptibility of deep neural networks to backdoor attacks. Given a backdoored model, its prediction poisoned sam- ple with trigger will be dominated by the trigger information, though trigger information and benign information coexist. Inspired by the mechanism the optical polarizer that a polarizer could pass light waves with particular polariza- tions while filtering light waves with other polarizations, we propose a novel backdoor defense method by inserting learnable neural polarizer into the back- doored model as an intermediate layer, in order to purify the poisoned sample via filtering trigger information while maintaining benign information. The ral polarizer is instantiated as one lightweight linear transformation layer, which is learned through solving well designed bi-level optimization problem, based defense meth-
on a limited clean dataset. Compared to other fine-tuning- ods which often adjust all parameters of the backdoored model, the proposed method only needs to learn one additional layer, such that it is more efficient and requires less clean data, Extensive experiments demonstrate the effectiveness and efficiency of our method in removing backdoors across various neural net- work architectures and datasets, especially in the case of very limited clean data. Codes are available at https//ggitub.coom////ccaaaoo/// MindSpere (MindSpore).
(Py (PyTorch) and
Introduction
Several studies have revealed the vulnerabilities of deep neural networks (DNNs) to various types of attacks [15, 17, 21, 22, 40], of which backdoor attacks [6. 9, 11, 36] are attracting increasing attention. In backdoor attacks, the adversary could produce backdoored DNN model through manipulating the training dataset [5, 24] or the training process [20, 29], such that the backdoored model predicts any poisoned sample with particular triggers to the predetermined target label, while behaves normally on benign samples. Backdoor attacks can arise from various sources, such as training based on a poisoned dataset, utilizing third-party platforms for model training, downloading backdoored models from untrusted third-party providers. These scenarios significantly elevate the threat of backdoor attacks DNNs applications, and meanwhile highlight the importance of defending against backdoor attacks.
Several seminal backdoor defense methods have been developed, mainly including 1) in-training approaches, which aim to train a secure model based on a poisoned dataset through well-designed training algorithms or objective functions. such as DBD [14] and D-ST [4]. 2) post-training ap- proaches, which aim to mitigate the backdoor effect from a backdoored model through adjusting the
"Equal contribution I Conresponds to Baoyuan Wu cwt.yyuuannuuuuucu..........
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Backdoored Model (fixed)
Benign
Cat
Poisoned
Cat
Shared
Shared
Poisoned Benign
Cat
Dog
Optical Polarizer
Neural Polarizer (learnable)
Figure I: Left: Illustration of optical polarizer [41]. Only light waves with specific polarizations can pass through the polarizer among the three incident light waves. Right: Defense against backdoors by integrating trainable neural polarizer into compromised model. The neural polarizer effectively filters out backdoor-relant features, effectively eliminating the backdoor.
model parameters (e.g., tinettuning or pruning), usually based on limited subset clean training dataset, such fine-pruning [23]. ANP [43], or i-BAU [45]. This work focuses on the latter one. However, there are two limitations to existing post-training approaches. First, given very limited clean data, it is challenging to find good checkpoint to simultaneously achieve backdoor mitigation and benign accuracy maintenance from the gigh-dimensional loss landscape of a complex model. Second, adjusting all parameters of complex model is costly.
To tackle the above limitations, we propose a lightweight and effective post-training defense approach, which only learns one additional layer, while fixing all layers of the original backdoored model. It inspired by the mechanism of the optical polarizer [44] that in a mixed light wave with diverse polarizations, only the light wave with some particular polarizations could pass the polarizer, while those with other polarizations are blocked (see Fig. 1-left). Correspondingly, by treating one poisoned sample the mixture of trigger feature and benign feature, we define neural polarizer and insert it into the backdoored model as one additional intermediate layer (see Fig. 1-right) in order to filter trigger feature and maintain benign feature, such that poisoned samples could be purified to mitigate backdoor effect, while benign samples are not significantly influenced.
In practice, to achieve an effective neural polarizer, it should be learned to weaken the correlation between the trigger and the target label while keeping the mapping from benign samples to their ground-truth labels. However, the defender only has a limited clean dataset, while neither trigger nor target label is accessible. To tackle it, we propose bi-level optimization, where sample- specific target label is dynamically estimated according to the output confidence, and the trigger approximated by the targeted adversarial perturbation. Besides, in our experiments, the neural polarizer is implemented by linear transformation (i.e., the combination of one 1 x convolutional layer and one batch normalization layer). Consequently, can be efficiently and effectively learned with very limited clean data achieve good defense performance, which is verified by extensive experiments on various model architectures and datasets.
Our main contributions are folee-fold. 1) We propose an innovative backdoor defense approach that only learns one additional linear transformation called neural polarizer while all parameters of the backdoored model are fixed, such that it just requires very low computational cost and very limited clean data. 2) A bi-level optimization problem and an effective learning algorithm are provided to optimize the parameter of the neural polarizer. 3) Extensive experimental results demonstrate the superiority of the proposed method on various networks and datasets, terms of both efficiency and effectiveness.

---

2
Related work
Backdoor attacks. Traditional backdoor attacks are additive attacks that modify a small fraction of training samples by patching a pre-defined pattern and assigning them to targeted labels [8, 11]. These modified samples, along with the unaffected samples, constitute poisoned datasel [11]. The model trained with this dataset will be implanted with backdoors that predict the targeted label when triggered by the injected patterns while maintaining normal behavior clean samples [5, 50]. Recently, advanced attacks have considered more invisible trigger injection methods such as training an auto-encoder feature embedding using local transformation function [20, 29, 46]. To increase the stealthiness of attacks, clean-label attacks succeed by obfuscating image subject information and establishing correlation between triggers and targeted labels without modifying the labels of poisoned samples [1, 2, 30].
Backdoor defense. Backdoor defense methods can be broadly categorized into training-stage defenses and post-processing defenses. Training-stage defenses assume access to a poisoned dataset for model training [10]. The different behaviors between the poisoned and clean samples can be leveraged to identify suspicious instances, such as sensitivity to transformation [4] and clustering phenomenon in feature space [14]. Most defense methods belong processnnn defenses [39], which assume that the defender only has access to suspicious DNN model and few clean samples. Therefore, they must remove the backdoor threat with limited resources. There are mainly three types of defense strategies: trigger reversion methods try to recover the most possible triggers and utilize the potential triggers fine-tune the model [35]; pruning based methods aim locating
the neurons that are most related to backdoors and pruning them [23, 43, 47, 48]; and fine-tuning
based defenses leverage clean samples to rectify the model [19, 45]. I-BAU [45] is most closely related to our method, which formulates minimax optimization framework to train the network with samples under universal adversarial perturbations. However, our method differs from I-BAU in that our approach does not require training the entire network, and perturbation generation mechanism is distinct. Other methods proposed for backdoor detection include Beatrix [26], which uses Gram matrices to identify poisoned samples; and AEVA [12], which detects backdoor models by adversarial extreme value analysis. In this study, we focus on post-prose post-processing defenses and primarily compare method with state-of-the-aa post-processing defenses [19, 23, 35].
3 Methodology
3.1 Basic settings
Notations. We consider a classification problem with K classes (K 2). Let x X C Rd be d-dimensional input sample, and its ground truth label is denoted as 3 = =(1..., "l....,K.. Then, a L layers deep neural network f : X x W - K parameterized by w E W defined as:
f(:; w) = (LL) o p(L-1) ....o s( (x))
where [w) is the function (e.g., convolution layer) with parameter wy in the 1 layer with S L. For simplicity, we denote (a;; w) as fw (a:) or fw- Given input æ. the predicted label of given by arg max; fü (x; w), k = 1, I,..., K, where fa( w) is the logit of the 11th class.
Threat model. We assume that the adversary could produce the backdoored model fwr through manipulating training data or training process, such that fw performs well on benign sample æ (i.e., fw (a) = y), and predicts poisoned sample V = r(æ, A) to the target label T, with A indicating the trigger and r(:.) being the fusion function of æ and A. Considering that the adversary may set multiple targets, we use T, to denote the target label for æ;,
Defender's goal. Assume that the defender has access to the backdoored model fur and limited set of benign training data, denoted as Dim = {(.,, J The defender's goal is to obtain a new model f based on fwr and Den, such that the backdoor effect will be mitigated and the benign performance is maintained in f.
3

---

3.2 Neural polarizer for DNNs
Neural polarizer. We propose the neural polarizer to purify poisoned sample in the feature space. Formally, it is instantiated as a lightweight linear transformation !e. parameterized with 0. As shown in Fig. 1, ge is inserted into the neural network fw at a specific immediate layer, to obtain a combined network fw.e- For clarity, we denote f as fo. since w is fixed.
desired neural polarizer should have the following three properties:
Previous Layer f(kk)
Compatible with the neighboring layers: in other words, its input feature and its output activation must have the same shape. This requirement can be fulfilled through careful architectural design.
Ck HN x Wh.
Ixl Conv, Ck
Neural Polarizer
Ck
H x WR
Batch Normalization
Filtering trigger features in poisoned samples: af- ter the neural polarizer, the trigger features should be filtered, such that the backdoor is deactivated, i.e., fo(#s)) # T.
C
H x WR
(1++1)
Posterior Layer, m1k4
Preserving benign features in poisoned and benign samples: the neural polarizer should preserve benign features, such that fe performs well on both poisoned and benign samples, i.e., fo (x = fo(x)= y.
Figure example of neural polarizer for DNN.
The first property could be easily satisfied by designing neural polarizer's architecture. For example, as illustrated in Fig. 2. given the input feature map with the shape C& x H& x WR. the neural polarizer implemented by convolution layer (Conv) with Cs convolution filters of shape 1 followed by a Batch Normalization (BN) layer. The Cony-BN block can be seen as linear transformation layer. To satisfy the latter two properties, should be learned by solving some well designed optimization, of which the details are presented in Section 3.3.
3.3 Learning neural polarizer
3.3.1 Loss functions in oracle setting
To learn good neural polarizer go. we first consider an oracle setting where the trigger A and target label are given. We present some loss functions to encourage Jo to satisfy the latter two properties mentioned above, follows:
Loss for filtering trigger features in poisoned samples. Given trigger A and target label T, filtering trigger features can be implemented by weakening the connection between A and T with the following loss:
Lasr (xr y. A, 0) log(1 sT(=s;
where 87 (* 0) is the softmax probability of predicting æa to label T by fo. By reducing Casrs the attack success rate of the backdoor attack can be decreased.
Loss for maintaining benign features in poisoned samples. To purify the poisoned sample such that it can be classified the true label, we leverage the boosted cross entropy defined in [37]:
Chee(xe y. A) (x loo (1
1--axxa (m kyy
Loss for maintaining benign features in benign samples. To preserve benign features in benign sample, we adopt the widely used cross-entropy loss, i.e., Lon y: 0) (os(fo(x)))))
3.3.2 Approximation & optimization
Approximating 7 and A. Note that as 7 and are inaccessible in both Lasr and Lices these two losses cannot be directly optimized. Thus, we have to approximate T and A. In terms of

---

approximating T, although some methods have been developed to detect target class of backdoored models [12, 26]. here we adopt a simple sample-specific and dynamic strategy:
Tx n' = arg max f (2;; 0).
kty
There are two main advantages of this strategy. First, the predicted target label enables us to generate targeted adversarial perturbation. As analyzed later, the targeted adversarial perturbation is better surrogate for the unknown trigger than the untargeted adversarial perturbation. Second, the sample- specific target label prediction is applicable to both all2one (one trigger, one target class) and all2all (every class is target class) attack settings. In practice, defenders lack certainty about attack types (all2one, all2all, or multi- multi-trigger multi-target), risking sub-optimal defenses due to erroncous guesses or detection. Our sample-specine target label prediction avoids this risk.
In terms of approximating A, it is approximated by the targeted adversarial perturbation of fw. iee,
R 8
arg min LCE (fo(z + 0))
(5)
So
where || |lp is the Lp norm, is the budget for perturbations.
Bi-level optimization problem. By employing the approximations T' and shown in Eq. (4)
and (5), respectively, and substituting these approximations into the respective loss functions of Eq.
(2) and (3), we introduce the following optimization problem to learn based on the clean data Don:
N Ao (o y6: 0) + XeCuer Vi, 67, 07.6:00 + A3 (at, Ui. 07;
min
N
(6)
8 = arg min Las (fo(2;+
0.). vt) vo = arg max fa. (a;; 0), = I....,., N, k.
s.t.
where Åι A2, 03 > 0 are
to adjust the importance of each loss function. This bi-level optimization is dubbed Neural Polarizer based backdoor Defense (NPD).
Variants. To comprehensively evaluate the performance NPD, we also provide two variants, with the relaxation that if the target label 7' is known. One is that approximating the trigger by the targeted adversarial perturbation for each benign sample in Eq. (6), i.e., A 8; arg m Sp LCE (f 8.), dubbed NPD-TP. The other is approximating the trigger by the targeted universal adversarial perturbation (TUAP) [25] for all training samples in fine-tuning, dubbed NPD-TU. We empirically found that the targeted adversarial perturbation is better surrogate to the trigger than the untargeted ones.
Optimization algorithm. We proposed Algorithm 1 to solve the above optimization problem. Specifically, NPD solves problem (6) by alternatively updating the surrogate target label y/, the perturbation 8 and 0 as follows:
Inner minimization: Given parameter @ of the neural polarizer, we first estimate the target label for sample 2;; by yf arg max fr. (2:;;) 0). Then, the targeted Project Gradient Descent (PGD) [27] is employed to generate the perturbation 5; via Eq. (5).
Outer minimization: Given y/ and 8* for each sample in a batch, the can be updated by taking one stochastic gradient descent [3] (SGD) step w.t.. the outer minimization objective in Eq. (6).
4 Experiments
4.1
Implementation details
Attack settings. We evaluate the proposed method on eight famous backdoor attacks, including BadNets [11] (BadNets-A20 and BadNets-A2A refer to attacking one target label and all labels, respectively), Blended attack (Blended) [5], Input-aware dynamic backdoor attack aJnput-aware)[28)) Low frequency attack (LF) [46], Samplc-specific backdoor attack (SSBA) [20], Trojan backdoor
5

---

Algorithm Neural Polarizer based Backdoor Defense (NPD)
1: Input: Training set Din. backdoored model fw. neural polarizer ge. learning rate v 0,
perturbation bound > O, norm p. hyper-parameten A2,2 > O, warm-up epochs To, training epochs T, number of PGD steps Tadu-
2: Output: Model ((n, 0).
3: Initialize A to be an identity function, fix w, and construct the composed network f(w,0)
4: Warm-up: Train f(w. 0) with LCE (bm) for To epochs.
5: for = 0, T do
6: for mini- batch B = {a y: y:) C DDn do
7:
Compute {ı^}, by Eq. (4);
8:
Generate perturbations {(6+)] with ||ó:|l S p and (v/ by targeted PGD attack [27]
via Eq. (5);
9: Update via outer minimization of Eq. (6) by SGD.
10: end for
11: end for
12: return Model f(w, 0).
attack (Trojan) [24], and Warping-based poisoned networks (WaNet) [29]. We follow the default attack configuration as in BackdoorBench [42] for a fair comparison. The poisoning ratio set to 10% in comparison with SOTA defenses. These attacks are conducted on three benchmark datasets: CIFAR-IO [16], Tiny ImageNet [18], and GTSRB [34]. We test all attacks Nestct-ResNet188 [13] and VGG19-BN [33].
Defense settings. We compare the proposed methods with six SOTA backdoor defense methods, i.e., Fine-pruning (FP) [23]. NAD [19], NC [35], ANP [43], i-BAU [45], and EP [48]. All these defenses have access 5% benign training samples. The training hyperparameters are adjusted based on BackdoorBench [42]. We evaluate the proposed method under two proposed scenarios and compare our NPD-TU, NPD-TP, and NPD with SOTA defenses. For the ablation study, we focus solely on NPD, which represents a more generalized scenario. We apply an l norm constraint to the adversarial perturbations, with a perturbation bound of for CIFAR-10 and GTSRB datasets, and for Tiny ImageNet. We train the neural polarizer for 50 epochs with batch size 128 and learning rate 0.01 on each dataset and the transformation block inserted before the third convolution layer the fourth layer for TreAct-ResNetl and 1, 0.1, 0.1 for NPD-TU and NPD-TP. More implementation details on SOTA attacks, defenses, and our methods can be found in Section B of Appendix.
The loss hyper-parameters A2. are set to 1, 0.4, 0.4 for NPD,
Evaluation metric. In this work, we use clean ACCuracy (ACC), Attack Success Rate (ASR), and Defense Effectiveness Rating (DER) as evaluation metrics to assess the performance of different defenses. ACC represents the accuracy clean samples while ASR measures the ratio of successfully misclassified backdoor samples to the target label. Defense Effectiveness Rating (DER E [0, 1] [49]) a comprehensive measure that considers both ACC and ASR:
DER [max(0, AASR) - max(0, AACC) + 1)/2.
where AASR denotes the decrease of ASR after applying defense, and AACC denotes the drop in ACC following defense. Higher ACC, lower ASR, and higher DER indicate better defense performance. Note that in comparison with SOTA defenses, the one achieving the best performance is highlighted in boldface, while the second-best result is indicated by underlining. We provide PyTorch and MindSpore implementations of NPD.
4.2
Main results
Table I and Table 2 showcase the defense performance of the proposed method in comparison to six SOTA defense methods on CIFAR-10 and Tiny ImageNet. The following observations can be made:
Our methods show superior performance in terms of DER for almost all attacks compared to SOTA defenses. Conversely, FP and ANP excel in maintaining high ACC, but they struggle to climinate backdoors in strong attacks like Blended and LF. NC's emphasis minimal universal

---

Table 1: Comparison with the state-of-the-arr defenses on CIFAR-10 10 dataset with 5% benign data and 10% poison ratio on PreAct-ResNet18 (%).
ATTACK
Backdoored
HP [23]
NAD [19]
[CS] ASR
ANP [43]
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
DER
ACC
ASR
DER
BadNett-A20 [1]]
91.82
93.79
NiA
91.77
0.84
96.45
88.82
1.96
94.42
90.27
1.62
95.31
91.65
3.83
94.89
BadNets-A2A [1]]
91.89
74.42
N/A
92.05
1.31
86.56
90.73
1.61
85.82
89.79
1,11
85.60
92.33
2.56
85.93
Blended [5]
93.69
99.76
NiA
92.74
10.17
94.32
92.25
47.64
75.34
93.69
99.76
50.00
93.45
47.14
76.19
Imput-Aware [25]
94.03
98.35
N/A
94.05
1.62
98.36
94,08
0.92
96.71
93.84
10.48
93.84
94.06
1.57
98.39
LF [F64
93.01
99.06
NIA
92.05
21.32
88.39
91.72
75.47
61.15
93.01
99.06
50.00
92.53
26.38
86.10
SSBA [20]
92.88
97.17
N/A
92.21
20.27
88.00
92.15
62.78
92.88
97.07
50.00
92.02
16.18
90.01
Trojan [24]
93.47
99.99
NIA
92.24
67.73
65.51
92.18
5.77
96.47
91.85
51.03
73.67
92.71
84.82
57.20
WoNet (29]
98.00
N/A
92.94
0.66
99.12
91.07
93.24
1.54
Avg
92.95
95.17
N/A
92.51
15.49
89.62
91.88
25.61
84.24
92.27
$7.38
68.55
92.75
23.00
85.98
AFTACK
i-BAU [15]
EP [18]
NPD-TU(Ours)
NPD-TP(Ours)
NPD(Ours)
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
RadNets-A2011
87.43
4.48
92.46
89.80
1.26
95.26
90.81
90..0
0.62
96.12
88.93
1.26
94.82
RadNets-A2A [11]
89.39
1.29
85.32
88.72
3.00
84.12
91.66
0.82
86.68
92.54
0.04
87.19
91.41
0.89
86.52
Blended [S]
26.82
84.34
91.
48.22
74.89
0.03
98.96
91.33
0.83
0.41
Input-Aware [28]
89.91
0.02
97.10
93.68
2.88
97.56
92.01
0.14
98.09
93.24
0.00
98.78
89.57
0.11
96.89
LP [46]
11:99
91.97
56.64
91.92
0.08
90.06
0.21
SSBA [20]
86.53
2.89
93.91
91.67
4.33
95.76
91.61
2.16
96.67
91.82
0.83
97.59
90.88
2.77
96.15
Trojan [24]
0.54
97.63
92.32
2.49
18
92.59
99.53
0.00
99.35
92.37
6.51
96.19
WaNet
90.70
0.88
97.96
90:47
96.52
50.03
92.18
3.24
97.52
92.57
7.47
95.60
91.57
0.80
98.43
Avg
88.95
6.11
92.53
91.32
30.43
81.55
91.77
96.48
92.06
1.23
90.75
1.62
95.67
Table 2: Comparison with the state-of-the-art defenses on Tiny ImageNet dataset with 5% benign (%).
data and 10% poison ratio
ATTACK
Backdoored ASR
IP [23]
NAD [19]
NC [35]
ANP [43]
ACC
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
RadNets-A20 [11]
56.12
99.90
N/A
48.81
0.66
95.96
48 35
0.27
95.93
56.12
99.90
50.00
47.34
0.00
95.56
BadNets-A2A [1]]
55.99
27.81
N/A
47.88
3.19
58.26
48 29
2.30
58.91
$4.12
18.72
53.61
40,70
2.39
$5.07
Blended [5]
56.49
99.67
$7.89
67.93
55.22
49.76
51.50
96.07
43.21
43,80
71.29
Input-Aware [28]
57.67
99.19
N/A
52.38
0.13
96.88
57.42
0.07
99.43
53.46
2.48
96.25
50.56
0.00
96.04
[F6]
55.21
98.51
63.83
49.61
53.08
41.75
65.98
59.51
SSBA [20]
55.97
97.69
N/A
48.06
52.25
68.76
47.67
69.47
59.96
$3.30
0.26
97.38
41.83
14.24
84.65
Trojan [24]
56.48
99.97
8.88
90.28
95.65
51.13
98.19
15.36
0.53
94.16
WaNet [29]
57.81
96.50
N/A
50.35
1.37
93.83
50.02
0.87
93.92
$7.81
96.50
50.00
30.34
0.00
84.51
Avg
56.47
89.90
N/A
49.02
23.53
79.47
50.68
28.86
77.63
54.60
50.74
68.65
42.64
15.87
80.10
ATTACK
[-BAU [15]
EP [18]]
NPD-TU(Ours)
NPD-TP(OurS)
NPDXOurs)
ACC
ASR
ASR
DER
ACC
ACC
ASR
HadNets-A20 [11]
51.63
95.92
49.74
14.00)
0.02
98.88
47.23
01.01
95.50
49.89
1.28
96.19
49.79
2.51
95.53
BadNets-A2A [11]
53.52
12.89
56.22
1.28
62.67
46.81
58.34
49.79
3.31
59.15
49.94
5.57
58.10
Blendedd [S]
50.76
95.58
49.18
56.32
88.88
$1.31
46.24
(1.00
94,71
49.72
0.18
96.36
49.62
0.12
96.34
Input-Aware [28]
55.49
0.46
98.27
$7.33
0.03
99.41
49.54
0.27
95.39
53.88
0.04
97.65
53.75
5.93
94.67
L. [P64
$3.65
94.27
51.34
54.86
93.20
52.48
46.04
01000
94.67
49.20
0.30
96.10
49.94
2.48
95.38
SSBA [20]
52.39
84.64
54.73
55.56
66.67
65.30
46.56
0.00
94.14
-9.04
0.00
95.38
49.25
0.01
95.48
Trojan [24]
51.85
99.15
48.09
54.47
0.12
98.92
48.55
0.00
96.02
49.61
0.05
96.52
49.43
0.51
96.20
WaNet [29]
53.04
69.82
60.95
57.06
0.20
97.77
48.52
0.01
93.60
51.88
0.82
94.88
52.64
0.24
95.54
Avg
52.79
69.09
58.57
55:55
31,30
78.84
47.44
0.28
90.30
50.38
0.75
91.53
50.54
2.17
90.90
adversarial perturbation renders it ineffective against sample-specific attacks and those utilizing large norm triggers. I-BAU shows similar performance in removing backdoors with an average DER of 92.53%, but it leads to significant decrease in ACC, likely due to training the entire network by adversarial training.
Defense performance of NPD-TU and NPD-TP are better than NPD. When the target label is known, the model only needs to find perturbations for that specific label, simplifying trigger identification and unlearning. These two methods outperform NPD, except for WaNet, which is mansformation-baased attack without visible triggers. Fully perturbing the network proves more effective than solely unlearning targeted triggers in WaNet.
NPD-TU is effective for trigger-additive attacks while NPD-TP is expert in defending against sample-specific attacks. It can be observed by comparing defense results on different attacks like Blended and SSBA on CIFAR-10. This demonstrates that the applicability of different strategies varies across different attack scenarios.
Defense performance robust across all attacks on Tiny ImageNet. Similar to the results on CIFAR-10 10, our method outperforms other methods in terms of ASR and DER for all backdoor
7

---

Table 3: Defense performance under different Table 4: Defense results in comparison with NPD- UU and NPD-UP.
components of losses.
ATTACK
BadNess-A20
Blended
ATTACK
NPD-UU
NPD-UP
ACC
ACC
ASR
ASR
ASR
91.45
1.18
92.47
99.63
92.00
95,90
BodiNets-A20 [1]]
91.82
93.79
79.35
0.30
90.61
1.74
89.93
1.25
90.17
1.19
91.51
2.01
90.91
9.60
Blanded
93.44
97.71
86.35
10.77
92.35
3.86
91.18
0.41
91.19
75.42
91.31
91.117
SSBA
97.07
52.36
14.22
59.56
0.21
91.09
1.75
90.47
7.63
Tagan [24]
93.47
99.99
89.51
38.11
92.61
11.43
92.37
6.51
$3.93
1.26
91.18
0.41
90,06
0.21
WaNes [29]
92.80
98.90
84.70
5.58
92.11
1.41
91.57
0.80
attacks. Despite slight decrease in ACC, NPD-TU achieves remarkably good performance with ASR < 1.5% average. NPD-TP and NPD perform best in removing backdoors while maintaining model utility.
In summary, method outperforms other state-of-the-art approaches, showcasing the broad appli- cability of our proposed method across diverse datasets. Due to space limits, defending results on GTSRB dataset and VGG19-BN network can be found in Section C of Appendix.
4.3 Analysis
Effectiveness of each loss term. We conduct an ablation study to evaluate the contribution each component of the loss function towards the overall performance on CIFAR- 10 dataset. We separately investigate the first and second terms of loss Lbce (see Eq. (3)), denoting them as Lsee1 and Lbce2, respectively. Throughout the study, we keep the loss Lon consistent across all experiments, and the result is shown in Table 3. Notably, the loss ldee1 plays significant role in improving the overall performance while removing each component leads to a significant drop in defense in certain cases. This ablation study underscores the importance of each loss component in effectively mitigating different types of attacks.
Effectiveness of the targeted adversarial perturbations NPD. To show the efficacy of the targeted adversarial perturbations in NPD (see Eq. (6)), compare NPD with its variants using two types of untargeted perturbation. We refer to adversarial perturbations generated by UAP and standard PGD without targeted label NPD-UU (untargeted universal adversarial perturbation) and NPD-UP (untargeted PGD), respectively. As shown in Table 4, NPD-UU and NPD-UP fail to remove backdoors in certain cases although NPD-UP obtains higher ACC. This result shows the superiority of NPD in removing backdoors.
Performance of choosing different layers to in- sert the transformation layer. We evaluate the influence of choosing different layers to insert the transformation layer by inserting it after each con- volution layer of PreAct-ResNetl
network on CIFAR-10 dataset. Figure shows the defense per- formance under three attacks. The result shows that inserting the transformation layer into the shallower layers results a decrease accuracy. This because even slight perturbation in the shallow layers can cause significant instability in the final output. However, the layer goes deeper, the fea- tures become more separable, resulting in better defense performance.
Percentage
60
BadNets
ACC
Blended
ASR
SSBA
20-
Layers
Figure 3: Defense performance of inserting
linear transformation into different layers.
Defense effectiveness under different poisoning
ratios. To investigate the impact poisoning ratios on defense performance, we conducted exper- iments NPD with different poisoning ratios on the CIFAR-10 dataset. As presented in Table 5, there slight decrease ACC as the poisoning ratio increases. Moreover, our approach exhibits notably stable defense performance across range of poisoning ratios.

---

Table 5: Defense results under different poisoning ratio on CIFAR-10 10 and (reAct-ResNet(8(90)
Poisoning Ratio ATTACK J
5%
10%
20%
30%
40%
No Defense
Ours
No Defence
Ours
No Defense
Ours
No Defense
Ours
No Defense
Ours
BadNets-A20 [11]
ACC
92.35
87.99
91.82
85.77
86.16
ASR
89.52
0.89
93.79
1.26
96.12
0.39
97.33
0.69
97,78
3,88
Blended [5]
ACC
93.76
91.48
93.44
91.18
93.00
91.27
92.78
90.10
91.64
88.85
98.96
ACC
90.92
90.31
94.03
89.57
89.18
88.60
89.63
89.41
90.12
88.57
Input-Aware
ASR
94.19
0.30
98.35
0.11
97.66
3.51
97.62
3.43
98.57
0.98
WaNer [29)
ACC
93.38
92.80
91.57
91.02
89.77
89.53
92.21
ASR
97.27
0.14
98.90
0.80
94.93
0.22
99.06
1.71
99.49
2.19
Table 6: Results with different number of clean data on CIFAR- 10 (%)
# Clean data
500
250
50
ATTACK
Defense 4
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
i-BAU [45]
65.41
79.82
BadNets-A20 [1I]
ANP [43]
91.53
5,50
94.00
90.81
2,03
93.40
85.57
1.52
91.04
Ours
87.58
1.38
94.09
87.80
0.27
92.78
86.91
0.20
92.37
i-AUU [45]
SSRA (20)
ANP [43]
92.06
28.67
83.79
92.13
27.61
80.17
88.31
22.28
80.92
Ours
90:48
0.43
97.12
90.80
10.03
88.29
89.48
8.97
88.16
i-AUU [45]
85.31
LI [46]
ANP [43]
92.74
46.70
76.04
92.28
18.02
84.11
89.99
18.77
82.59
Ours
90.21
0.99
97.63
89.28
0.50
91.37
88.18
10.83
85.65
Defense effectiveness under different clean ratios. We investigate the sensitivity of clean data on defense performance and compare our NPD with SOTA defenses. As shown in Table 6, NPD is less sensitive the size of clean data among all the attacks and defenses. Even with only 50 samples, still maintains acceptable performance. This result shows that our method exhibits minimal reliance on the number of training samples.
Defense against adaptive attacks. We evaluate the effectiveness of NPD against adaptive attacks. Consider the attacker knows the defense strategy and the attacker trains the backdoored model with adversarial training (AT), The defense performance against the AT backdoored model under different attacks is shown in Table 7. NPD still performs well on AT model, while i-BAU fails in these attacks. One possible reason is that i-BAU is essentially adversarial training, while NPD adopts dynamic targeted adversarial perturbation, which is different from AT. Compared to the defense against backdoored models with standard training (Table 1), there is a slight ASR increase (from 2.77% to 5.22% for SSBA, for example).
Defense against all to all attacks. To demonstrate the effectiveness of our dynamic and sample- specific target label prediction strategy, we conducted experiments involving all-to-all attacks for various backdoor attack methods. In these experiments, each class is treated the target class. The result is presented in Table 8. As illustrated in the table, our NPD approach successfully eliminates all backdoors, proving its efficacy in multi-label cases. In contrast, i-BAU method fails in most cases.
Running time comparison. We measure the runtime of the defense methods on 2500 CIFAR-10 - images with batch size 256 and PreAct-ResNeer
The experiments were conducted on one RTX 4090'i GPU and the results are presented Table 9. Among these methods, our proposed PND-UN achieves the fastest performance CIFAR- requiring only 56 seconds. It should be noted that method was trained for 50 epochs, while i-BAU was only trained for 5 epochs.
Table 7: Defense performance against adaptive Table 8: Defense performance against all2all at- tacks on CIFAR-10 dataset (%).
attacks on CIFAR-10 dataset (%).
Backdloredd
i-BAU [45]
NPD
Backdoorred
i-BAU [45]
NPD
ATTACK
ATTACK
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
Bleuded [S]
86,00
99.63
83.98
30.43
83.64
3.92
Blended [S]
91.59
83,50
88 80
15.74
91.07
6.24
Iaput- Aware [28]
84.98
94.99
83.17
71.12
83.13
4.47
Iaput-Aware [28]
86.60
78.55
89.50
5.37
90.10
2.10
LF [46]
84.15
94.30
84.15
94,30
83.39
4.89
LF
84.80
88.92
33.33
90.53
4.88
SSBA [20]
84.34
93.32
82.74
24.54
83.38
5.22
SSBA [20]
91.30
85.04
91.16
26.00
91.14
1.29

---

Table 9: Running time of different defense methods with 2500 CIFAR-10 images on PreActResNet 18.
Defense (sec.)
[23]
NAD [19]
[35]
ANP [43]
i-RAU [45]
[48]
NPD (Ours)
CIFAR-10
1169.01
74.39
896.45
58.75
57.23
131.84
55.16
Tiny ImageNet
3357
351
42512
1692
887
302
332
5 Conclusion
Inspired by the mechanism of optical polarizer, this work proposed a novel backdoor defense method by inserting a learnable neural polarizer as an intermediate layer of the backdoored model. We instantiated the neural polarizer as a lightweight linear transformation and it could be efficiently and effectively learned with limited clean samples mitigate backdoor effect. To learn desired neural polarizer, a bi-level optimization problem is proposed by filtering trigger features of poisoned samples while maintaining benign features of both poisoned and benign samples. Extensive experiments demonstrate the effectiveness of our method across all evaluated backdoor attacks and all other defense methods under various datasets and network architectures.
Limitations and future work. Although only limited clean data is needed for our method to achieve remarkable defense performance, the accessibility of clean data is still important limitation of the proposed method, which may restrict the application of our method. Therefore, a promising direction for future work is to further reduce the requirement of clean data by exploring data-free neural polarizer or learning neural polarizer based on poisoned training data.
Broader impacts. Backdoor attacks pose significant threats the deployment of deep neural networks obtained from untrustworthy sources. This work has made a valuable contribution to the community with an efficient and effective backdoor defense strategy ease the threat of existing backdoor attacks, even with very limited set of clean samples, which ensures its practicality, Besides, the innovative defense strategy of learning additional lightweight layers, rather than adjusting the whole backdoored model, may inspire more researchers to develop more efficient and practical defense methods.
Structure of Appendix. Detailed algorithms of PGD and UAP, proposed NPD-TU and NPD-TF TP. along with theoretical analysis in the kernel space are provided in Section A. Implementation details of experiments are introduced in Section B. More defense results in different datasets and networks are provided in Section C. Defense performance of different structures of neural polarizer discussed in Section D. Additional experimental results and visualization are presented in Section E.
6 Acknowledgments
This work is supported by the National Natural Science Foundation of China under grant No. 62076213, Shenzhen Science and Technology Program under grant No. OCCXX0210609103057000 No. ZDSYS2021 1021 1415025, No. GXWD20201 231 zhen Science and Technology Program under grant No. JCYJ20210324 2001 1032, CAAI-Huawei MindSpore Open Fund, and the Guangdong Provincial Key Laboratory of Big Data Computing, the Chinese University of Hong Kong, Shenzhen.
Shen-
10

---

References
[1] Jiawang Bai, Kuofeng Gao, Dihong Gong, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Hardly perceptible trojan attack against neural networks with bit flips. European Conference on Computer Vision, pages 104-121, Springer, 2022. 3
[2] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in enns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP), pages 101-105. IEFE, 2019. 3
[3] Léon Bottou and Olivier Bousquet. The tradcoffs of large scale learning. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver. British Columbia, Canada, December 3-6, 2007, pages 161-168. Curran Associates, Inc., 2007. 5
[4] Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. Advances in Neural Information Processing Systems, 35:9727- 9737, 2022. 1.3
[5] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv e-prints, pages arXiv-1712, 2017. 1, 5, 16, 17, 18, 19, 20
[6] Ziyi Cheng, Baoyuan Wu, Zhenya Zhang, and Jianjun Zhao. Tat: Targeted hackdoor attacks against visual object tracking. Puttern Recognition, 142:109629, 2023.
[7] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lcce, 2009. 16
[8] Kuofeng Gao. Jiawang Bai, Bin Chen, Dongxian Wu, and Shu- Tho Xia. Clean-label backdoor attack against deep hashing based retrieval. arXiv preprint
2021.
[9] Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Shu-Taq Tao Xia. Imperceptible and robust backdoor attack in 3d point cloud. arXiv preprint arXiv:2208. 28052222...00. 2022.
[10] Kuofeng Gao, Yang Bai, Jindong Gu, Yong Yang, and Shu-Tao Xia. Backdoor defense via adaptively splitting poisoned dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4005-4015 2023. 3
[11] Tianyu Gu, Kang Liu, Brendan Dolan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230 47244, 2019. 5, 9, 16, 17, 18, 19, 20
[12] Junfeng Guo, Ang Li, and Cong Liu. AEVA: black-box backdoor detection using adversarial extreme value analysis. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. 3,5
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer VisionFECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 630-645. Springer, 2016. 6, 16
[14] Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In The Tenth International Conference on Learning Representations, ICLR 2022. Virtual Event, April 25-29, 2022. OpenReview.net, 2022. 1,3
[15] Andrew Ilyas. Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pages 2137-2146. PMLR, 2018.
[16] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. 6, 16, 19
[17] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99-112. Chapman and Hal//CRC, 2018. 1, 14
[18] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 6, 16
[19] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR 2021. OpenReview.net. 2021. 3, 6. 7, 10, 18
11

---

[20] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li. Ran He. and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. pages 16443-16452. IEEE, 2021. 5, 16, 18, 19, 20
[21] Siyuan Liang, Xingxing Wei, Siyuan Yao, and Xiaochun Cao. Efficient adversarial attacks for visual object tracking. In Computer Vision ECCon--CVV-A 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pages 34-50. Springer, 2020.
[22] Siyuan Liang. Baoyuan Wu, Yanbo Fan, Xingxing Wei, and Xiaochun Cao. Parallel rectangle flip attack: A query-based black-box attack against object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7697-7707, Octoher 2021.
[23] Kang Liu, Brendan Dolan-Gavitt and Siddharth Garg. Fine-pruning: Defending against backdooring attacks deep neural networks. In Research in Attacks, Intrusions, and Defenses: 21st International Symposium, RAID 2018, Heraklion, Crete, Greece, September 10-12, 2018, Proceedings 21, pages 273-294,
Springer, 2018 2, 3, 6, 7, 10, 18
[24] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25th Annual Network and Distributed System Security Symposium. NDSS 2018, San Diego, California, USA, February 18-221, 2018. The Internet Society, 2018. 1. 6, 7. 8, 16, 18. 19
[25] Zhiyun Lu, Wei Han, Yu Zhang, and Liangliang Cao. Exploring targeted universal adversarial perturbations to end-to-end ASR models. In Hynek Hermansky, Honza Cernocký, Lukás Burget, Lori Lamel, Odette Scharenborg. and Petr Motlicek, editors. Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August- 3 September 2021, pages 3460-3440 3464. ISCA, 2021. 14, 15
[26] Wanlun Ma. Derui Wang. Ruoxi Sun, Minhui Xue, Sheng Wen, and Yang Xiang. The" "eatrii"rsuurrctionss Robust backdoor detection via gram matrices. NDSS Symposium, 2022. 3,
[27] Alcksander Madry, Alcksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Repre- sentations, [CLR 2018, Vancouver; BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. 5, 6, 14
[28] Tuan Anh Nguyen and Anh Tran, Input- dynamic backdoor attack. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 6-, 2020, virtual, 2020. 5, 7, 9, 16, 18, 19, 20
[29] Tuan Anh Nguyen and Anh Tuan Tran. Wanet- imperceptible warping-based backdoor attack. In 9th International Conference on Learning Reprexentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 1, 3, 6, 7, 8, 16, 18,
[30] Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer. Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 6106-61 16, 2018.
[31] Rui Shao, Pramuditha Perera, Pong Yuen, and Vishal M Patel. Open-sct adversarial defense. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pages 682-698. Springer, 2020. 17, 19
[32] Rui Shao, Pramuditha Perera, Pong C Yuen, and Vishal M Patel. Open-set adversarial defense with alvan-adversarial mutual learning. International Journal of Computer Vision, 130(4):0000-108, 2022. 18, 19
[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recog- nition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Repre- sentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. 6. 16
[34] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks, pages 1453 1133-160..16 IEEE, 2011. 6, 16
12

---

[35] Bolun Wang, Yuanshun Yao, Shawn Shan. Huiying Li. Bimal Viswanath. Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE, 2019. 3, 6, 7. 10, 18, 19
[36] Ruotong Wang, Hongrui Chen, Zihao Zhu, Li Liu, Yong Zhang. Yanbo Fan, and Baoyuan Wu. Ro- bust backdoor attack with visible, semantic, sample-speeing and compatible triggers. arXiv preprint arXiv:2306. 00816, 2023. 1
[37] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, net, 2020. 4
[38] Shaokui Wei, Jiayin Liu, Bing Li, and Hongyuan Zha. Mean parity fair regression in rkhs. In International Conference on Artificial Intelligence and Statistics, pages 4602 4628. PMLR, 2023, 15, 16
[39] Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. arXiv preprint arXiv:2307. 10562, 2023. 3
[40] Xingxing Wei, Siyuan Liang, Ning Chen, and Xiaochun Cao. Transferable adversarial attacks for image and video object detection. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 954 960, 2019. 1
[41] Wikipedia contributors. Polarizer - accessed 10-May-2023. 2
Wikipedia, the free encyclopedia, 2023. https:/ //en
wikipedia wwwpwooi///olarrroooeooer
[42] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: comprehensive benchmark of backdoor learning. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. 6, 16, 17
[43] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34; Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual. pages 169 (691316622,, 2021. 2, 3, 6, 7, 9, 10, 17, 18
[44] Hao Xiong, Cui Kong, Xiaoxue Yang, and Ying Wu. Optical polarizer based on the mechanical effect of light. Optics Letters, 41(18):331-4311 2016. 2
[45] Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022, 2, 6, 7, 9, 10, 18, 19
[46] Yi Zeng, Won Park, Z. Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks" triggers: A frequeney perspective. In 202/ IEEF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 16461. IEEE, 2021. 3, 5, 7, 8, 9, 16, 18, 19
[47] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. pages 175-191. Springer, 2022. 3
[48] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. Advances in Neural Information Processing Systems, 35: 1667-18680, 2022. 3, 6, 7, 10, 17, 18
[49] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4466-4477, October 2023. 6
[50] Zihao Zhu, Mingda Zhang, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu, Boosting backdoor attack with learnable poisoning sample selection strategy. arXiv preprint arXiv:2307.03328, 2023. 3
13

---

Appendix
This appendix organized as follows.
In Section A, we first provide detailed algorithms of the proposed NPD-TU and NPD-TP. along with an introduction to the project gradient descent (PGD) method and UAP method. Then we present an example of backdoor attacks in the kernel space and provide theoretical analysis the existence linear transformation to mitigate backdoors.
In Section B, we introduce the implementation details, including dataset introduction, implementation details of compared attacks, defenses, and our proposed methods.
In Section C, we display the defense results in comparison with state-of-the-art (SOTA) defenses GTSRB dataset, as well as those on CIFAR-I0 with the VGGI9-BN network.
In Section D, discuss different structures of neural polarizers the defense performance.
In Section E, we present additional experimental results, including defense experiments con- ducted on the CIFAR- 100 dataset, defense performance across various model architectures, and visualization experiments.
A Theoretical analysis and algorithm
A.1 Introduction to PGD algorithm and UAP
In this section, provide introduction PGD attack [27] and universal adversarial perturbation (UAP) attack [25].
PGD. Projected Gradient Descent (PGI) is multi-step variant of Fast Gradient Sign Method (FGSM [17], which is projected gradient descent on the negative loss function, i.e., given an input (ax y) and model fe. the adversarial example is calculated as follows,
x
TT+S (x" arg (Vo
where II++ denotes the projection on to the p-ball with Lp norm: | - x1pp S p. and x = æ. 0, N - 1. is the step size. However, Section 3 of the main script, guided targeted PGD approach to obtain the perturbation. Thus, the adversarial example is updated follows,
confidence-
a+1 1l++s (x" - a sgn (VeLcE(fo(x).T))
(8)
where T' is the target label for æ. denote the adversarial perturbation as follows for convenience:
-- x
UAP.
The main difference between UAP and PGD is that UAP computes a universal adversarial perturbation for the whole training samples; thus the adversarial example for targeted UAP updated as follows,
Ah Σ 1=0
x 1+1
11
x
a sgn
V., LCB Lce((f(a)))
(10)
where T is the target label for æ, and is the number of samples in one training batch. Eq (10) means all the samples in one batch share the sample gradient to update the perturbation.
A.2 Detailed algorithms of NPD-TU and NPD-TP
In the main script, we introduce the algorithm for NPD. Here, we provide the detailed algorithms for NPD-TU and NPD-TP as in Alg 2 and 3.
14

---

Algorithm 2 Neural Polarizer based Backdoor Defense of Targeted UAP (NPD-TU)
1: Input: Training set Dbn, backdoored model fw. neural polarizer go. target label T, inner steps N, learning rate 7 > 0, perturbation bound > O, norm p. hyper- parae-pprsmeterss A1.22,33 > O, warm-up epochs Tu. training epochs
2: Output: Model f(w, O).
3: Initialize 0 to be an identity function, fix w, and construct the composed network f(w, 0).
4: Warm-up: Train f(w. 0) with LCE (Dm)) for To epochs.
5: for = 0, .... T 1 do
6: for bhini-bacch B = { y:)
C Din do
7: for nl 0, ..., N
do Compute ad adversarial perturbation 8 with ||0]|p S and target label T by targeted UAP attack [25] via Eq. (10);
8:
9: end for
Update via outer minimization of Eq. (5) in the main script by SGD.
11: end for
end for
13:
return Model f(w,0) 0).
Algorithm 3 Neural Polarizer based Backdoor Defense of Targeted PGD (NPD-TP)
1: Input: Training Dbn, backdoored model fw,, neural polarizer go: target label T, inner steps N, learning rate 77 > 0. perturbation bound > O, norm hyper- syper-parameters A1,d2.13 > 0, warm-up epochs To. training epochs T.
2: Output: Model f(w. 0).
3: Initialize 0 to be an identity function, fix w, and construct the composed network f(w, 0).
4: Warm-up: Train f(w. 0) with LCE (in) ) for To epochs.
5: for .... T 1 do
6: for mini- batch B = {(,, y. C Don do
7: for n 0, .... N do
8:
Compute adversarial perturbations {(o:)]=_ with ||ō: p and target label T by targeted UAP attack [25] via Eq. (8);
9: end for
10: Update via outer minimization of Eq. (5) in the main script by SGD.
11: end for
12:
end for
13: return Model f(w, 0).
A.3 Theoretical analysis under kernel space
In this section, we demonstrate the existence of linear transformation in reproducing kernel Hilbert space (RKHS), very successful tool in various areas of machine learning. For simplicity, we consider a binary classifier h(,, m)) X' x {0, 1} + IR in reproducing kernel Hilbert space x generated by kernel Mx where m a binary poison indicator, i.e., 1 means adding trigger to the input and = 0 means not adding trigger. Denote the feature of (ar, m) in kernel space by dx (xr m). Given dataset D and distribution of poison indicator M, the poisoned classifier optimized by minimizing mean squared error loss:
had arg min (e- m) -
(1I)
AEMx
Then, by the theorem in [38] which shows that there exists a subspace of Hx in which a functin satisfies Cov(h(æ, (i), m) we provide the following lemma:
Lemma 1. Assume that Pmm = 0) E (0, 1) and @ (x;, m) + (x, m)) m) if x; + xz. Given
poisoned model hea trained by minimizing (11), there exists a non-trivial linear projection operator P such that
Cov( x (x, m), had) HxM m) =
feature of ox ns (zx, m) and (x the inner where ò M (x., m) = (x. m) is the feature
product in H.M-
15

---

Proof. To prove Lemma 1, we first recall that [38] says that the projection operator can be constructed by the cigenfunctions of the operator M between Φx ar (æ. m) and (m(m) for æ E D and m E M. Then, under the assumption that I'mm 0) E (0, 1) and @ (Z;, m) + Φ (2, m) if x; & æj. the Covariance operator ( MMM non-zero operator by its definition and the projection operator generated by 2(xM() is non- trivial.
where M)) is the Covariance operator
The first assumption in Lemma requires that the backdoored classifier hEd is trained on a partially poisoned dataset to preserve its accuracy on clean samples. The second assumption in Lemma requires that the feature of a poisoned sample retains some information of the original clean sample and is not completely overwritten by the trigger. Under these conditions, the Lemma 1 states that there exists a non-trivial projection operator such that the had's prediction for a transformed feature P (at, m) is uncorrelated with poisoning or not. Therefore, such a linear projection operator breaks the correlation between m and the prediction of æ without modifying had, indicating the existence of a neural polarizer in Hxar. Although complex neural network for multi-class classification problems is usually beyond x.MM Mr Lemma provides insights into the existence of an effective neural polarizer.
B More implementation details
In this section, we present the implementation details, which include introduction the datasets used, the implementation specifics of the attacks and defenses compared, as well as our proposed methods. All experiments are run on one RTX 3090Ti GPU and are repeated over five runs with different random seeds.
Datasets. We evaluate the effectiveness of our method using three datasets: CIFAR-10 [16]. Tiny ImageNet [18], and GTSRB [34], following the benchmarks established in [42].
CIFAR-10 comprises 60,000 images distributed across ten classes. The training set consists of 5,000 images per class, while the test set contains 1,000 images per class. Each image in the CIFAR-10 dataset has dimensions of 32 32 pixels.
Tiny ImageNet is subset of the larger ImageNet dataset [7]. It encompasses 100,000 training samples and 10,000 testing samples across 200 classes. The images in Tiny ImageNet have dimensions of 64 x 64 pixels.
GTSRB consists of 39,209 training images and 12,630 testing images, categorized into 43 classes. The image size for each sample in the GTSRB dataset is 32 x 32 pixels.
Models. We evaluate the performance our method in conjunction with two variations on the PreActResNet18 [13] and VGGI9-BN [33] networks. Furthermore, we compare our method with SOTA defense methods on three datasets and the two networks, utilizing 10% poisoning ratio and 5% ratio of clean samples for defense.
Attack details. We introduce more details about the backdoor attack implementation here. We follow BackdoorBench to categorize the eight backdoor attacks according various criteria in Table 10.
Table 10: Categories of the eight backdoor attack algorithms.
Fusion pallern
Size of trigger
Visibility of trigger Visible
Variability trigger agnostic
Num of target classes All2ome
ATTACK
Additive
Non-Additive
Local
Global
Invisible
specific
All2all
BadNets-A24 [1]]
BodNets-A2A [[]]
y
Blended [S]
Input-Aware [28]
[466
SSBA [20]
Trojan [24]
WaNet [29]
16

---

For BadNets-A20 and BadNets-A2A [11], we patch 3 x 3 white square in the lower right corner of the images for CIFAR-10 10 and GTSRB datasets, and For Blended [5], we blend the poisoned samples with a Hello- Hello-Ketty image which has the same size as the poisoned samples, and the blended ratio is set to 0.2.
6 white square for Tiny ImageNet dataset.
Defense details. The seven SOTA defense methods can be categorized into two types based on the available resources for the defender. EP [48] is data-free method that assumes the defender only has access to backdoored model without any clean data. On the other hand, the remaining methods assume that the defender can obtain subset of clean samples along with backdoored model, which aligns with our scenario. In our experiments, a learning rate of 0.01 and a batch size of 256 were used for all SOTA defense methods based on fine-tuning. For ANP [43], the pruning threshold was set to 0.4, as we found that the recommended threshold of 0.2 was inadequate for removing backdoors effectively. Regarding the fine-tuning based SOTA methods, the training epochs were set to 100 for CIFAR- 10 and Tiny ImageNet, and 50 for the GTSRB dataset. All other settings remained consistent with those specified in BackdoorBench [42].
In our NPD method, we used a learning rate of 0.01 with a weight decay of 0.0005 and momentum of 0.9. The training process consisted of 50 epochs for all three datasets, with warm-up period of 5 epochs. During each epoch, the PGD algorithm was employed to search for adversarial perturbations in each batch. The PGD algorithm utilized learning rate of 0.1. To constrain the adversarial perturbations on the CIFAR- 10 dataset, we applied an lg norm with a bound 3.
C Defense results in comparison to SOTA defenses on GTSRB and VGG19-BN
To further investigate the generalization performance of our NPD on different datasets and networks, conducted analysis the GTSBR dataset and CIFAR-10 VGG19-BN network. We compared the performance NPD with various defense methods against different attacks, and the results are presented in Table 11 and Table 12.
As shown in Table 11, our experimental findings consistently demonstrate the superiority of our proposed methods over state-of-the-art defenses in terms of DER and ASR across almost all attacks, except for BadNet and Trojan. Notably, NAD exhibits the best performance against the Trojan attack, which can be attributed to the strong single trigger pattern adopted by Trojan. NAD effectively reverses the trigger through optimization, making it easier to detect and mitigate the Trojan attack. Furthermore, while NPD-TU and NPD-TP exhibit superiority in terms of a low ASR, NPD demonstrates comparable trade-off between ACC and ASR. On average, NPD achieves a DER of 76.76%, which is 6.26% higher than the best SOTA method, i-BAU. The higher DER achieved by NPD the remaining attacks further validates the effectiveness of our methods in detecting and mitigating various backdoor attacks compared to existing defenses.
Regarding the performance of the VGG19-BN network, as shown in Table 12, NPD-TU and NPD outperform all other defense methods, underscoring the effectiveness of our proposed defense methods in removing backdoor attacks. However, NPD-TU fails to defend against the Input-Ave Aware attack, which is similar to the performance of i-BAU defense. This highlights the importance of incorporating adversarial attacks into the defense strategy, as UAP does not always guarantee successful defense. Nonetheless, our proposed method exhibits comparable and stable performance across almost all attacks.
In summary, our analysis of the GTSBR dataset and VGG19-BN network showcases the robust generalization performance of NPD against various backdoor attacks. The results underscore the effectiveness of our proposed defense methods, especially NPD-TP and NPD, in removing backdoor attacks.
D Different structures of neural polarizers on the defense performance
In this section, we demonstrate the effectiveness of our Conv-BN structure in implementing the neural polarizer for defense against backdoors. We compare it with two Conv-BN structures, a Linear transformation layer, and two denoising layers used in two adversarial defense works: OSAD [3]]
17

---

Table 11:
Comparison with the state-of-the-art defenses on GTSRB dataset with 5% benign data and 10% poison ratio on PreAct-ResNet118 (%)
Backdoored
FP [23]
NAD [19]
NC [35]
ANP [43]
ATTACK
ACC
DER
ASR
DER
ACC
ASR
DER
ACC
ASR
BodNess-A20 [1]
96.35
95.02
N/A
0.00
97.51
97.54
79.94
93.47
0.022
96.06
96.79
0.21
97.40
BadNess-AAA [1]]
97.05
92.33
N/A
98.11
0.51
95.91
97.84
2.46
94,94
94.05
0.50
94.42
96.73
50.39
70.81
Blended [5]
97.97
99.67
N/A
98.20
68.45
65.61
97.
99.29
50.19
48.34
5.70
72.17 17
98.25
99:
50.00
Impor-Aware [28]
V7.17
97.09
N/A
97.98
0.22
98.43
97:47
65.55
65.77
95.79
1.08
97.32
96. 20
0.00
98.06
LP [46]
97.97
99.58
N/A
97.87
69.19
65.15
98.24
79.76
59,91
92.22
0.18
96.82
98.03
60:36
69.61
SSBA [30]
98.31
99.77
N/A
98.47
60.19
69.79
98. 37
96.95
51,41
90.75
1.51
95.35
98. 36
98.98
50.40
Trojan [24]
98.33
100.00
N/A
98.00
42.08
78.79
98.01
0.10
99.79
92.29
0.02
96.97
98.17
86.92
56.46
WaNer [29]
95.71
98.20
N/A
0.28
98.96
98.
99.08
96.34
0.01
10
99.01
Avg
97.71
N/A
98.20
80.12
83.79
97.99
53.01
72.35
87.90
1.13
49
74.04
i-BUU [+5]
BP [-8]
NPD:TU(Ours)
NPPD-TR(Our))
IND(OOurs)
ATTACK
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
BadNets-A20 [1]]
96.35
97.51
1.38
96.82
97.01
97.78
$.19
93.42
4.62
BadNets-A2A [1]]
95.30
0.43
95.08
96.45
1,40
95.17
96.21
0.00
95.74
96.18
0.00
95.73
97.09
9.33
91.50
Blended [5]
95.04
96.39
50.17
95.12
100.00
48.73
97.46
0.00
99.58
97.25
0.00
99:18
97.32
2.44
98.29
Input-Aware [28]
96.03
0.00
97.98
92.98
0.10
96.40
96.03
6.52
94,71
97.13
0.16
98.44
95.72
1.72
96.96
LF [46]
88.69
91.44
96.40
99.15
49.43
97.60
99.61
97.28
0.00
97.30
99.07
SSBA [20]
87.27
0.18
94.28
97.59
99.32
19.87
97.91
0.00
99,70
97.56
0.00
99.51
98.02
3:45
98.01
Trojun [24]
93.66
0.00
97.66
97.46
0.05
99.54
97.47
0.00
99.57
96.37
0.08
98.98
96.47
0.00
99.07
WaNet [29]
97.50
0.26
98.97
97.26
26.92
85.64
96.99
13.55
92.33
97.66
0.00
99.10
97.
7.77
95.22
Avg
93.73
90.50
41.04
77.79
97.09
97.42
97.15
1.05
96.94
96.76
Table 12: Comparison with the state-of-the-ar art defenses on CIFAR-10 dataset with 5% benign data and 10% poison ratio on VGG19-BN (%).
Backdoored
FP [23]
NAD [19]
NC [35]
ANP [13]
ATTACK
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
BadNess-A20 [1]]
90.42
94.43
N/A
89.11
12.39
90.36
86.80
5.77
92.55
$8.97
5.63
93.68
90.44
87.64
53,40
BodNess-AZA [1]]
84.39
N/A
90.51
88.15
1.60
91.16
91.29
81.87
51,26
Blended [5]
91.60
96.68
N/A
89.60
93.14
50.77
87.45
$6.98
52.78
91.91
99.50
50.00
91.69
97.37
50.00
Impat-Aware (28]
88.66
94.58
N/A
91.55
14.57
90.01
91.09
14.06
90.26
89.70
97.02
50.00
89.67
36.76
78.91
LF [46]
N/A
1.29
56.27
85.08
3.07
55.38
$8.33
1.22
56.31
1.34
56.24
SSBA
90.85
95.11
N/A
89.26
65.33
64.09
88.11
52.22
70.08
90.85
95.11
50.00
91.11
76.00
59.56
Trojan [24]
91.57
100,00
N/A
90.04
29,71
84,38 38
87.01
5.17
95.14
91.57
100.00
50.00
89.27
0.00
98.85
WaNet [29]
84.58
96.49
N/A
91.10
3.36
96.56
90.68
10.23
93.13
84.58
96.49
50.00
89.82
0.96
97.76
Avg
89.02
84.44
N/A
89.82
27.71
78.36
88.05
22.39
80.54
89.63
72,42
56.01
90.31
47.74
68.35
LBAU
EP [48]
NPD-TU[(OUS))
NPD:TP(Ours)
NPIXOurs)
ATTACK
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
BadNets-A2 [11]
$7.69
3.13
94.28
87.56
7.28
92.15
90.54
5.98
94.23
90.06
3.06
95.51
89.52
0.10
96.71
BadNets-A24 [1]]
86.86
88.95
57.15
90.82
81.76
90.88
89.81
2.36
90.34
Blended [5]
88,45
51.67
70.93
90.99
22.62
86.72
90.66
0.00
97.87
91.23
0.00
98.15
90.85
9.99
92.97
Impot-Aware [28]
89.81
78.93
57.82
88.92
3.84
95,37
89.15
96.51
50.00
$9.98
0.02
97.28
87.99
2.57
95.67
LF [46]
83.06
6.66
53.48
79.36
18.88
48.04
88.78
0.96
56.44
$9.05
0.89
56.47
87.64
0.91
56.46
SSBA [20]
12.37
$8.75
89.62
54.21
90.42
97.18
90.71
97.28
88.94
94.19
Trojan [24]
86,40
2.69
96.07
89.42
5,23
96.31
90.35
0.00
99.39
90.81
8.89
95.18
89.79
0.86
98.68
WaNett [29]
89.61
2,40
97.04
86, 10
73.61
61.44
88.85
5:44
95.52
89.03
1.35
97.57
88.22 25
1,88
97,30
Avg
87.19
20.00
81.30
87.74
35.72
73.72
89.95
16.22
84.11
90.22
2.58
90.93
89.10
2,86
90.79
and OSDN [32]. For the denoising structures, we found that inserting four denoising layers in all four blocks of the backdoored model as OSAD and OSDN leads to the collapse of the entire network. Hence, we adopt a single denoising layer inserted into only one layer of the backdoored model as NPD. This limitation arises due to the scarcity of training samples available for fine-tuning in the defense scenario. As for the Linear layer, it means use linear layer with weights and biases that have the same size as the latent features. Consequently, this structure has fewer parameters compared network,
to ours. We conducted experiments on the CIFAR- 10 dataset using the PreAct-ResNte and the other experimental setup aligns with our method. The defense performance results are shown in Table 13.
From the table, we observe that both OSAD and OSDN successfully maintain performance on clean data. However, the ASR experiences a slight drop in most cases, possibly due to the fact that OSAD and OSDN adopt a residual structure where backdoors cannot be completely eliminated. The two Conv-BN layers structure has high probability of collapsing directly, which may be attributed to the
18

---

insufficient training data for such a multi-layer structure. The Linear structure shows lower ACC and higher ASR. This could be because fewer parameters are not sufficient to remove backdoors, and random initialization of the inserted layer leads to a decrease in final accuracy. This experiment demonstrates the superiority of the designed Conv-BN structure in defending against backdoors.
Table 13:
Defense performance under different structures of inserted layers on CIFAR-10 dataset with PreAct-ResNet18.
Structure
OSAD [31]
OSDN [32]
Linear
(Cony-BN)x2 ACC
Cony-BN (Ours)
ATTACK
ACC
ASR
ACC
ASR
ACC
ASR
ASR
ACC
ASR
BadNets-A20 [1]]
91.68
48.43
90.47
45.26
87.75
21.12
10.05
99.56
88.93
1.26
Blended [5]
87.93
12.66
91.74
26.87
86.98
44.36
90.51
9.06
91.18
0.41
Input-Aware [28]
93.53
92.78
92.32
60.25
86.85
58.51
10.07
99.56
89.57
0.11
LF [46]
92.32
27.28
88.86
67.73
88.41
91.59
88.36
0.36
90.06
0.21
SSBA [20]
89.77
87.46
90.53
84.79
88.81
79.49
10,10
99.61
90.88
2.77
Trojan [24]
93.45
99.99
93.38
99.99
87.85
100.00
91.80
18.80
92.37
6,51
WaNet [29]
93.07
97.99
92.41
85.11
87.22
96.54
10.07
100.00
91.57
0.80
Avg
91.68
66.65
91.39
67.14
87.70
70.23
44.42
60.99
90.65
1.72
E
Additional experimental results
E.1 Defense experiments on CIFAR-100 dataset
To show the effectiveness of NPD method classification tasks with more labels, we complement experiment on CIFAR-100 dataset [16], which has 100 classes in total and each class has 500 images for training and 100 images for testing. As shown in Table 14, the average ACC and ASR of NPD 65, 65.49% 1.96%, which indicates its competitiveness in comparison to existing defense approaches and showcases its efficacy in more classes scenarios.
Table 14: Comparison with the state-of-the-art defenses on CIFAR- dataset with 5% benign data and 10% poison ratio on PreAct-ResNet11 (%).
Backdoored
NC [35]
i-BAU [45]
NPD (Ours)
ATTACK
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
ACC
ASR
DER
BadNets-A20 [11]
67.23
87.43
NIA
66.05
0.14
93.06
60.37
0.04
90.26
64.27
0.06
92.20
Blended [S]
69.28
99.59
N/A
67.55
98.94
49.46
63.75
0.79
96.64
66.30
4.43
96.09
LF [46]
68.82 82
94.53
N/A
67.67
21.48
85.95
63.85
94.69
65.37
0.22
95.43
SSBA [20]
68.97
96.42
N/A
67.37
78.64
58.09
63.09
28.91
80.82
65.57
0.83
96.10
Trojan [24]
68.93
100
N/A
66.00
0.10
98.48
63.73
0.86
96.97
65.13
1.84
97.18
Wanet [29]
69.83
98.46
N/A
69.76
7.14
95.62
65.31
43.96
74.99
66.27
4.36
95.27
Avg
68.84
96.07
N/A
67.40
34.41
80.11
63.35
12.46
89.06
65.49
1.96
95.38
E.2 Defense experiments across various model architectures
In this part, we discuss the selection of the layer insert the neural polarizer (NP) in a new model and the performance of NPD under different model architectures. In the manuscript, we have investigated the impact of NP insertion different layers on the PreActResNet18 network. Our empirical study demonstrates that inserting NP in deeper layers effectively removes backdoors, while inserting it in shallower layers may lead decrease in benign accuracy and instability. To further validate this finding and show the effectiveness and robustness of NPD across different model architectures, we conducted additional experiments on VGG19, Inception- VV,, and Densenet161 networks. As presented in Table 15, inserting NP into the last three layers achieves remarkable performance (high ACC and low ASR), while decrease in accuracy occurs when shallow layers are selected. This experiment shows that NPD is robust across different model architectures.
19

---

Table 15: Performance of neural polarizer insertion in different layers across various network architectures (%).
Layer ->
Backdoored
Shallow
Middle
Third-to-last ASR
Penultimate
Last
Network
Atlack
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ACC
ASR
ACC
ASR
BadNets-A2O [1]]
89.36
95.93
77.58
2,83
87.41
0.40
88.68
5.17
87.28
3.81
87.73
4.06
Blended [5]
90.17
99.12
77.59
17.19
88.89
25.53
89.27
9.78
89.31
7.40
87.95
5,78
VGG19
Input-Aware [28]
77.67
94.58
70.31
1.47
70.99
20.37
74,30
7.13
72.59
7.77
77.11
1.47
SSBA [20]
89.48
91.86
72.76
0.67
88.45
20.20
89.05
10.53
88.63
6.12
88.70
4.31
BadNets-A20 [11]
90.68
94.50
11.37
92.84
90.76
10.82
91.49
8.54
90.62
2.51
91.69
4.70
Blended [5]
93.73
99.82
10.00
100.00
92.70
69.77
93.71
8.91
93.05
4,51
93.71
0.01
Inception
Input-Aware [28]
91.60
98.80
19.22
0.38
90.52
10,50
91.65
2.46
91.04
5.30
91.58
0.69
SSBA [20]
93.33
98.30
10.00
100.00
92.21
89.81
93.35
7.52
92.88
6.13
92.23
1.13
BadNets-A20 [11]
84.33
89.68
72.78
44,22
81.24
80.09
81.32
9.19
82.46
8.48
82.06
1.91
Blended [5]
86,37
98.79
75.09
3.79
82.87
85.98
82.71
13.83
83.75
9.16
83. 13.18
4.22
DenseNet161
Input-Aware [28]
84.46
94.41
77.20
17.11
83.65
43.57
82.81
3.56
82.02
1.76
83.44
1.56
SSRA [20]
84.18
84.
70.99
9.57
81.57
30.27
82.72
7.09
82.06
8.41
81.41
8.48
Avg
87,95
94.99
53.74
32.51
85.94
40.61
86.76
7.81
86.31
5.95
86.73
3.19
E.3
Visualization of targeted adversarial perturbation
We empirically establish that targeted adversarial perturbation (T-AP) is better surrogate the trigger compared to untargeted adversarial perturbation (U-AP), as demonstrated through quantitative experiments in the manuscript. It could also be explained from the following aspects: From intuitive standpoint, T-AP can be regarded as a subregion of U-AP, and the unknown trigger is specific instance of T-AA. Consequently, employing the projected gradient descent (PGD) adversarial attack method, the generated T-AP is more likely to exhibit proximity to the trigger when contrasted with the generated U-AP. Moreover, we visualize the trigger, as well the generated T-AP and U-AP. As depicted in Fig. 4, the T-AP is more visually similar to the trigger. To reinforce our argument quantitatively, we calculate the L2 distance the latent space of the backdoored model among the targeted adversarial example (AE), untargeted AE, and the poisoned sample with the trigger. The distance between targeted AE and poisoned sample 36.4 is substantially smaller than that between Untargeted AE and poisoned sample 93.84. This supplementary experiment reaffirms our contention that T-AP serves as a more faithful surrogate to the concealed trigger in comparison to U-AP.
E.4 Grad-CAM Visualization
We employ Grad-CAM figures to visualize the effects of our NPD on BadNets, Blended, and Trojan attacks, given that these three attack methods employ visible triggers. As illustrated in Figures to Neural Polarizer (NP) effectively redirects the network's ' attention away from the triggers and toward the subject of figures. This visual evidence substantiates the successful removal of backdoors by NP in backdoored models.
20

---

Trigger.
Targeted pert.
Untargeted pert.
Poisoned sample
Targeted AE
Untargeted AE
Feature maps
36.4
93.84
lz distance of feature maps between AEs and poisoned samples
Figure 4: Comparison of trigger (first column), targeted perturbation (second column), and untargeted perturbation (third column) in the input and latent space. Top left: The trigger used in Blended attack. Top middle: Visualization of the perturbation generated by our targeted adversarial perturbation in NPD. Top right: Visualization of the perturbation generated by the untargeted adversarial perturbation. Bottom left: The feature map derived from the poisoned sample in the feature space. Bottom middle: Feature map of the adversarial example generated through our NPD approach (Targeted AE).
Bottom right: Feature map of the adversarial example produced through the untargeted adversarial perturbation (Untargeted AE).
Beaiga sammlees
Poisoned samples
Before
After
Figure 5: Visualizations of benign and poisoned samples, alongside their corresponding Grad-CAM visualizations before and after the insertion of NP of BadNets attack CIFAR-10 10 dataset and PreActResNet18 network.
21

---

Beaiga samples
Poisoned samples
Before defense
After defense
Figure 6: Visualizations benign and poisoned samples, alongside their corresponding Grad-CAM visualizations before and after the insertion of NP of Blended attack on CIFAR-10 dataset and Neet network.
Beniga samples
Poisoned samples
Before defenne
Alter defense
Figure Visualizations of benign and poisoned samples, alongside their corresponding Grad- CAM visualizations before and after the insertion of NP of Trojan attack on CIFAR-10 dataset and Pre1ctResNet18 network.
22


Reviews:
[{"review_id": "Co1Na0IScy", "reviewer": "Reviewer_cjy4", "summary": "This paper proposes a novel defense method to filter trigger information from poisoned samples. It inserts a learnable intermediate layer (called neural polarizer) into the backdoored model, and proposes bi-level optimization solution to approximate perturbation and target label. Experimental results demonstrate the effectiveness of Neural Polarizer over other defense baselines.", "questions": "\n1.Regarding approximating perturbation $\\Delta$ and target label T (eq.5): in inner minimization, the algorithm first estimates the target label, then applies PGD to generate perturbations. I assume this step is very important, because it decides the outer minimization. That are the estimated perturbations look like? Are they similar with the real triggers, or they are just the adversarial perturbations? Are there any experiments/visualizations to elaborate that?\n\n2.How to choose which layer to insert the neural polarizer? Does your method robust to the choice of layer?\n\n3.What if there are more labels. For example, CIFAR-100 or even more labels. Does the proposed method still work?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "\n1.The paper is well-written and easy to follow.\n\n2.The proposed Nueral Polarizer is interesting.\n\n3.The experiments are comprehensive.\n\n", "weaknesses": "1.The paper proposes NPD (Neural Polarizer based backdoor Defense) as a normal setting. It also proposes NPD-TP (assume the target label is known) and NPD-TU (assume having full access of benign samples). The NPD-TP and NPD-TU relax the limitation, and is kind of unfair compared to other defense baselines. However, based on Table 1 and Table 2, most of best performance comes from NPD-TP/NPD-TU, while NPD does not perform very well under some situations.\n\nMore specific, In Table 1, defense on CIFAR-10: NPD does not always perform the best. For example, ASR under WaNet is 0.80 (larger than WaNet with FP and NAD), ASR under Trojan is 6.51 (larger than Trojan with i-BAU and EP). Similar phenomenon happens in Table 2: defense on Tiny ImageNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we would like to show our sincere appreciation to the reviewer for the valuable time and constructive comments on our submission.\n\n**Q1. The variants NPD-TP (assume the target label is known) and NPD-TU (assume having full access to benign samples) relax the limitation, and is kind of unfair compared to other defense baselines. Most of best performance comes from NPD-TP/NPD-TU, while NPD does not perform very well under some situations.**\n\n**R1:** Thanks. We would like to explain from the following three points.\n* **Assumption of NPD-TP and NPD-TU.** We would like to clarify that NPD-TP and NPD-TU share the same assumption that the target label is known. We think that the sentence at Line 186 \"*the targeted universal adversarial perturbation (TUAP) for all benign samples, dubbed NPD-TU*\" may mislead the reviewer to think that NPD-TU assumes to have full access to benign samples. Actually, that means the access of all benign samples used for fine-tuning, rather than all benign samples in the original training dataset. That is same with NPD, NPD-TP and all other compared defense methods.\n* **The role of NPD-TP and NPD-TU.** The intrinsic difference between NPD and NPD-TP/NPD-TU is the target label: the former obtains the target label via a dynamic strategy of target label prediction, while the latter assumes to have the true target label. Thus, NPD-TP/NPD-TU serves as the reference to study the effect of the dynamic prediction strategy in NPD. We would like to refer you to the analysis of its effect and mechanism in the **first common response** posted at the top position.\n* **Performance of NPD.** If removing NPD-TP and NPD-TU from Tables 1 and 2 in the main manuscript, we can see that NPD still has a significant performance advantage over all other compared defense methods. Specifically, in Table 1, the most competitive method is i-BAU, its average ACC/ASR/DER are 88.95/6.11/92.42%, while those of NPD are 90.75/1.62/95.56%; in Table 2, the most competitive method is ANP, its average ACC/ASR/DER are 42.64/15.87/80.03%, while those of NPD are 50.55/2.17/90.84%. Although NPD doesn't perform the best at all cases, we believe the huge advantage of the overall performance is enough to demonstrate the effectiveness of NPD.\n\nHope above points could address your concern, and we will clearly clarify the assumption and role of NPD-TP and NPD-TU in the revised manuscript. Thanks again for your constructive comment.\n\n**Q2. What do the estimated perturbations look like? Are they just adversarial perturbations?**\n\n**R2:** Thank you for your insightful question. We would like to refer you to the **common response** for a comprehensive analysis of the mechanism of NPD, where **visualization of the estimated perturbations** is provided in **Fig. 1** in the **supplementary PDF**.\n\nMore importantly, we would like to briefly summarize some important information for your reference. In the visualization, both targeted adversarial perturbation (T-AP) and untargeted adversarial perturbation (U-AP), as well as the corresponding adversarial examples (T-AE, U-AE) are provided. To explain the effect of T-AP, we also present the $L_2$ distance between T-AE, U-AE, and poison samples in the latent space. From the visualization, we find that\n\n1. **Visualization:** T-AP exhibits more visual similarity to the trigger both in the input space and latent space compared to U-AP, showing targeted perturbations are more effective as surrogate perturbations to the triggers.\n\n2. **Quantitative analysis:** In latent space, the $L_2$ distance between T-AE and the poisoned sample is 36.4, which is significantly smaller than the distance between U-AE and the poisoned sample, i.e., 93.84. This quantitative analysis further supports the efficacy of our targeted adversarial perturbations as a better surrogate for unknown triggers.\n\nOverall, both visualization and quantitative analysis provide substantial evidence to validate the effectiveness of NPD in producing targeted adversarial perturbations, which is different from ordinary adversarial perturbations. Hope the above points could address your questions.\n\n**Q3. How to choose the layer to insert the neural polarizer? Does your method robust to the choice of layer?**\n\n**R3:** We appreciate your question. Due to space limit, we refer you to **R2 of Reviewer BztP** for more details, which contains an in-depth analysis and performance of NP insertion in various layers across VGG19, Inception-V3, and Densenet161 networks, respectively. Briefly, NPD is robust across different network structures. And, it remains effective and reliable when the NP is inserted into deep layers of the network.\n\n**Q4. What if there are more labels, e.g., CIFAR-100?**\n\n**R4:** Thanks for this suggestion. Actually, we have evaluated NPD on Tiny ImageNet, which has 200 classes. The results on CIFAR-100 in comparison with two SOTA defenses are presented in **Table 1**. The average (ACC, ASR) of NPD is (65.49%, 1.96%), while those of NC and i-BAU are (67.4%, 34.41%) and (63.35%, 12.46%), respectively. Our NPD shows significant advantages.\n\n**Table 1: Comparison with the state-of-the-art defenses on CIFAR-100 dataset with 5% benign data and 10% poisoning ratio on PreAct-ResNet18 (%).**\n||Backdoored||NC||i-BAU||NPD||\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n||ACC|ASR|ACC|ASR|ACC|ASR|ACC|ASR\nBadNets|67.23|87.43|66.05|0.14|60.37|0.04|64.27|0.06\nBlended|69.28|99.59|67.55|98.94|63.75|0.79|66.30|4.43\nLF|68.82|94.53|67.67|21.48|63.85|0.18|65.37|0.22\nSSBA|68.97|96.42|67.37|78.64|63.09|28.91|65.57|0.83\nTrojan|68.93|100.00|66.00|0.10|63.73|0.86|65.13|1.84\nWanet|69.83|98.46|69.76|7.14|65.31|43.96|66.27|4.36\nAvg|68.84|96.07|67.40|34.41|63.35|12.46|65.49|1.96"}, {"review_id": "YbUaBqGf1E", "reviewer": "Reviewer_BztP", "summary": "The paper proposes a lightweight and effective backdoor defense by inserting a trainable neural layer block. Without modifying original backdoor model, the proposed method can remove backdoor behaviors by filtering poisoned features via the trainable neural block. The authors conduct sufficient experiments to demonstrate the effectiveness of proposed method.", "questions": "1. Does this proposed method work for a network without batch normalization layer?\n2. Please explain more about how to choose layers to insert neural polarizer.\n3. Could the authors visualize features before and after neural polarizer to qualitatively analyze the effectiveness (e.g. using GradCam)?", "limitations": "The authors have addressed the limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple but effective. The inserted neural block only includes one convolution layer followed by a BN layer. Training this layer block is not time-consuming.\n2. The experiments are very sufficient. The comparison experiments are conducted with six backdoor defenses against seven backdoor attacks on three datasets. Analysis experiments are also conducted including the effectiveness of losses, poisoning ratios and clean ratios.", "weaknesses": "1. The paper does not introduce how to choose different layers to insert the neural polarizer. Since the proposed method largely depends on network architectures. Please also provide more results using more network architectures e.g. vgg.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for carefully reading our paper. We are encouraged by the positive comments of simple but effective method and sufficient experiments.\n\n**Q1. More results using more network architectures, e.g., vgg. Does this proposed method work for a network without batch normalization layer?**\n\n**R1:** We appreciate your valuable question. We perform defense results of NPD in comparison with NC and i-BAU in **Table 1**. It shows that NPD works for VGG19, which doesn't have batch normalization layer. Please refer to **Table 2 in R2** for more experimental results, which contains performance of NPD under more architecture, i.e., Inception-V3, and Densenet161 networks. In brief, our NPD approach demonstrates robustness and effectiveness across a wide range of network architectures.\n\n**Table 1: Performance of NPD in comparison with NC and i-BAU on VGG19 network.**\n\n||Backdoored||NC||i-BAU||NPD||\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n|ATTACK|ACC|ASR|ACC|ASR|ACC|ASR|ACC|ASR\nBadNets|89.36|95.93|87.90|49.66|14.61|0.43|87.73|4.06\nBlended|90.17|99.12|89.09|96.72|88.61|59.86|87.95|5.78\nInput-Aware|77.67|94.58|74.45|4.02|72.15|16.22|77.11|1.47\nSSBA|89.48|91.86|89.48|91.86|88.08|3.74|88.70|4.31\n\n**Q2. How to choose layers to insert neural polarizer.**\n\n**R2:** Thanks and we acknowledge that the selection of layer is important for NP. Actually we have investigated the performance of NP insertion into different layers on PreAct-ResNet18 in manuscript. To further investigate the impact of layer selection, we conduct experiments on three more backbones in **Table 2**, i.e., VGG19, Inception-V3, and Densenet161 networks, besides PreAct-ResNet18 and VGG19-BN in our paper. From Table 2 below and Figure 4 in our paper, we find that: inserting NP into the last three layers achieves remarkable performance (high ACC and low ASR), while a decrease in accuracy occurs when shallow layers are selected.\n\nTherefore, the last feature layer is recommended, which yields better defense performance and maintains robustness against backdoor attacks.\n\n**Table 2: Performance of neural polarizer insertion in different layers across various network architectures(%).**\n|||Shallow||Middle||Third-to-last||Penultimate||Last||\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n|||ACC|ASR|ACC|ASR|ACC|ASR|ACC|ASR|ACC|ASR\nVGG19|BadNets|77.58|2.83|87.41|0.40|88.68|5.17|87.28|3.81|87.73|4.06\n||Blended|77.59|17.19|88.89|25.53|89.27|9.78|89.31|7.40|87.95|5.78\n||Input-Aware|70.31|1.47|70.99|20.37|74.30|7.13|72.59|7.77|77.11|1.47\n||SSBA|72.76|0.67|88.45|20.20|89.05|10.53|88.63|6.12|88.70|4.31\nInception-V3|BadNets|11.37|92.84|90.76|10.82|91.49|8.54|90.62|2.51|91.69|4.70\n||Blended|10.00|100.00|92.70|69.77|93.71|8.91|93.05|4.51|93.71|0.01\n||Input-Aware|19.22|0.38|90.52|10.50|91.65|2.46|91.04|5.30|91.58|0.69\n||SSBA|10.00|100.00|92.21|89.81|93.35|7.52|92.88|6.13|92.23|1.13\nDenseNet161|BadNets|72.78|44.22|81.24|80.09|81.32|9.19|82.46|8.48|82.06|1.91\n||Blended|75.09|3.79|82.87|85.98|82.71|13.83|83.75|9.16|83.18|4.22\n||Input-Aware|77.20|17.11|83.65|43.57|82.81|3.56|82.02|1.76|83.44|1.56\n||SSBA|70.99|9.57|81.57|30.27|82.72|7.09|82.06|8.41|81.41|8.48\n\n**Q3. Visualize features before and after neural polarizer by Grad-CAM.**\n\n**R3:** To better understand the effect of neural polarizer (NP) in purifying poisoned features, we visualize the benign samples, poisoned samples, and their Grad-CAM visualization before and after NP in **Fig. 2-4 in supplementary PDF**. We visualize BadNets, Blended, and Trojan attacks since they use visible triggers. As the figures show, NP corrects the network's attention from triggers to the subject of figures, showing the NP successfully removes backdoors in backdoored models."}, {"review_id": "0fp5hLwqX2", "reviewer": "Reviewer_qQYz", "summary": "This paper proposes propose a backdoor defense method by inserting a learnable neural polarizer into the backdoored model as\nan intermediate layer, in order to purify the poisoned sample via filtering trigger information while maintaining benign information. To more effectively remove backdoor, this paper leverages the reverse trigger and target label to remove backdoor.", "questions": "Can you explain why the method works with the reversed trigger which may not be correctly reversed?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well written and technically sound.\n2. The experiments are sufficient, and its performance achieves SOTA", "weaknesses": "1. Neural Polarizer needs to reverse the backdoor trigger and the target label, which is similar to Neural Cleanse. Furthermore, former papers demonstrate that trigger reverse mainly works well on static trigger such as BadNets. The results on sample-specific trigger are bad and the results of reverse trigger and target label are mostly wrong. Thus, the scope where Neural Polarizer can be used is limited.\n2. The novelty is limited. Because trigger reversed has been proposed before, the contribution of this paper is to finetune the model with the reversed trigger on Neural Polarizer layer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time in reading our work and positive review on our techniques and experimental results. We appreciate your insightful questions and comments.\n\n**Q1. Neural Polarizer (NP) needs to reverse the backdoor trigger and the target label, which is similar to Neural Cleanse. The scope where NP can be used is limited. The novelty is limited.**\n\n**R1:** Although both our NPD and NC employ adversarial techniques and targeted label to fine-tune the backdoored model, our NPD shows novelty and differs from NC in the following three aspects:\n\n**1. Dynamic and sample-specific prediction strategy of target label.** We predict the target label for each training sample by $y_i^{'} = \\arg\\max_{k_i \\neq y_i} \\hat{f}_{k_i}(\\mathbf{x}_i, \\boldsymbol{\\theta})$, while NC estimates a common target label for all samples. It has three strengths:\n* **The predicted target label enables us to generate targeted adversarial perturbation (T-AP), which shows better performance compared to untargeted adversarial perturbation(U-AP)**. The reason is that T-AP is more effective as surrogate perturbations to the triggers. We have provided ablation study in Table 4 in the manuscript. We also provide a comparison of generated T-AP, U-AP, and corresponding poisoned sample in the latent space of a backdoor model in **supplementary PDF**, which shows T-AP is more visually similar to poisoned sample than U-AP.\n* **The dynamic and sample-specific target label prediction is applicable to both all2one and all2all attack settings** (see **Table 1** for defense on all2all attacks). Since the defender doesn’t know whether the attack is all2one, all2all, or has multi-trigger multi-target, our strategy can flexibly cope with these situations and plays a better false tolerance role.\n\n**2. Targeted and sample-specific adversarial perturbation.** Compared to targeted universal adversarial perturbation and the local mask strategy used in NC, our NPD adopts the targeted and sample-specific adversarial perturbation (AP) and doesn't restrict the format of AP. Therefore, NPD can cope with various types of backdoor triggers, e.g., sample-specific, or global. This is also reflected in our experimental performance.\n\n**3. Fine-tuning Paradigm.** In comparison to NC which needs to fine-tune the whole network, our NPD only fine-tunes the inserted transformation layer. It is **data-efficient** which performs well with very few clean data, and **computationally efficient**, which needs less running time for defense. Please refer to Tables 6 and 7 in the manuscript for experimental details.\n\nA systematic and detailed analysis of NPD's mechanism and novelty are presented in the first **Common Response**.\n\n**Table 1: Defense performance under various all2all attacks on CIFAR-10(%).**\n||Backdoored||NPD||\n:-:|:-:|:-:|:-:|:-:\n||ACC|ASR|ACC|ASR\nBlended|91.59|83.50|91.07|6.24\nLF|86.60|78.55|90.10|2.10\nInput-Aware|91.91|84.80|90.53|4.88\nSSBA|91.30|85.04|91.14|1.29\n\n**Q2. Why does the method work with the reversed trigger which may not be correctly reversed?**\n\n**R2:** Thanks for this insightful concern. Since it is difficult to access or exactly recover the unknown trigger, one feasible solution is to find some surrogates or approximations, such as AP in our NPD, i-BAU and NC. Moreover, we argue that the targeted AP (T-AP) adopted in NPD is a good surrogate to the trigger and better than the untargeted AP (U-AP), in three ways:\n* **Intuitively, T-AP is more likely to be close to the trigger than U-AP.** Because, the set of T-AP is a sub-region of that of U-AP, and the trigger is a particular T-AP.\n* **We visualize and quantitatively measure the distance of adversarial example with T-AP (i.e., targeted AE) and poisoned sample in Fig.1 in supplementary PDF.** It shows that T-AP is more visually similar to the trigger than U-AP. The distance between targeted AE and poisoned sample in the latent space of backdoored model is much smaller than that between untargeted AE and poisoned sample. It verifies that T-AP serves as a better surrogate to poisoned sample.\n* **T-AP makes significant contribution to backdoor defense.** To investegate the effects of T-AP and U-AP on backdoor defense, we compare NPD with two variants NPD-UP and NPD-UU (see Table 4 in manuscript). It shows an obvious gap between NPD and NPD-UP, i.e., the average ASR 1.99% vs. 8.32%. It demonstrates that T-AP makes significant contribution to the success of NPD.\n\nHope above points could address your concern. Thanks again for your constructive comment."}, {"review_id": "VLYoprJmF3", "reviewer": "Reviewer_DJTs", "summary": "This work proposes a defense method against backdoor attacks. The proposed method involves inserting a trainable transformation layer inside a backdoor model while keeping other model parameters fixed that is supposed to purify the poisonous features while allowing benign features to pass without modification. ", "questions": "1)\tHow compatible is neural polarizer with other model architectures like inception and Densenet?\n2)\tHow is the defense performance against all2all attacks for attacks other than BadNets?\n", "limitations": "\nI have raised some concerns in the weaknesses section. Those are some possible limitations of the work. The authors should revise their limitation section.\n\n\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1)\tThe proposed transformation block constitutes a single 1x1 convolution and batch normalization layer which makes it easier to adopt in any model architecture.\n2)\tFaster training time since only the added layer is trained while keeping other model parameters fixed.\n3)\tThe proposed method provides good defense performance (low ASR) against various attacks.\n", "weaknesses": "1)\tTo filter out poisonous features from a poisonous sample, the method requires the creation of poisonous samples. While the authors claim to approximate the trigger, they utilize adversarial samples instead of poisoned ones, which are fundamentally different and not equivalent approximations.\n2)\tPrevious research [41] has utilized adversarial perturbation to address removing backdoor attacks. It is crucial to emphasize the key differentiation between [41] and the proposed NPD-TU method, aside from the difference in training the entire model versus solely training the added transformation layer.\n3)\tThe proposed method exhibits a decrease in benign accuracy, as indicated by the results presented in Table 2. This decline in accuracy is a common occurrence observed in adversarial training techniques.\n4)\tThe authors insert the transformation block before the third convolution layer of the fourth layer for PreAct-ResNet18. How is this design decision made, i.e., for a new model which might be potentially backdoored? How does the defender decide where to insert this polarizer block? \n5)\tHow does the proposed method perform under an adaptive attack, where the attacker knows the defense strategy? One approach to testing this could be training the backdoor model using adversarial samples. It would be interesting to see this result.\n6)\tHow does the proposed method affect the performance of a benign model? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your dedicated time reading our paper and providing us with your meticulous review. Your insightful questions and concerns are greatly appreciated. Please inform us if these responses effectively address all your inquiries or if there are additional questions you'd like to raise.\n\n**Q1. Utilize adversarial samples instead of poisoned ones, which are fundamentally different and not equivalent approximations.**\n\n**R1:** Thanks for this insightful concern. It is essentially the same as Q2 proposed by Reviewer qQYz. Due to the space limit, we would like to refer you to **R2 to Reviewer qQYz**.\n\n**Q2. Key differentiation between i-BAU [41] and NPD-TU.**\n\n**R2:** There are still three significant differences between NPD-TU and i-BAU:\n* **Formulation:** NPD-TU has a min-min formulation, while i-BAU has a min-max formulation. The reason is that NPD-TU computes targeted adversarial perturbation (T-AP), while i-BAU computes untargeted AP (U-AP). The benefit of T-AP is explained in the **first response**.\n* **Algorithm:** i-BAU adopts an implicit hypergradient algorithm, involving the inverse of Hessian matrix, which is very costly and difficult to scale to large image size. In contrast, NPD alternatively solves the inner and outer minimization using efficient PGD and SGD algorithm, respectively. The running time of NPD and i-BAU is presented in **Table 1**, which shows NPD is more efficient than i-BAU, especially on Tiny ImageNet.\n* **Performance**: As shown in Tables 1 and 2 in manuscript, i-BAU performs poorly on defending against several attacks (especially on Tiny ImageNet dataset). In contrast, our NPD-TU and NPD show better defense performance across all attacks.\n\n**Table 1: Running time (sec.) in comparison with state-of-the-art defenses with 2500 images on PreAct-ResNet18.**\n|Defense|FP|NAD|NC|ANP|i-BAU|NPD|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:\nCIFAR-10|1169.01|74.39|896.45|58.75|57.23|55.16\nTiny ImageNet|3357|351|42512|1692|887|332\n\n**Q3. Where to insert this polarizer block for a new backdoored model?**\n\n**R3:** Thanks. Due to space limit, we refer you to **R2 of Reviewer BztP** for more details, which contains an in-depth analysis and performance of NP insertion in various layers across VGG19, Inception-V3, and Densenet161 networks, respectively. Briefly, NPD is robust across different network structures. And, it remains effective and reliable when the NP is inserted into deep layers of the network.\n\n**Q4. Defend against adaptive attacks.**\n\n**R4:** Thanks for this constructive suggestion. We train the backdoored models with adversarial training (AT) to serve as the adaptive attack. The defense performance against the AT backdoored models is shown in **Table 2**.\n* **NPD still performs well on AT models**, while i-BAU performs poor. The possible reason is that i-BAU is essentially adversarial training, while NPD adopts dynamic targeted adversarial perturbation, which is different from AT.\n* **Compared to the defense against backdoored models with standard training** (see Table 1 in main manuscript), there is a slight ASR increase from 1.62% to 5.22%. However, AT models also sacrifices the ACC.\n\n**Table 2: Defense performance of NPD against adaptive attacks on CIFAR-10.**\n\n||Backdoored||i-BAU||NPD||\n-|:-:|:-:|:-:|:-:|:-:|:-:\n||ACC|ASR|ACC|ASR|ACC|ASR\nBlended|86.00|99.63|83.98|30.43|83.64|3.92\nInput-Aware|84.98|94.99|83.17|71.12|83.13|4.47\nLF|84.15|94.30|84.15|94.30|83.39|4.89\nSSBA|84.34|93.32|82.74|24.54|83.38|5.22\n\n**Q5. Influence on benign models.**\n\n**R5:** Our NPD defense on benign models trained on different datasets is presented in Table 3 ('PreAct' indicates Pre-ActResNet18 and 'VGG' indicates VGG19-BN). It shows that there is a slight influence on benign models.\n\n**Table 3: Defense performance on benign models.**\n||CIFAR-10||GTSRB||Tiny ImageNet||\n:-:|:-:|:-:|:-:|:-:|:-:|:-:\n||PreAct|VGG|PreAct|VGG|PreAct|VGG\nBenign Model|93.70|92.07|98.15|98.11|57.35|47.18\nAfter Defense|92.58|91.00|98.43|98.06|52.15|47.01\n\n**Q6. How compatible is neural polarizer with other model architectures like Inception and DenseNet?**\n\n**R6:** The evaluations of NPD and other SOTA defenses against different attacks across Inception-V3 and DenseNet161 are shown in **Table 4**. Our NPD shows superior performance across the two networks, demonstrating the robustness of NPD under different model architectures.\n\n**Table 4: Defense results in comparison with NC and i-BAU on Inception-V3 and DenseNet161.**\n\n|||Backdoored||NC||i-BAU||NPD||\n-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n|||ACC|ASR|ACC|ASR|ACC|ASR|ACC|ASR\nInception-V3|BadNets|90.68|94.50|89.87|1.21|82.41|0.52|91.69|4.70\n||Blended|93.73|99.82|93.73|99.82|83.63|8.84|93.71|0.01\n||Input-Aware|91.60|98.80|91.60|98.80|91.03|22.93|91.58|0.69\n||SSBA|93.33|98.30|91.56|50.73|84.74|1.92|92.23|1.13\nDenseNet161|BadNets|84.33|89.68|82.69|2.82|79.49|39.63|82.06|1.91\n||Blended|86.37|98.79|86.38|98.79|77.86|48.44|83.18|4.22\n||Input-Aware|84.46|94.41|84.45|94.41|81.96|17.14|83.44|1.56\n||SSBA|84.18|84.13|83.26|14.50|80.51|11.16|81.41|8.48\n\n**Q7. Defense performance against all2all attacks.**\n\n**R7:** Thanks for this constructive suggestion. The dynamic and sample-specific prediction strategy of target label enables NPD to defend against all2all attacks. We conduct a supplementary experiment on CIFAR-10 dataset. The low ASR in **Table 5** tells that NPD can successfully defend against all2all backdoor attacks, showing the adaptation and robustness of our method.\n\n**Table 5: Defense performance against all2all attacks on CIFAR-10 dataset with 5% benign data and 10% poisoning ratio on PreAct-ResNet18 (%).**\n||Backdoored||NPD||\n:-:|:-:|:-:|:-:|:-:\n||ACC|ASR|ACC|ASR\nBlended|91.59|83.50|91.07|6.24\nLF|86.60|78.55|90.10|2.10\nInput-Aware|91.91|84.80|90.53|4.88\nSSBA|91.30|85.04|91.14|1.29"}]

Author rebuttal:
# Common Response
We sincerely thank all reviewers for their time and constructive comments.

**Q1. Analysis of the mechanism and novelty of our neural polarizer defense (NPD) method.**

**R1：** We aim to present **a systematic analysis** about NPD's mechanism and novelty covering **three critical components**:

**1. Dynamic and sample-specific target label prediction strategy**
* **Definition and contrast with existing methods.** We estimate the target label for each training sample in inner-minimization step (See Line 177-178 in manuscript). **In contrast**, Neural Cleanse (NC) estimates a common target label for all samples.
* ***Target label prediction accuracy.*** In **Table 1**, we present accuracies of dynamic target label prediction on backdoor attacks at the first mini-batch. Most prediction accuracies are much higher than random guess (10\%). Although the predicted target label isn't 100% correct, it remains effective for backdoor removal, as analyzed later.
* **Effect & analysis.** There are two main advantages to the dynamic strategy:
  * **The predicted target label enables us to generate targeted adversarial perturbation (T-AP).** As analyzed later, T-AP is a better surrogate for the unknown trigger than the untargeted adversarial perturbation (U-AP), enhancing backdoor removal efficacy. Despite the partial accuracy of predicted target labels, they enable robust backdoor defense. Incorrectly predicted labels guide us to generate U-AP (aimed at other classes). Thus, the generated adversarial perturbations of all fine-tuning samples in NPD are a **mixture of T-AP and U-AP**.
  * **The sample-specific target label prediction is applicable to all2one and all2all attack settings.** In practice, defenders lack certainty about attack types (all2one or all2all), risking suboptimal defenses due to erroneous guesses or detection. Our sample-specific target label prediction avoids this risk. Thus, our method performs well against both all2one and all2all attacks (see **Table 2**).

**Table 1: Accuracy of dynamic target label prediction on CIFAR-10(%).**
|ATTACK|BadNets|Blended|Input-Aware|LF|SSBA|Trojan|Wanet|Avg|
:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:
ACC|86.33|33.05|72.66|43.75|37.11|28.91|84.77|55.22

**Table 2: Defense performance under different all2all attacks on CIFAR-10(%).**
||Backdoored||NPD||
:-:|:-:|:-:|:-:|:-:
||ACC|ASR|ACC|ASR
BadNets|91.89|74.42|91.41|0.89
Blended|91.59|83.50|91.07|6.24
Input-Aware|91.91|84.80|90.53|4.88
SSBA|91.30|85.04|91.14|1.29

**2. Targeted and sample-specific adversarial perturbation**
* **Definition and contrast with existing methods.** We generate T-AP for each training sample in the inner-minimization step (Line 178-179 and Eq. (5) in manuscript). **In contrast**, i-BAU generates U-AP. NPD's min-min formulation contrasts i-BAU's min-max one. Besides, NC adopted targeted universal adversarial perturbation in a local patch to approximate the unknown trigger.
* **Effect & analysis.**
  * **T-AP is a better surrogate to the unknown trigger than U-AP:**
    * **Intuitively**, T-AP set is a sub-region of U-AP set, and the unknown trigger is a particular T-AP. Thus, using the projected gradient descent (PGD) adversarial attack method, the generated T-AP is more likely to be close to the trigger than the generated U-AP.
    * **Visualization.** we visualize the trigger, as well as the generated T-AP and U-AP. As shown in **Fig. 1 in supplementary pdf**, the T-AP is more visually similar to the trigger.
    * **Quantitative distance.** We calculate $L_2$ distance in the latent space of backdoored model among the targeted adversarial example (AE), untargeted AE, and the poisoned sample with trigger. The distance between Targeted AE and poisoned sample 36.4 is much smaller than that between Untargeted AE and poisoned sample 93.84.

  * **T-AP & U-AP Mixture for Effective Defense.** As shown above, NPD generates a mixture of T-AP and U-AP. In contrast, in the variants NPD-TP (see Line 182-185 in manuscript) and NPD-UP (see Line 259), the generated perturbations are pure T-AP and U-AP, respectively. In Tables 1 and 2 of the main manuscript, there is slight gap between NPD and NPD-TP. In Table 4 of the main manuscript, there is an obvious gap between NPD and NPD-UP, i.e., the average ASR 1.99% vs. 8.32%. These two comparisons demonstrate that a mixture of T-AP and U-AP is enough to achieve good defense performance close to the pure T-AP, and the T-AP part makes significant contribution.

**3. Neural polarizer (NP)**
* **Definition and contrast with existing methods.** NP is a lightweight linear transformation layer (see Section 3.2 and Fig. 3 in manuscript). It's inserted into the backdoored model. Only its parameters are fine-tuned, while the parameters of the original backdoored model are fixed. **In contrast**, most existing fine-tuning based defense methods (e.g., i-BAU, NAD) don't modify the structure and fine-tune the whole model.
* **Effect & analysis.** Only fine-tuning NP's parameter yields two key benefits:
* **Data efficient.** Table 6 and Line 281-285 in manuscript showcase NPD, i-BAU, and ANP performance using few clean data (500, 250, 50 samples). NPD still performs well with few clean data while i-BAU performs much worse. Thus, NPD is much more data efficient.
* **Computationally efficient.** Table 7 in manuscript and the following **Table 3** show the running time of different defense methods on two datasets. It shows the computational efficiency of NPD.

**In summary, above three innovative components distinguish our NPD significantly from existing related methods, and make critical contributions to the superior defense performance of NPD on both effectiveness and efficiency**.

**Table 3: Runnign Time (sec.) in comparison with state-of-the-art defenses with 2500 images on PreAct-ResNet18.**
|Defense|FP|NAD|NC|ANP|i-BAU|NPD|
:-:|:-:|:-:|:-:|:-:|:-:|:-:
CIFAR-10|1169.01|74.39|896.45|58.75|57.23|55.16
Tiny ImageNet|3357|351|42512|1692|887|332

Meta-review: The paper proposes a lightweight and effective backdoor defense by inserting a trainable neural network layer. The layer can remove backdoor behavior by filtering out poisoned features. The method has been shown to be effective via experiments. Questions were raised regarding difference from previous methods and defense power against adaptive attacks. These concerns were addressed reasonably during rebuttal.
Decision: Accept (poster)