Title: Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Features
Venue: NeurIPS 2023
Accepted: yes
Keywords: Backdoor Defense

Backdoor Learning

Trustworthy AI
Abstract: Recent studies have demonstrated the susceptibility of deep neural networks to backdoor attacks. Given a backdoored model, its prediction of a poisoned sample with trigger will be dominated by the trigger information, though trigger information and benign information coexist. Inspired by the mechanism of the optical polarizer that a polarizer could pass light waves with particular polarizations while filtering light waves with other polarizations, we propose a novel backdoor defense method by inserting a learnable neural polarizer into the backdoored model as an intermediate layer, in order to purify the poisoned sample via filtering trigger information while maintaining benign information. The neural polarizer is instantiated as one lightweight linear transformation layer, which is learned through solving a well designed bi-level optimization problem, based on a limited clean dataset. Compared to other fine-tuning-based defense methods which often adjust all parameters of the backdoored model, the proposed method only needs to learn one additional layer, such that it is more efficient and requires less clean data. Extensive experiments demonstrate the effectiveness and efficiency of our method in removing backdoors across various neural network architectures and datasets, especially in the case of very limited clean data. Codes are available at \href{https://github.com/SCLBD/BackdoorBench}{https://github.com/SCLBD/BackdoorBench} (PyTorch) and \href{https://github.com/JulieCarlon/NPD-MindSpore}{https://github.com/JulieCarlon/NPD-MindSpore} (MindSpore).

Neural Polarizer: A Lightweight and Effective Backdoor Defense via Purifying Poisoned Features

Abstract

Recent studies have demonstrated the susceptibility of deep neural networks to backdoor attacks. Given a backdoored model, its prediction of a poisoned sample with trigger will be dominated by the trigger information, though trigger information and benign information coexist. Inspired by the mechanism of the optical polarizer that a polarizer could pass light waves with particular polarizations while filtering light waves with other polarizations, we propose a novel backdoor defense method by inserting a learnable neural polarizer into the backdoored model as an intermediate layer, in order to purify the poisoned sample via filtering trigger information while maintaining benign information. The neural polarizer is instantiated as one lightweight linear transformation layer, which is learned through solving a well designed bi-level optimization problem, based on a limited clean dataset. Compared to other fine-tuning-based defense methods which often adjust all parameters of the backdoored model, the proposed method only needs to learn one additional layer, such that it is more efficient and requires less clean data. Extensive experiments demonstrate the effectiveness and efficiency of our method in removing backdoors across various neural network architectures and datasets, especially in the case of very limited clean data. Codes are available at https://github.com/SCLBD/BackdoorBench (PyTorch) and https://github.com/JulieCarlon/NPD-MindSpore (MindSpore).

1 Introduction

Several studies have revealed the vulnerabilities of deep neural networks (DNNs) to various types of attacks [15, 17, 21, 22, 40], of which backdoor attacks [6, 9, 11, 36] are attracting increasing attention. In backdoor attacks, the adversary could produce a backdoored DNN model through manipulating the training dataset [5, 24] or the training process [20, 29], such that the backdoored model predicts any poisoned sample with particular triggers to the predetermined target label, while behaves normally on benign samples. Backdoor attacks can arise from various sources, such as training based on a poisoned dataset, or utilizing third-party platforms for model training, or downloading backdoored models from untrusted third-party providers. These scenarios significantly elevate the threat of backdoor attacks to DNNs' applications, and meanwhile highlight the importance of defending against backdoor attacks.

Several seminal backdoor defense methods have been developed, mainly including 1) in-training approaches, which aim to train a secure model based on a poisoned dataset through well-designed training algorithms or objective functions, such as DBD [14] and D-ST [4]. 2) post-training approaches, which aim to mitigate the backdoor effect from a backdoored model through adjusting the

Cat

Cat

Dog

Shared

learnable

)

fixed

)

Shared

Cat

…

…

Benign

Poisoned

…

…

Benign

Poisoned

model parameters ( e.g. , fine-tuning or pruning), usually based on a limited subset of clean training dataset, such as fine-pruning [23], ANP [43], or i-BAU [45]. This work focuses on the latter one. However, there are two limitations to existing post-training approaches. First, given very limited clean data, it is challenging to find a good checkpoint to simultaneously achieve backdoor mitigation and benign accuracy maintenance from the high-dimensional loss landscape of a complex model. Second, adjusting all parameters of a complex model is costly.

To tackle the above limitations, we propose a lightweight and effective post-training defense approach, which only learns one additional layer, while fixing all layers of the original backdoored model. It is inspired by the mechanism of the optical polarizer [44] that in a mixed light wave with diverse polarizations, only the light wave with some particular polarizations could pass the polarizer, while those with other polarizations are blocked (see Fig. 1-left). Correspondingly, by treating one poisoned sample as the mixture of trigger feature and benign feature, we define a neural polarizer and insert it into the backdoored model as one additional intermediate layer (see Fig. 1-right) in order to filter trigger feature and maintain benign feature, such that poisoned samples could be purified to mitigate backdoor effect, while benign samples are not significantly influenced.

In practice, to achieve an effective neural polarizer, it should be learned to weaken the correlation between the trigger and the target label while keeping the mapping from benign samples to their ground-truth labels. However, the defender only has a limited clean dataset, while neither trigger nor target label is accessible. To tackle it, we propose a bi-level optimization, where a samplespecific target label is dynamically estimated according to the output confidence, and the trigger is approximated by the targeted adversarial perturbation. Besides, in our experiments, the neural polarizer is implemented by a linear transformation ( i.e. , the combination of one 1 × 1 convolutional layer and one batch normalization layer). Consequently, it can be efficiently and effectively learned with very limited clean data to achieve good defense performance, which is verified by extensive experiments on various model architectures and datasets.

Our main contributions are three-fold. 1) We propose an innovative backdoor defense approach that only learns one additional linear transformation called neural polarizer while all parameters of the backdoored model are fixed, such that it just requires very low computational cost and very limited clean data. 2) A bi-level optimization problem and an effective learning algorithm are provided to optimize the parameter of the neural polarizer. 3) Extensive experimental results demonstrate the superiority of the proposed method on various networks and datasets, in terms of both efficiency and effectiveness.

2 Related work

Backdoor attacks. Traditional backdoor attacks are additive attacks that modify a small fraction of training samples by patching a pre-defined pattern and assigning them to targeted labels [8, 11]. These modified samples, along with the unaffected samples, constitute a poisoned dataset [11]. The model trained with this dataset will be implanted with backdoors that predict the targeted label when triggered by the injected patterns while maintaining normal behavior on clean samples [5, 50]. Recently, advanced attacks have considered more invisible trigger injection methods such as training an auto-encoder feature embedding or using a local transformation function [20, 29, 46]. To increase the stealthiness of attacks, clean-label attacks succeed by obfuscating image subject information and establishing a correlation between triggers and targeted labels without modifying the labels of poisoned samples [1, 2, 30].

Backdoor defense. Backdoor defense methods can be broadly categorized into training-stage defenses and post-processing defenses. Training-stage defenses assume access to a poisoned dataset for model training [10]. The different behaviors between the poisoned and clean samples can be leveraged to identify suspicious instances, such as sensitivity to transformation [4] and clustering phenomenon in feature space [14]. Most defense methods belong to post-processing defenses [39], which assume that the defender only has access to a suspicious DNN model and a few clean samples. Therefore, they must remove the backdoor threat with limited resources. There are mainly three types of defense strategies: trigger reversion methods try to recover the most possible triggers and utilize the potential triggers to fine-tune the model [35]; pruning-based methods aim at locating the neurons that are most related to backdoors and pruning them [23, 43, 47, 48]; and fine-tuning based defenses leverage clean samples to rectify the model [19, 45]. I-BAU [45] is most closely related to our method, which formulates a minimax optimization framework to train the network with samples under universal adversarial perturbations. However, our method differs from I-BAU in that our approach does not require training the entire network, and our perturbation generation mechanism is distinct. Other methods proposed for backdoor detection include Beatrix [26], which uses Gram matrices to identify poisoned samples; and AEVA [12], which detects backdoor models by adversarial extreme value analysis. In this study, we focus on post-processing defenses and primarily compare our method with state-of-the-art post-processing defenses [19, 23, 35].

3 Methodology

3.1 Basic settings

Notations. We consider a classification problem with K classes ( K ≥ 2 ). Let x ∈ X ⊂ R d be a d -dimensional input sample, and its ground truth label is denoted as y ∈ Y = { 1 , . . . , K } . Then, a L layers deep neural network f : X × W → R K parameterized by w ∈ W is defined as:

where f ( l ) w l is the function ( e.g. , convolution layer) with parameter w l in the l th layer with 1 ≤ l ≤ L . For simplicity, we denote f ( x ; w ) as f w ( x ) or f w . Given input x , the predicted label of x is given by arg max k f k ( x ; w ) , k = 1 , . . . , K , where f k ( x ; w ) is the logit of the k th class.

Threat model. We assume that the adversary could produce the backdoored model f w through manipulating training data or training process, such that f w performs well on benign sample x ( i.e. , f w ( x ) = y ), and predicts poisoned sample x ∆ = r ( x , ∆) to the target label T , with ∆ indicating the trigger and r ( · , · ) being the fusion function of x and ∆ . Considering that the adversary may set multiple targets, we use T i to denote the target label for x i .

Defender's goal. Assume that the defender has access to the backdoored model f w and a limited set of benign training data, denoted as D bn = { ( x i , y i ) } N i =1 . The defender's goal is to obtain a new model ˆ f based on f w and D bn , such that the backdoor effect will be mitigated and the benign performance is maintained in ˆ f .

3.2 Neural polarizer for DNNs

Neural polarizer. We propose the neural polarizer to purify poisoned sample in the feature space. Formally, it is instantiated as a lightweight linear transformation g θ , parameterized with θ . As shown in Fig. 1, g θ is inserted into the neural network f w at a specific immediate layer, to obtain a combined network f w , θ . For clarity, we denote f w , θ as ˆ f θ , since w is fixed.

A desired neural polarizer should have the following three properties:

Compatible with the neighboring layers : in other words, its input feature and its output activation must have the same shape. This requirement can be fulfilled through careful architectural design.

Filtering trigger features in poisoned samples : after the neural polarizer, the trigger features should be filtered, such that the backdoor is deactivated, i.e. , ˆ f θ ( x ∆ ) = T .

Preserving benign features in poisoned and benign samples : the neural polarizer should preserve benign features, such that ˆ f θ performs well on both poisoned and benign samples, i.e. , ˆ f θ ( x ∆ ) = ˆ f θ ( x ) = y .

Previous Layer

f

(

k

)

w

k

f

(

k

+1)

w

k

+1

Posterior Layer

C

k

×

H

k

×

W

k

1x1 Conv,

C

k

×

H

k

×

W

k

C

k

×

H

k

×

W

k

Neural

Polarizer

C

k

The first property could be easily satisfied by designing neural polarizer's architecture. For example, as illustrated in Fig. 2, given the input feature map with the shape C k × H k × W k , the neural polarizer is implemented by a convolution layer (Conv) with C k convolution filters of shape 1 × 1 , followed by a Batch Normalization (BN) layer. The Conv-BN block can be seen as a linear transformation layer. To satisfy the latter two properties, θ should be learned by solving some well designed optimization, of which the details are presented in Section 3.3.

3.3 Learning neural polarizer

3.3.1 Loss functions in oracle setting

To learn a good neural polarizer g θ , we first consider an oracle setting where the trigger ∆ and target label T are given. We present some loss functions to encourage g θ to satisfy the latter two properties mentioned above, as follows:

Loss for filtering trigger features in poisoned samples. Given trigger ∆ and target label T , filtering trigger features can be implemented by weakening the connection between ∆ and T with the following loss:

where s T ( x ∆ ; θ ) is the softmax probability of predicting x ∆ to label T by ˆ f θ . By reducing L asr , the attack success rate of the backdoor attack can be decreased.

Loss for maintaining benign features in poisoned samples. To purify the poisoned sample such that it can be classified to the true label, we leverage the boosted cross entropy defined in [37]:

̸

Loss for maintaining benign features in benign samples. To preserve benign features in benign sample, we adopt the widely used cross-entropy loss, i.e. , L bn ( x , y ; θ ) = L CE ( ˆ f θ ( x ) , y ) .

3.3.2 Approximation & optimization

Approximating T and ∆ . Note that as T and ∆ are inaccessible in both L asr and L bce , these two losses cannot be directly optimized. Thus, we have to approximate T and ∆ . In terms of

̸

<latexi sh

1_b

64="mVu+kHz

M

R

5

2T

pDLGIE

>A

c

S

N

F

r

f0

7d

Q

Wo

9

Y

J

3

y

OZ

C

U

8

P

g

n

v

q

X

j

w

K

<latexi sh

1_b

64="zGI2D

RU

7T

E

u8

>A

B

n

c

V

NS

3

r

/

9L

0

O

F

Q

oW

m

pZ

P

j

k

M

Y

H

+

d

f

5

K

v

<latexi sh

1_b

64="MqDzB

yo7QHW

JS

d

ZCg

k

>A

9

c

V

5T

N

E

2

K

r

j

I

Y

03

R

/

U

Pw

m

O

G

n

F

X

u

8

p

f

L

+

<latexi sh

1_b

64="LynF

M

QXzAH

3W

p

RqT75

>

C

c

ZD

S

N

I

Pv

o

U

K

2E

w

m0

k

d

J

O

GV

j

8

B

f

+

/

ur

Y

9

g

<latexi sh

1_b

64="LynF

M

QXzAH

3W

p

RqT75

>

C

c

ZD

S

N

I

Pv

o

U

K

2E

w

m0

k

d

J

O

GV

j

8

B

f

+

/

ur

Y

9

g

<latexi sh

1_b

64="LynF

M

QXzAH

3W

p

RqT75

>

C

c

ZD

S

N

I

Pv

o

U

K

2E

w

m0

k

d

J

O

GV

j

8

B

f

+

/

ur

Y

9

g

approximating T , although some methods have been developed to detect target class of backdoored models [12, 26], here we adopt a simple sample-specific and dynamic strategy:

̸

There are two main advantages of this strategy. First, the predicted target label enables us to generate targeted adversarial perturbation. As analyzed later, the targeted adversarial perturbation is a better surrogate for the unknown trigger than the untargeted adversarial perturbation. Second, the samplespecific target label prediction is applicable to both all2one (one trigger, one target class) and all2all (every class is target class) attack settings. In practice, defenders lack certainty about attack types (all2one, all2all, or multi-trigger multi-target), risking sub-optimal defenses due to erroneous guesses or detection. Our sample-specific target label prediction avoids this risk.

In terms of approximating ∆ , it is approximated by the targeted adversarial perturbation of f w , i.e. ,

where ∥ · ∥ p is the L p norm, ρ is the budget for perturbations.

Bi-level optimization problem. By employing the approximations of T and ∆ shown in Eq. (4) and (5), respectively, and substituting these approximations into the respective loss functions of Eq. (2) and (3), we introduce the following optimization problem to learn θ based on the clean data D bn :

̸

where λ 1 , λ 2 , λ 3 > 0 are hyper-parameters to adjust the importance of each loss function. This bi-level optimization is dubbed Neural Polarizer based backdoor Defense ( NPD ).

Variants. To comprehensively evaluate the performance of NPD, we also provide two variants, with the relaxation that if the target label T is known. One is that approximating the trigger by the targeted adversarial perturbation for each benign sample in Eq. (6), i.e. , ∆ ≈ δ ∗ i = arg min ∥ δ i ∥ p ≤ ρ L CE ( f ( x i + δ i ) , T ) , dubbed NPD-TP . The other is approximating the trigger by the targeted universal adversarial perturbation (TUAP) [25] for all training samples in fine-tuning, dubbed NPD-TU . We empirically found that the targeted adversarial perturbation is a better surrogate to the trigger than the untargeted ones.

Optimization algorithm. We proposed Algorithm 1 to solve the above optimization problem. Specifically, NPD solves problem (6) by alternatively updating the surrogate target label y ′ , the perturbation δ and θ as follows:

̸

Inner minimization: Given parameter θ of the neural polarizer, we first estimate the target label for sample x i by y ′ i = arg max k i = y i ˆ f k i ( x i ; θ ) . Then, the targeted Project Gradient Descent (PGD) [27] is employed to generate the perturbation δ ∗ i via Eq. (5).

Outer minimization: Given y ′ and δ ∗ for each sample in a batch, the θ can be updated by taking one stochastic gradient descent [3] (SGD) step w.r.t. the outer minimization objective in Eq. ( 6 ).

4 Experiments

4.1 Implementation details

Attack settings. We evaluate the proposed method on eight famous backdoor attacks, including BadNets [11] (BadNets-A2O and BadNets-A2A refer to attacking one target label and all labels, respectively), Blended attack (Blended) [5], Input-aware dynamic backdoor attack (Input-aware)[28], Low frequency attack (LF) [46], Sample-specific backdoor attack (SSBA) [20], Trojan backdoor

Algorithm 1 Neural Polarizer based Backdoor Defense (NPD)

1: Input: Training set D bn , backdoored model f w , neural polarizer g θ , learning rate η > 0 , perturbation bound ρ > 0 , norm p , hyper-parameters λ 1 , λ 2 , λ 3 > 0 , warm-up epochs T 0 , training epochs T , number of PGD steps T adv . 2: Output: Model ˆ f ( w , θ ) . 3: Initialize θ to be an identity function, fix w , and construct the composed network ˆ f ( w , θ ) . 4: Warm-up: Train ˆ f ( w , θ ) with L CE ( D bn ) for T 0 epochs. 5: for t = 0 , ..., T 1 do 6: for mini-batch B = { ( x i , y i ) } b i =1 ⊂ D bn do 7: Compute { y ′ i } b i =1 by Eq. (4); 8: Generate perturbations { ( δ i ) } b i =1 with ∥ δ i ∥ p ≤ ρ and { y ′ i } b i =1 by targeted PGD attack [27] via Eq. (5); 9: Update θ via outer minimization of Eq. (6) by SGD. 10: end for 11: end for 12: return Model ˆ f ( w , θ ) .

attack (Trojan) [24], and Warping-based poisoned networks (WaNet) [29]. We follow the default attack configuration as in BackdoorBench [42] for a fair comparison. The poisoning ratio is set to 10% in comparison with SOTA defenses. These attacks are conducted on three benchmark datasets: CIFAR-10 [16], Tiny ImageNet [18], and GTSRB [34]. We test all attacks on PreAct-ResNet18 [13] and VGG19-BN [33].

Defense settings. We compare the proposed methods with six SOTA backdoor defense methods, i.e. , Fine-pruning (FP) [23], NAD [19], NC [35], ANP [43], i-BAU [45], and EP [48]. All these defenses have access to 5% benign training samples. The training hyperparameters are adjusted based on BackdoorBench [42]. We evaluate the proposed method under two proposed scenarios and compare our NPD-TU, NPD-TP, and NPD with SOTA defenses. For the ablation study, we focus solely on NPD, which represents a more generalized scenario. We apply an l 2 norm constraint to the adversarial perturbations, with a perturbation bound of 3 for CIFAR-10 and GTSRB datasets, and 6 for Tiny ImageNet. We train the neural polarizer for 50 epochs with batch size 128 and learning rate 0 . 01 on each dataset and the transformation block is inserted before the third convolution layer of the fourth layer for PreAct-ResNet18. The loss hyper-parameters λ 1 , λ 2 , λ 3 are set to 1 , 0 . 4 , 0 . 4 for NPD, and 1 , 0 . 1 , 0 . 1 for NPD-TU and NPD-TP. More implementation details on SOTA attacks, defenses, and our methods can be found in Section B of Appendix .

Evaluation metric. In this work, we use clean ACCuracy (ACC), Attack Success Rate (ASR), and Defense Effectiveness Rating (DER) as evaluation metrics to assess the performance of different defenses. ACC represents the accuracy of clean samples while ASR measures the ratio of successfully misclassified backdoor samples to the target label. Defense Effectiveness Rating (DER ∈ [0 , 1] [49]) is a comprehensive measure that considers both ACC and ASR:

where ∆ ASR denotes the decrease of ASR after applying defense, and ∆ ACC denotes the drop in ACC following defense. Higher ACC, lower ASR, and higher DER indicate better defense performance. Note that in comparison with SOTA defenses, the one achieving the best performance is highlighted in boldface , while the second-best result is indicated by underlining. We provide PyTorch and MindSpore implementations of NPD.

4.2 Main results

Table 1 and Table 2 showcase the defense performance of the proposed method in comparison to six SOTA defense methods on CIFAR-10 and Tiny ImageNet. The following observations can be made:

Our methods show superior performance in terms of DER for almost all attacks compared to SOTA defenses. Conversely, FP and ANP excel in maintaining high ACC, but they struggle to eliminate backdoors in strong attacks like Blended and LF. NC's emphasis on minimal universal

adversarial perturbation renders it ineffective against sample-specific attacks and those utilizing large norm triggers. I-BAU shows similar performance in removing backdoors with an average DER of 92.53%, but it leads to a significant decrease in ACC, likely due to training the entire network by adversarial training.

Defense performance of NPD-TU and NPD-TP are better than NPD. When the target label is known, the model only needs to find perturbations for that specific label, simplifying trigger identification and unlearning. These two methods outperform NPD, except for WaNet, which is a transformation-based attack without visible triggers. Fully perturbing the network proves more effective than solely unlearning targeted triggers in WaNet.

NPD-TU is effective for trigger-additive attacks while NPD-TP is expert in defending against sample-specific attacks. It can be observed by comparing defense results on different attacks like Blended and SSBA on CIFAR-10. This demonstrates that the applicability of different strategies varies across different attack scenarios.

Defense performance is robust across all attacks on Tiny ImageNet. Similar to the results on CIFAR-10, our method outperforms other methods in terms of ASR and DER for all backdoor

attacks. Despite a slight decrease in ACC, NPD-TU achieves a remarkably good performance with ASR < 1 . 5% on average. NPD-TP and NPD perform best in removing backdoors while maintaining model utility.

In summary, our method outperforms other state-of-the-art approaches, showcasing the broad applicability of our proposed method across diverse datasets. Due to space limits, defending results on GTSRB dataset and VGG19-BN network can be found in Section C of Appendix .

4.3 Analysis

Effectiveness of each loss term. We conduct an ablation study to evaluate the contribution of each component of the loss function towards the overall performance on CIFAR-10 dataset. We separately investigate the first and second terms of loss L bce (see Eq. (3)), denoting them as L bce 1 and L bce 2 , respectively. Throughout the study, we keep the loss L bn consistent across all experiments, and the result is shown in Table 3. Notably, the loss l bce 1 plays a significant role in improving the overall performance while removing each component leads to a significant drop in defense in certain cases. This ablation study underscores the importance of each loss component in effectively mitigating different types of attacks.

Effectiveness of the targeted adversarial perturbations in NPD. To show the efficacy of the targeted adversarial perturbations in NPD (see Eq. (6)), we compare NPD with its two variants using two types of untargeted perturbation. We refer to adversarial perturbations generated by UAP and standard PGD without a targeted label as NPD-UU (untargeted universal adversarial perturbation) and NPD-UP (untargeted PGD), respectively. As shown in Table 4, NPD-UU and NPD-UP fail to remove backdoors in certain cases although NPD-UP obtains a higher ACC. This result shows the superiority of NPD in removing backdoors.

Performance of choosing different layers to insert the transformation layer. We evaluate the influence of choosing different layers to insert the transformation layer by inserting it after each convolution layer of PreAct-ResNet18 network on CIFAR-10 dataset. Figure 3 shows the defense performance under three attacks. The result shows that inserting the transformation layer into the shallower layers results in a decrease in accuracy. This is because even a slight perturbation in the shallow layers can cause significant instability in the final output. However, as the layer goes deeper, the features become more separable, resulting in better defense performance.

Defense effectiveness under different poisoning

ratios. To investigate the impact of poisoning ratios on defense performance, we conducted experiments on NPD with different poisoning ratios on the CIFAR-10 dataset. As presented in Table 5, there is a slight decrease in ACC as the poisoning ratio increases. Moreover, our approach exhibits a notably stable defense performance across a range of poisoning ratios.

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Layers

0

20

40

60

80

Percentage

BadNets

Blended

SSBA

ACC

ASR

Defense effectiveness under different clean ratios. We investigate the sensitivity of clean data on defense performance and compare our NPD with SOTA defenses. As shown in Table 6, NPD is less sensitive to the size of clean data among all the attacks and defenses. Even with only 50 samples, it still maintains acceptable performance. This result shows that our method exhibits minimal reliance on the number of training samples.

Defense against adaptive attacks. We evaluate the effectiveness of NPD against adaptive attacks. Consider the attacker knows the defense strategy and the attacker trains the backdoored model with adversarial training (AT). The defense performance against the AT backdoored model under different attacks is shown in Table 7. NPD still performs well on AT model, while i-BAU fails in these attacks. One possible reason is that i-BAU is essentially adversarial training, while NPD adopts dynamic targeted adversarial perturbation, which is different from AT. Compared to the defense against backdoored models with standard training (Table 1), there is a slight ASR increase (from 2.77% to 5.22% for SSBA, for example).

Defense against all to all attacks. To demonstrate the effectiveness of our dynamic and samplespecific target label prediction strategy, we conducted experiments involving all-to-all attacks for various backdoor attack methods. In these experiments, each class is treated as the target class. The result is presented in Table 8. As illustrated in the table, our NPD approach successfully eliminates all backdoors, proving its efficacy in multi-label cases. In contrast, i-BAU method fails in most cases.

Running time comparison. We measure the runtime of the defense methods on 2500 CIFAR-10 images with batch size 256 and PreAct-ResNet18. The experiments were conducted on one RTX 4090Ti GPU and the results are presented in Table 9. Among these methods, our proposed PND-UN achieves the fastest performance on CIFAR-10, requiring only 56 seconds. It should be noted that our method was trained for 50 epochs, while i-BAU was only trained for 5 epochs.

5 Conclusion

Inspired by the mechanism of optical polarizer, this work proposed a novel backdoor defense method by inserting a learnable neural polarizer as an intermediate layer of the backdoored model. We instantiated the neural polarizer as a lightweight linear transformation and it could be efficiently and effectively learned with limited clean samples to mitigate backdoor effect. To learn a desired neural polarizer, a bi-level optimization problem is proposed by filtering trigger features of poisoned samples while maintaining benign features of both poisoned and benign samples. Extensive experiments demonstrate the effectiveness of our method across all evaluated backdoor attacks and all other defense methods under various datasets and network architectures.

Limitations and future work. Although only limited clean data is needed for our method to achieve a remarkable defense performance, the accessibility of clean data is still an important limitation of the proposed method, which may restrict the application of our method. Therefore, a promising direction for future work is to further reduce the requirement of clean data by exploring data-free neural polarizer or learning neural polarizer based on poisoned training data.

Broader impacts. Backdoor attacks pose significant threats to the deployment of deep neural networks obtained from untrustworthy sources. This work has made a valuable contribution to the community with an efficient and effective backdoor defense strategy to ease the threat of existing backdoor attacks, even with a very limited set of clean samples, which ensures its practicality. Besides, the innovative defense strategy of learning additional lightweight layers, rather than adjusting the whole backdoored model, may inspire more researchers to develop more efficient and practical defense methods.

Structure of Appendix. Detailed algorithms of PGD and UAP, proposed NPD-TU and NPD-TP, along with a theoretical analysis in the kernel space are provided in Section A. Implementation details of experiments are introduced in Section B. More defense results in different datasets and networks are provided in Section C. Defense performance of different structures of neural polarizer is discussed in Section D. Additional experimental results and visualization are presented in Section E.

6 Acknowledgments

This work is supported by the National Natural Science Foundation of China under grant No. 62076213, Shenzhen Science and Technology Program under grant No. RCYX20210609103057050, No. ZDSYS20211021111415025, No. GXWD20201231105722002-20200901175001001, Shenzhen Science and Technology Program under grant No. JCYJ20210324120011032, CAAI-Huawei MindSpore Open Fund, and the Guangdong Provincial Key Laboratory of Big Data Computing, the Chinese University of Hong Kong, Shenzhen.

References

Jiawang Bai, Kuofeng Gao, Dihong Gong, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Hardly perceptible trojan attack against neural networks with bit flips. In European Conference on Computer Vision , pages 104-121. Springer, 2022. 3

Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In 2019 IEEE International Conference on Image Processing (ICIP) , pages 101-105. IEEE, 2019. 3

Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007 , pages 161-168. Curran Associates, Inc., 2007. 5

Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. Advances in Neural Information Processing Systems , 35:9727-9737, 2022. 1, 3

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv e-prints , pages arXiv-1712, 2017. 1, 3, 5, 7, 8, 9, 16, 17, 18, 19, 20

Ziyi Cheng, Baoyuan Wu, Zhenya Zhang, and Jianjun Zhao. Tat: Targeted backdoor attacks against visual object tracking. Pattern Recognition , 142:109629, 2023. 1

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009. 16

Kuofeng Gao, Jiawang Bai, Bin Chen, Dongxian Wu, and Shu-Tao Xia. Clean-label backdoor attack against deep hashing based retrieval. arXiv preprint arXiv:2109.08868 , 2021. 3

Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Shu-Tao Xia. Imperceptible and robust backdoor attack in 3d point cloud. arXiv preprint arXiv:2208.08052 , 2022. 1

Kuofeng Gao, Yang Bai, Jindong Gu, Yong Yang, and Shu-Tao Xia. Backdoor defense via adaptively splitting poisoned dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4005-4014, 2023. 3

Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access , 7:47230-47244, 2019. 1, 3, 5, 7, 8, 9, 16, 17, 18, 19, 20

Junfeng Guo, Ang Li, and Cong Liu. AEVA: black-box backdoor detection using adversarial extreme value analysis. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. 3, 5

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 630-645. Springer, 2016. 6, 16

Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. 1, 3

Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning , pages 2137-2146. PMLR, 2018. 1

Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. 6, 16, 19

Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security , pages 99-112. Chapman and Hall/CRC, 2018. 1, 14

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015. 6, 16

Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR 2021 . OpenReview.net, 2021. 3, 6, 7, 10, 18

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 16443-16452. IEEE, 2021. 1, 3, 5, 7, 8, 9, 16, 18, 19, 20

Siyuan Liang, Xingxing Wei, Siyuan Yao, and Xiaochun Cao. Efficient adversarial attacks for visual object tracking. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16 , pages 34-50. Springer, 2020. 1

Siyuan Liang, Baoyuan Wu, Yanbo Fan, Xingxing Wei, and Xiaochun Cao. Parallel rectangle flip attack: A query-based black-box attack against object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7697-7707, October 2021. 1

Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In Research in Attacks, Intrusions, and Defenses: 21st International Symposium, RAID 2018, Heraklion, Crete, Greece, September 10-12, 2018, Proceedings 21 , pages 273-294. Springer, 2018. 2, 3, 6, 7, 10, 18

Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-221, 2018 . The Internet Society, 2018. 1, 6, 7, 8, 16, 18, 19

Zhiyun Lu, Wei Han, Yu Zhang, and Liangliang Cao. Exploring targeted universal adversarial perturbations to end-to-end ASR models. In Hynek Hermansky, Honza Cernocký, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlícek, editors, Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021 , pages 3460-3464. ISCA, 2021. 5, 14, 15

Wanlun Ma, Derui Wang, Ruoxi Sun, Minhui Xue, Sheng Wen, and Yang Xiang. The" beatrix'resurrections: Robust backdoor detection via gram matrices. NDSS Symposium , 2022. 3, 5

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. 5, 6, 14

Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. 5, 7, 9, 16, 18, 19, 20

Tuan Anh Nguyen and Anh Tuan Tran. Wanet - imperceptible warping-based backdoor attack. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. 1, 3, 6, 7, 8, 9, 16, 18, 19

Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 6106-6116, 2018. 3

Rui Shao, Pramuditha Perera, Pong C Yuen, and Vishal M Patel. Open-set adversarial defense. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16 , pages 682-698. Springer, 2020. 17, 19

Rui Shao, Pramuditha Perera, Pong C Yuen, and Vishal M Patel. Open-set adversarial defense with clean-adversarial mutual learning. International Journal of Computer Vision , 130(4):1070-1087, 2022. 18, 19

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. 6, 16

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks , pages 1453-1460. IEEE, 2011. 6, 16

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP) , pages 707-723. IEEE, 2019. 3, 6, 7, 10, 18, 19

Ruotong Wang, Hongrui Chen, Zihao Zhu, Li Liu, Yong Zhang, Yanbo Fan, and Baoyuan Wu. Robust backdoor attack with visible, semantic, sample-specific, and compatible triggers. arXiv preprint arXiv:2306.00816 , 2023. 1

Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. 4

Shaokui Wei, Jiayin Liu, Bing Li, and Hongyuan Zha. Mean parity fair regression in rkhs. In International Conference on Artificial Intelligence and Statistics , pages 4602-4628. PMLR, 2023. 15, 16

Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. arXiv preprint arXiv:2307.10562 , 2023. 3

Xingxing Wei, Siyuan Liang, Ning Chen, and Xiaochun Cao. Transferable adversarial attacks for image and video object detection. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 954-960, 2019. 1

Wikipedia contributors. Polarizer - Wikipedia, the free encyclopedia, 2023. https://en. wikipedia.org/wiki/Polarizer ; accessed 10-May-2023. 2

Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022. 6, 16, 17

Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 16913-16925, 2021. 2, 3, 6, 7, 9, 10, 17, 18

Hao Xiong, Cui Kong, Xiaoxue Yang, and Ying Wu. Optical polarizer based on the mechanical effect of light. Optics Letters , 41(18):4316-4319, 2016. 2

Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. 2, 3, 6, 7, 9, 10, 18, 19

Yi Zeng, Won Park, Z. Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 16453-16461. IEEE, 2021. 3, 5, 7, 8, 9, 16, 18, 19

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V , pages 175-191. Springer, 2022. 3

Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. Advances in Neural Information Processing Systems , 35:18667-18680, 2022. 3, 6, 7, 10, 17, 18

Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 4466-4477, October 2023. 6

Zihao Zhu, Mingda Zhang, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Boosting backdoor attack with a learnable poisoning sample selection strategy. arXiv preprint arXiv:2307.07328 , 2023. 3

Appendix

This appendix is organized as follows.

In Section A, we first provide detailed algorithms of the proposed NPD-TU and NPD-TP, along with an introduction to the project gradient descent (PGD) method and UAP method. Then we present an example of backdoor attacks in the kernel space and provide a theoretical analysis of the existence of linear transformation to mitigate backdoors.

In Section B, we introduce the implementation details, including dataset introduction, implementation details of compared attacks, defenses, and our proposed methods.

In Section C, we display the defense results in comparison with state-of-the-art (SOTA) defenses on GTSRB dataset, as well as those on CIFAR-10 with the VGG19-BN network.

In Section D, we discuss different structures of neural polarizers on the defense performance.

In Section E, we present additional experimental results, including defense experiments conducted on the CIFAR-100 dataset, defense performance across various model architectures, and visualization experiments.

A Theoretical analysis and algorithm

A.1 Introduction to PGD algorithm and UAP

In this section, we provide an introduction to PGD attack [27] and universal adversarial perturbation (UAP) attack [25].

PGD. Projected Gradient Descent (PGD) is a multi-step variant of Fast Gradient Sign Method (FGSM [17], which is projected gradient descent on the negative loss function, i.e. , given an input ( x , y ) and a model ˆ f θ , the adversarial example is calculated as follows,

where Π x + S denotes the projection on to the ρ -ball with L p norm: ∥ x n +1 -x ∥ p ≤ ρ , and x 0 = x , n = 0 , · · · , N -1 . α is the step size. However, in Section 3 of the main script, we use a confidenceguided targeted PGD approach to obtain the perturbation. Thus, the adversarial example is updated as follows,

where T is the target label for x . We denote the adversarial perturbation as follows for convenience:

UAP. The main difference between UAP and PGD is that UAP computes a universal adversarial perturbation for the whole training samples; thus the adversarial example for targeted UAP is updated as follows,

where T is the target label for x , and b is the number of samples in one training batch. Eq (10) means all the samples in one batch share the sample gradient to update the perturbation.

A.2 Detailed algorithms of NPD-TU and NPD-TP

In the main script, we introduce the algorithm for NPD. Here, we provide the detailed algorithms for NPD-TU and NPD-TP as in Alg 2 and 3.

Algorithm 2 N eural P olarizer based Backdoor D efense of T argeted U AP (NPD-TU)

1: Input: Training set D bn , backdoored model f w , neural polarizer g θ , target label T , inner steps N , learning rate η > 0 , perturbation bound ρ > 0 , norm p , hyper-parameters λ 1 , λ 2 , λ 3 > 0 , warm-up epochs T 0 , training epochs T .

2: Output: Model ˆ f ( w , θ ) .

3: Initialize θ to be an identity function, fix w , and construct the composed network ˆ f ( w , θ ) .

4: Warm-up: Train ˆ f ( w , θ ) with L CE ( D bn ) for T 0 epochs.

6: for mini-batch B = { ( x i , y i ) } b i =1 ⊂ D bn do

5: for t = 0 , ..., T 1 do

7: for n = 0 , ..., N -1 do

8: Compute adversarial perturbation δ with ∥ δ ∥ p ≤ ρ and target label T by targeted UAP attack [25] via Eq. (10);

9: end for

10: Update θ via outer minimization of Eq. (5) in the main script by SGD.

11: end for

12: end for

13: return Model ˆ f ( w , θ ) .

Algorithm 3 N eural P olarizer based Backdoor D efense of T argeted P GD (NPD-TP)

1: Input: Training set D bn , backdoored model f w , neural polarizer g θ , target label T , inner steps N , learning rate η > 0 , perturbation bound ρ > 0 , norm p , hyper-parameters λ 1 , λ 2 , λ 3 > 0 , warm-up epochs T 0 , training epochs T .

2: Output: Model ˆ f ( w , θ ) .

3: Initialize θ to be an identity function, fix w , and construct the composed network ˆ f ( w , θ ) .

4: Warm-up: Train ˆ f ( w , θ ) with L CE ( D bn ) for T 0 epochs.

6: for mini-batch B = { ( x i , y i ) } b i =1 ⊂ D bn do

5: for t = 0 , ..., T 1 do

7: for n = 0 , ..., N -1 do

8: Compute adversarial perturbations { ( δ i ) } b i =1 with ∥ δ i ∥ p ≤ ρ and target label T by targeted UAP attack [25] via Eq. (8);

9: end for

10: Update θ via outer minimization of Eq. (5) in the main script by SGD.

11: end for

12: end for

13: return Model ˆ f ( w , θ ) .

A.3 Theoretical analysis under kernel space

In this section, we demonstrate the existence of linear transformation in reproducing kernel Hilbert space (RKHS), a very successful tool in various areas of machine learning. For simplicity, we consider a binary classifier h ( · , m ) : X × { 0 , 1 } → R in a reproducing kernel Hilbert space H XM generated by kernel κ XM , where m is a binary poison indicator, i.e. , m = 1 means adding trigger to the input and m = 0 means not adding trigger. Denote the feature of ( x , m ) in kernel space by ϕ XM ( x , m ) . Given a dataset D and a distribution of poison indicator M , the poisoned classifier is optimized by minimizing a mean squared error loss:

Then, by the theorem in [ 38 ] which shows that there exists a subspace of H XM in which a functin h satisfies Cov ( h ( x , m ) , m ) = 0 , we provide the following lemma:

̸

Lemma 1. Assume that P ( m = 0) ∈ (0 , 1) and ϕ XM ( x i , m ) = ϕ XM ( x j , m ) if x i = x j . Given a poisoned model h bd trained by minimizing (11), there exists a non-trivial linear projection operator P such that

̸

where ˆ ϕ XM ( x , m ) = Pϕ XM ( x , m ) is the projected feature of ϕ XM ( x , m ) and ⟨· , ·⟩ XM is the inner product in H XM .

̸

Proof. To prove Lemma 1, we first recall that [ 38 ] says that the projection operator can be constructed by the eigenfunctions of the operator Σ ( XM ) M Σ M ( XM ) where Σ ( XM ) M is the Covariance operator between ϕ XM ( x , m ) and ϕ M ( m ) for x ∈ D and m ∈ M . Then, under the assumption that P ( m = 0) ∈ (0 , 1) and ϕ XM ( x i , m ) = ϕ XM ( x j , m ) if x i = x j , the Covariance operator Σ ( XM ) M is a non-zero operator by its definition and the projection operator generated by Σ ( XM ) M is nontrivial.

̸

The first assumption in Lemma 1 requires that the backdoored classifier h bd is trained on a partially poisoned dataset to preserve its accuracy on clean samples. The second assumption in Lemma 1 requires that the feature of a poisoned sample retains some information of the original clean sample and is not completely overwritten by the trigger. Under these conditions, the Lemma 1 states that there exists a non-trivial projection operator such that the h bd 's prediction for a transformed feature Pϕ XM ( x , m ) is uncorrelated with poisoning x or not. Therefore, such a linear projection operator breaks the correlation between m and the prediction of x without modifying h bd , indicating the existence of a neural polarizer in H XM . Although a complex neural network for multi-class classification problems is usually beyond H XM , Lemma 1 provides insights into the existence of an effective neural polarizer.

B More implementation details

In this section, we present the implementation details, which include an introduction to the datasets used, the implementation specifics of the attacks and defenses compared, as well as our proposed methods. All experiments are run on one RTX 3090Ti GPU and are repeated over five runs with different random seeds.

Datasets. We evaluate the effectiveness of our method using three datasets: CIFAR-10 [16], Tiny ImageNet [18], and GTSRB [34], following the benchmarks established in [42].

CIFAR-10 comprises 60,000 images distributed across ten classes. The training set consists of 5,000 images per class, while the test set contains 1,000 images per class. Each image in the CIFAR-10 dataset has dimensions of 32 × 32 pixels.

Tiny ImageNet is a subset of the larger ImageNet dataset [7]. It encompasses 100,000 training samples and 10,000 testing samples across 200 classes. The images in Tiny ImageNet have dimensions of 64 × 64 pixels.

GTSRB consists of 39,209 training images and 12,630 testing images, categorized into 43 classes. The image size for each sample in the GTSRB dataset is 32 × 32 pixels.

Models. We evaluate the performance of our method in conjunction with two variations on the PreActResNet18 [13] and VGG19-BN [33] networks. Furthermore, we compare our method with SOTA defense methods on three datasets and the two networks, utilizing a 10% poisoning ratio and a 5% ratio of clean samples for defense.

Attack details. We introduce more details about the backdoor attack implementation here. We follow BackdoorBench to categorize the eight backdoor attacks according to various criteria as in Table 10.

For BadNets-A2O and BadNets-A2A [11], we patch a 3 × 3 white square in the lower right corner of the images for CIFAR-10 and GTSRB datasets, and 6 × 6 white square for Tiny ImageNet dataset. For Blended [5], we blend the poisoned samples with a Hello-Ketty image which has the same size as the poisoned samples, and the blended ratio is set to 0 . 2 .

Defense details. The seven SOTA defense methods can be categorized into two types based on the available resources for the defender. EP [48] is a data-free method that assumes the defender only has access to a backdoored model without any clean data. On the other hand, the remaining methods assume that the defender can obtain a subset of clean samples along with a backdoored model, which aligns with our scenario. In our experiments, a learning rate of 0.01 and a batch size of 256 were used for all SOTA defense methods based on fine-tuning. For ANP [43], the pruning threshold was set to 0.4, as we found that the recommended threshold of 0.2 was inadequate for removing backdoors effectively. Regarding the fine-tuning based SOTA methods, the training epochs were set to 100 for CIFAR-10 and Tiny ImageNet, and 50 for the GTSRB dataset. All other settings remained consistent with those specified in BackdoorBench [42].

In our NPD method, we used a learning rate of 0.01 with a weight decay of 0 . 0005 and a momentum of 0.9. The training process consisted of 50 epochs for all three datasets, with a warm-up period of 5 epochs. During each epoch, the PGD algorithm was employed to search for adversarial perturbations in each batch. The PGD algorithm utilized a learning rate of 0.1. To constrain the adversarial perturbations on the CIFAR-10 dataset, we applied an l 2 norm with a bound of 3.

C Defense results in comparison to SOTA defenses on GTSRB and VGG19-BN

To further investigate the generalization performance of our NPD on different datasets and networks, we conducted an analysis on the GTSBR dataset and CIFAR-10 on VGG19-BN network. We compared the performance of NPD with various defense methods against different attacks, and the results are presented in Table 11 and Table 12.

As shown in Table 11, our experimental findings consistently demonstrate the superiority of our proposed methods over state-of-the-art defenses in terms of DER and ASR across almost all attacks, except for BadNet and Trojan. Notably, NAD exhibits the best performance against the Trojan attack, which can be attributed to the strong single trigger pattern adopted by Trojan. NAD effectively reverses the trigger through optimization, making it easier to detect and mitigate the Trojan attack. Furthermore, while NPD-TU and NPD-TP exhibit superiority in terms of a low ASR, NPD demonstrates a comparable trade-off between ACC and ASR. On average, NPD achieves a DER of 96 . 76% , which is 6 . 26% higher than the best SOTA method, i-BAU. The higher DER achieved by NPD on the remaining attacks further validates the effectiveness of our methods in detecting and mitigating various backdoor attacks compared to existing defenses.

Regarding the performance of the VGG19-BN network, as shown in Table 12, NPD-TU and NPD outperform all other defense methods, underscoring the effectiveness of our proposed defense methods in removing backdoor attacks. However, NPD-TU fails to defend against the Input-Aware attack, which is similar to the performance of i-BAU defense. This highlights the importance of incorporating adversarial attacks into the defense strategy, as UAP does not always guarantee successful defense. Nonetheless, our proposed method exhibits comparable and stable performance across almost all attacks.

In summary, our analysis of the GTSBR dataset and VGG19-BN network showcases the robust generalization performance of NPD against various backdoor attacks. The results underscore the effectiveness of our proposed defense methods, especially NPD-TP and NPD, in removing backdoor attacks.

D Different structures of neural polarizers on the defense performance

In this section, we demonstrate the effectiveness of our Conv-BN structure in implementing the neural polarizer for defense against backdoors. We compare it with two Conv-BN structures, a Linear transformation layer, and two denoising layers used in two adversarial defense works: OSAD [31]

and OSDN [32]. For the denoising structures, we found that inserting four denoising layers in all four blocks of the backdoored model as in OSAD and OSDN leads to the collapse of the entire network. Hence, we adopt a single denoising layer inserted into only one layer of the backdoored model as NPD. This limitation arises due to the scarcity of training samples available for fine-tuning in the defense scenario. As for the Linear layer, it means we use a linear layer with weights and biases that have the same size as the latent features. Consequently, this structure has fewer parameters compared to ours. We conducted experiments on the CIFAR-10 dataset using the PreAct-ResNet18 network, and the other experimental setup aligns with our method. The defense performance results are shown in Table 13.

From the table, we observe that both OSAD and OSDN successfully maintain performance on clean data. However, the ASR experiences a slight drop in most cases, possibly due to the fact that OSAD and OSDN adopt a residual structure where backdoors cannot be completely eliminated. The two Conv-BN layers structure has a high probability of collapsing directly, which may be attributed to the

insufficient training data for such a multi-layer structure. The Linear structure shows lower ACC and higher ASR. This could be because fewer parameters are not sufficient to remove backdoors, and a random initialization of the inserted layer leads to a decrease in final accuracy. This experiment demonstrates the superiority of the designed Conv-BN structure in defending against backdoors.

E Additional experimental results

E.1 Defense experiments on CIFAR-100 dataset

To show the effectiveness of NPD method on classification tasks with more labels, we complement experiment on CIFAR-100 dataset [16], which has 100 classes in total and each class has 500 images for training and 100 images for testing. As shown in Table 14, the average ACC and ASR of NPD are 65.49%,1.96%, which indicates its competitiveness in comparison to existing defense approaches and showcases its efficacy in more classes scenarios.

E.2 Defense experiments across various model architectures

In this part, we discuss the selection of the layer to insert the neural polarizer (NP) in a new model and the performance of NPD under different model architectures. In the manuscript, we have investigated the impact of NP insertion in different layers on the PreActResNet18 network. Our empirical study demonstrates that inserting NP in deeper layers effectively removes backdoors, while inserting it in shallower layers may lead to a decrease in benign accuracy and instability. To further validate this finding and show the effectiveness and robustness of NPD across different model architectures, we conducted additional experiments on VGG19, Inception-V3, and Densenet161 networks. As presented in Table 15, inserting NP into the last three layers achieves remarkable performance (high ACC and low ASR), while a decrease in accuracy occurs when shallow layers are selected. This experiment shows that NPD is robust across different model architectures.

E.3 Visualization of targeted adversarial perturbation

We empirically establish that targeted adversarial perturbation (T-AP) is a better surrogate to the trigger compared to untargeted adversarial perturbation (U-AP), as demonstrated through quantitative experiments in the manuscript. It could also be explained from the following aspects: From an intuitive standpoint, T-AP can be regarded as a subregion of U-AP, and the unknown trigger is a specific instance of T-AP. Consequently, employing the projected gradient descent (PGD) adversarial attack method, the generated T-AP is more likely to exhibit proximity to the trigger when contrasted with the generated U-AP. Moreover, we visualize the trigger, as well as the generated T-AP and U-AP. As depicted in Fig. 4, the T-AP is more visually similar to the trigger. To reinforce our argument quantitatively, we calculate the L 2 distance in the latent space of the backdoored model among the targeted adversarial example (AE), untargeted AE, and the poisoned sample with the trigger. The distance between targeted AE and poisoned sample 36.4 is substantially smaller than that between Untargeted AE and poisoned sample 93.84. This supplementary experiment reaffirms our contention that T-AP serves as a more faithful surrogate to the concealed trigger in comparison to U-AP.

E.4 Grad-CAM Visualization

We employ Grad-CAM figures to visualize the effects of our NPD on BadNets, Blended, and Trojan attacks, given that these three attack methods employ visible triggers. As illustrated in Figures 5 to 7, Neural Polarizer (NP) effectively redirects the network's attention away from the triggers and toward the subject of figures. This visual evidence substantiates the successful removal of backdoors by NP in backdoored models.

Trigger.                Targeted pert.        Untargeted pert.

Poisoned sample

Targeted AE

Untargeted AE

Feature maps

36.4

93.84

𝑳

𝟐

distance of feature maps between AEs and poisoned samples

Poisoned

samples

Before

defense

After

defense

Benign

samples

Poisoned

samples

Before

defense

After

defense

Benign

samples

Poisoned

samples

Before

defense

After

defense

Benign

samples

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a defense method against backdoor attacks. The proposed method involves inserting a trainable transformation layer inside a backdoor model while keeping other model parameters fixed that is supposed to purify the poisonous features while allowing benign features to pass without modification.\n\nSTRENGTHS:\n1)\tThe proposed transformation block constitutes a single 1x1 convolution and batch normalization layer which makes it easier to adopt in any model architecture.\n2)\tFaster training time since only the added layer is trained while keeping other model parameters fixed.\n3)\tThe proposed method provides good defense performance (low ASR) against various attacks.\n\nWEAKNESSES:\n1)\tTo filter out poisonous features from a poisonous sample, the method requires the creation of poisonous samples. While the authors claim to approximate the trigger, they utilize adversarial samples instead of poisoned ones, which are fundamentally different and not equivalent approximations.\n2)\tPrevious research [41] has utilized adversarial perturbation to address removing backdoor attacks. It is crucial to emphasize the key differentiation between [41] and the proposed NPD-TU method, aside from the difference in training the entire model versus solely training the added transformation layer.\n3)\tThe proposed method exhibits a decrease in benign accuracy, as indicated by the results presented in Table 2. This decline in accuracy is a common occurrence observed in adversarial training techniques.\n4)\tThe authors insert the transformation block before the third convolution layer of the fourth layer for PreAct-ResNet18. How is this design decision made, i.e., for a new model which might be potentially backdoored? How does the defender decide where to insert this polarizer block? \n5)\tHow does the proposed method perform under an adaptive attack, where the attacker knows the defense strategy? One approach to testing this could be training the backdoor model using adversarial samples. It would be interesting to see this result.\n6)\tHow does the proposed method affect the performance of a benign model?\n\nQUESTIONS:\n1)\tHow compatible is neural polarizer with other model architectures like inception and Densenet?\n2)\tHow is the defense performance against all2all attacks for attacks other than BadNets?\n\nLIMITATIONS:\nI have raised some concerns in the weaknesses section. Those are some possible limitations of the work. The authors should revise their limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper proposes propose a backdoor defense method by inserting a learnable neural polarizer into the backdoored model as
an intermediate layer, in order to purify the poisoned sample via filtering trigger information while maintaining benign information. To more effectively remove backdoor, this paper leverages the reverse trigger and target label to remove backdoor.

STRENGTHS:
1. This paper is well written and technically sound.
2. The experiments are sufficient, and its performance achieves SOTA

WEAKNESSES:
1. Neural Polarizer needs to reverse the backdoor trigger and the target label, which is similar to Neural Cleanse. Furthermore, former papers demonstrate that trigger reverse mainly works well on static trigger such as BadNets. The results on sample-specific trigger are bad and the results of reverse trigger and target label are mostly wrong. Thus, the scope where Neural Polarizer can be used is limited.
2. The novelty is limited. Because trigger reversed has been proposed before, the contribution of this paper is to finetune the model with the reversed trigger on Neural Polarizer layer.

QUESTIONS:
Can you explain why the method works with the reversed trigger which may not be correctly reversed?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
The paper proposes a lightweight and effective backdoor defense by inserting a trainable neural layer block. Without modifying original backdoor model, the proposed method can remove backdoor behaviors by filtering poisoned features via the trainable neural block. The authors conduct sufficient experiments to demonstrate the effectiveness of proposed method.

STRENGTHS:
1. The proposed method is simple but effective. The inserted neural block only includes one convolution layer followed by a BN layer. Training this layer block is not time-consuming.
2. The experiments are very sufficient. The comparison experiments are conducted with six backdoor defenses against seven backdoor attacks on three datasets. Analysis experiments are also conducted including the effectiveness of losses, poisoning ratios and clean ratios.

WEAKNESSES:
1. The paper does not introduce how to choose different layers to insert the neural polarizer. Since the proposed method largely depends on network architectures. Please also provide more results using more network architectures e.g. vgg.

QUESTIONS:
1. Does this proposed method work for a network without batch normalization layer?
2. Please explain more about how to choose layers to insert neural polarizer.
3. Could the authors visualize features before and after neural polarizer to qualitatively analyze the effectiveness (e.g. using GradCam)?

LIMITATIONS:
The authors have addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel defense method to filter trigger information from poisoned samples. It inserts a learnable intermediate layer (called neural polarizer) into the backdoored model, and proposes bi-level optimization solution to approximate perturbation and target label. Experimental results demonstrate the effectiveness of Neural Polarizer over other defense baselines.\n\nSTRENGTHS:\n1.The paper is well-written and easy to follow.\n\n2.The proposed Nueral Polarizer is interesting.\n\n3.The experiments are comprehensive.\n\nWEAKNESSES:\n1.The paper proposes NPD (Neural Polarizer based backdoor Defense) as a normal setting. It also proposes NPD-TP (assume the target label is known) and NPD-TU (assume having full access of benign samples). The NPD-TP and NPD-TU relax the limitation, and is kind of unfair compared to other defense baselines. However, based on Table 1 and Table 2, most of best performance comes from NPD-TP/NPD-TU, while NPD does not perform very well under some situations.\n\nMore specific, In Table 1, defense on CIFAR-10: NPD does not always perform the best. For example, ASR under WaNet is 0.80 (larger than WaNet with FP and NAD), ASR under Trojan is 6.51 (larger than Trojan with i-BAU and EP). Similar phenomenon happens in Table 2: defense on Tiny ImageNet.\n\nQUESTIONS:\n1.Regarding approximating perturbation $\\Delta$ and target label T (eq.5): in inner minimization, the algorithm first estimates the target label, then applies PGD to generate perturbations. I assume this step is very important, because it decides the outer minimization. That are the estimated perturbations look like? Are they similar with the real triggers, or they are just the adversarial perturbations? Are there any experiments/visualizations to elaborate that?\n\n2.How to choose which layer to insert the neural polarizer? Does your method robust to the choice of layer?\n\n3.What if there are more labels. For example, CIFAR-100 or even more labels. Does the proposed method still work?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-20"}