SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Our methods show superior performance in terms of DER for almost all attacks compared to SOTA defenses. Conversely, FP and ANP excel in maintaining high ACC, but they struggle to eliminate backdoors in strong attacks like Blended and LF. NC's emphasis on minimal universal
Table 1: Comparison with the state-of-the-art defenses on CIFAR-10 dataset with  $5\%$  benign data and  $10\%$  poison ratio on PreAct-ResNet18  $(\%)$ .
ATTACK	Backdoored	FP [23]	NAD [19]	NC [35]	ANP [43]
ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER
BadNets-A2O [11]	91.82	93.79	N/A	91.77	0.84	96.45	88.82	1.96	94.42	90.27	1.62	95.31	91.65	3.83	94.89
BadNets-A2A [11]	91.89	74.42	N/A	92.05	1.31	86.56	90.73	1.61	85.82	89.79	1.11	85.60	92.33	2.56	85.93
Blended [5]	93.69	99.76	N/A	92.74	10.17	94.32	92.25	47.64	75.34	93.69	99.76	50.00	93.45	47.14	76.19
Input-Aware [28]	94.03	98.35	N/A	94.05	1.62	98.36	94.08	0.92	98.71	93.84	10.48	93.84	94.06	1.57	98.39
LF [46]	93.01	99.06	N/A	92.05	21.32	88.39	91.72	75.47	61.15	93.01	99.06	50.00	92.53	26.38	86.10
SSBA [20]	92.88	97.07	N/A	92.21	20.27	88.06	92.15	70.77	62.78	92.88	97.07	50.00	92.02	16.18	90.01
Trojan [24]	93.47	99.99	N/A	92.24	67.73	65.51	92.18	5.77	96.47	91.85	51.03	73.67	92.71	84.82	57.20
WaNet [29]	92.80	98.90	N/A	92.94	0.66	99.12	93.07	0.73	99.08	92.80	98.90	50.00	93.24	1.54	98.68
Avg	92.95	95.17	N/A	92.51	15.49	89.62	91.88	25.61	84.24	92.27	57.38	68.55	92.75	23.00	85.98
ATTACK	i-BAU [45]	EP [48]	NPD-TU(Ours)	NPD-TP(Ours)	NPD(Ours)
ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER
BadNets-A2O [11]	87.43	4.48	92.46	89.80	1.26	95.26	90.81	1.44	95.67	90.90	0.62	96.12	88.93	1.26	94.82
BadNets-A2A [11]	89.39	1.29	85.32	88.72	3.00	84.12	91.66	0.82	86.68	92.54	0.04	87.19	91.41	0.89	86.52
Blended [5]	89.43	26.82	84.34	91.94	48.22	74.89	91.88	0.03	98.96	91.33	0.83	98.28	91.18	0.41	98.42
Input-Aware [28]	89.91	0.02	97.10	93.68	2.88	97.56	92.01	0.14	98.09	93.24	0.00	98.78	89.57	0.11	96.89
LF [46]	88.92	11.99	91.49	91.97	84.73	56.64	91.42	0.01	98.73	91.92	0.08	98.94	90.06	0.21	97.95
SSBA [20]	86.53	2.89	93.91	91.67	4.33	95.76	91.61	2.46	96.67	91.82	0.83	97.59	90.88	2.77	96.15
Trojan [24]	89.29	0.54	97.63	92.32	2.49	98.18	92.59	0.04	99.53	92.19	0.00	99.35	92.37	6.51	96.19
WaNet [29]	90.70	0.88	97.96	90.47	96.52	50.03	92.18	3.24	97.52	92.57	7.47	95.60	91.57	0.80	98.43
Avg	88.95	6.11	92.53	91.32	30.43	81.55	91.77	1.02	96.48	92.06	1.23	96.52	90.75	1.62	95.67
Table 2: Comparison with the state-of-the-art defenses on Tiny ImageNet dataset with  $5\%$  benign data and  $10\%$  poison ratio on PreAct-ResNet18 (\%).
ATTACK	Backdoored	FP [23]	NAD [19]	NC [35]	ANP [43]
ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER
BadNets-A2O [11]	56.12	99.90	N/A	48.81	0.66	95.96	48.35	0.27	95.93	56.12	99.90	50.00	47.34	0.00	95.56
BadNets-A2A [11]	55.99	27.81	N/A	47.88	3.19	58.26	48.29	2.30	58.91	54.12	18.72	53.61	40.70	2.39	55.07
Blended [5]	56.49	99.67	N/A	50.58	57.89	67.93	55.22	98.88	49.76	54.50	96.07	50.80	43.21	43.80	71.29
Input-Aware [28]	57.67	99.19	N/A	52.38	0.13	96.88	57.42	0.07	99.43	53.46	2.48	96.25	50.56	0.00	96.04
LF [46]	55.21	98.51	N/A	48.18	63.83	63.83	49.61	58.01	67.45	53.08	90.48	52.95	41.75	65.98	59.54
SSBA [20]	55.97	97.69	N/A	48.06	52.25	68.76	47.67	69.47	59.96	53.30	0.26	97.38	41.83	14.24	84.65
Trojan [24]	56.48	99.97	N/A	45.96	8.88	90.28	48.83	1.01	95.65	54.43	1.54	98.19	45.36	0.53	94.16
WaNet [29]	57.81	96.50	N/A	50.35	1.37	93.83	50.02	0.87	93.92	57.81	96.50	50.00	30.34	0.00	84.51
Avg	56.47	89.90	N/A	49.02	23.53	79.47	50.68	28.86	77.63	54.60	50.74	68.65	42.64	15.87	80.10
ATTACK	i-BAU [45]	EP [48]	NPD-TU(Ours)	NPD-TP(Ours)	NPD(Ours)
ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER
BadNets-A2O [11]	51.63	95.92	49.74	54.00	0.02	98.88	47.23	0.01	95.50	49.89	1.28	96.19	49.79	2.51	95.53
BadNets-A2A [11]	53.52	12.89	56.22	54.79	1.28	62.67	46.81	1.96	58.34	49.79	3.31	59.15	49.94	5.57	58.10
Blended [5]	50.76	95.58	49.18	56.32	88.88	55.31	46.24	0.00	94.71	49.72	0.18	96.36	49.62	0.12	96.34
Input-Aware [28]	55.49	0.46	98.27	57.33	0.03	99.41	49.54	0.27	95.39	53.88	0.04	97.68	53.75	5.93	94.67
LF [46]	53.65	94.27	51.34	54.86	93.20	52.48	46.04	0.00	94.67	49.20	0.30	96.10	49.94	2.48	95.38
SSBA [20]	52.39	84.64	54.73	55.56	66.67	65.30	46.56	0.00	94.14	49.04	0.00	95.38	49.25	0.01	95.48
Trojan [24]	51.85	99.15	48.09	54.47	0.12	98.92	48.56	0.00	96.02	49.61	0.05	96.52	49.43	0.51	96.20
WaNet [29]	53.04	69.82	60.95	57.06	0.20	97.77	48.52	0.01	93.60	51.88	0.82	94.88	52.64	0.24	95.54
Avg	52.79	69.09	58.57	55.55	31.30	78.84	47.44	0.28	90.30	50.38	0.75	91.53	50.54	2.17	90.90
adversarial perturbation renders it ineffective against sample-specific attacks and those utilizing large norm triggers. I-BAU shows similar performance in removing backdoors with an average DER of  $92.53\%$ , but it leads to a significant decrease in ACC, likely due to training the entire network by adversarial training.
- Defense performance of NPD-TU and NPD-TP are better than NPD. When the target label is known, the model only needs to find perturbations for that specific label, simplifying trigger identification and unlearning. These two methods outperform NPD, except for WaNet, which is a transformation-based attack without visible triggers. Fully perturbing the network proves more effective than solely unlearning targeted triggers in WaNet.
- NPD-TU is effective for trigger-additive attacks while NPD-TP is expert in defending against sample-specific attacks. It can be observed by comparing defense results on different attacks like Blended and SSBA on CIFAR-10. This demonstrates that the applicability of different strategies varies across different attack scenarios.
- Defense performance is robust across all attacks on Tiny ImageNet. Similar to the results on CIFAR-10, our method outperforms other methods in terms of ASR and DER for all backdoor
Table 3: Defense performance under different Table 4: Defense results in comparison with NPD- components of losses. UU and NPD-UP.
ATTACK →
Lbce1	Lbce2	Lasr	BadNets-A2O [11]	Blended [5]	LF [46]
ACC	ASR	ACC	ASR	ACC	ASR
91.45	1.18	92.47	99.63	92.00	95.90
✓	90.17	1.19	91.51	2.01	90.91	9.60
✓	90.46	0.38	91.68	18.28	91.19	1.06
✓	90.02	0.27	91.31	98.32	91.07	0.80
✓	✓	89.56	0.21	91.09	1.73	90.47	7.63
✓	✓	✓	88.93	1.26	91.18	0.41	90.06	0.21
ATTACK ↓	No defense	NPD-UU	NPD-UP	NPD (Ours)
ACC	ASR	ACC	ASR	ACC	ASR	ACC	ASR
BadNets-A2O [11]	91.82	93.79	79.35	0.10	90.61	1.74	88.93	1.26
Blended [5]	93.44	97.71	86.35	10.77	92.35	3.86	91.18	0.41
LF [46]	93.01	99.06	82.95	75.42	91.53	17.24	90.06	0.21
SSBA [20]	92.88	97.07	84.31	52.36	91.49	14.22	90.88	2.77
Trojan [24]	93.47	99.99	89.81	38.11	92.61	11.43	92.37	6.51
WaNet [29]	92.80	98.90	84.70	5.98	92.11	1.41	91.57	0.80
attacks. Despite a slight decrease in ACC, NPD-TU achieves a remarkably good performance with  $\mathrm{ASR} < 1.5\%$  on average. NPD-TP and NPD perform best in removing backdoors while maintaining model utility.
In summary, our method outperforms other state-of-the-art approaches, showcasing the broad applicability of our proposed method across diverse datasets. Due to space limits, defending results on GTSRB dataset and VGG19-BN network can be found in Section C of Appendix.
Effectiveness of each loss term. We conduct an ablation study to evaluate the contribution of each component of the loss function towards the overall performance on CIFAR-10 dataset. We separately investigate the first and second terms of loss  $\mathcal{L}_{bce}$  (see Eq. (3)), denoting them as  $\mathcal{L}_{bce1}$  and  $\mathcal{L}_{bce2}$ , respectively. Throughout the study, we keep the loss  $\mathcal{L}_{bn}$  consistent across all experiments, and the result is shown in Table 3. Notably, the loss  $l_{bce1}$  plays a significant role in improving the overall performance while removing each component leads to a significant drop in defense in certain cases. This ablation study underscores the importance of each loss component in effectively mitigating different types of attacks.
Effectiveness of the targeted adversarial perturbations in NPD. To show the efficacy of the targeted adversarial perturbations in NPD (see Eq. (6)), we compare NPD with its two variants using two types of untargeted perturbation. We refer to adversarial perturbations generated by UAP and standard PGD without a targeted label as NPD-UU (untargeted universal adversarial perturbation) and NPD-UP (untargeted PGD), respectively. As shown in Table 4, NPD-UU and NPD-UP fail to remove backdoors in certain cases although NPD-UP obtains a higher ACC. This result shows the superiority of NPD in removing backdoors.
Performance of choosing different layers to insert the transformation layer. We evaluate the influence of choosing different layers to insert the transformation layer by inserting it after each convolution layer of PreAct-ResNet18 network on CIFAR-10 dataset. Figure 3 shows the defense performance under three attacks. The result shows that inserting the transformation layer into the shallower layers results in a decrease in accuracy. This is because even a slight perturbation in the shallow layers can cause significant instability in the final output. However, as the layer goes deeper, the features become more separable, resulting in better defense performance.
Figure 3: Defense performance of inserting linear transformation into different layers.
Defense effectiveness under different poisoning
ratios. To investigate the impact of poisoning ratios on defense performance, we conducted experiments on NPD with different poisoning ratios on the CIFAR-10 dataset. As presented in Table 5, there is a slight decrease in ACC as the poisoning ratio increases. Moreover, our approach exhibits a notably stable defense performance across a range of poisoning ratios.
Table 5: Defense results under different poisoning ratio on CIFAR-10 and PreAct-ResNet18(%)
Poisoning Ratio → ATTACK ↓	5% No Defense	Ours	10% No Defense	Ours	20% No Defense	Ours	30% No Defense	Ours	40% No Defense	Ours
BadNets-A2O [11]	ACC	92.35	87.99	91.82	88.93	90.17	85.77	88.32	86.76	86.16	82.31
ASR	89.52	0.89	93.79	1.26	96.12	0.39	97.33	0.69	97.78	3.88
Blended [5]	ACC	93.76	91.48	93.44	91.18	93.00	91.27	92.78	90.10	91.64	88.85
ASR	99.31	11.06	97.71	0.41	99.92	4.41	99.98	3.78	99.96	24.92
Input-Aware [28]	ACC	90.92	90.31	94.03	89.57	89.18	88.60	89.63	89.41	90.12	88.57
ASR	94.19	0.30	98.35	0.11	97.66	3.51	97.62	3.43	98.57	0.98
WaNet [29]	ACC	93.38	91.38	92.80	91.57	91.02	89.77	92.35	89.53	92.21	89.53
ASR	97.27	0.14	98.90	0.80	94.93	0.22	99.06	1.71	99.49	2.19
Table 6: Results with different number of clean data on CIFAR-10  $(\%)$
ATTACK	# Clean data → Defense ↓	500	250	50
ACC	ASR	DER	ACC	ASR	DER	ACC	ASR	DER
BadNets-A2O [11]	i-BAU [45]	65.41	7.74	79.82	66.05	2.02	81.03	64.60	22.10	70.27
ANP [43]	91.53	5.50	94.00	90.81	2.03	93.40	85.57	1.52	91.04
Ours	87.58	1.38	94.09	87.80	0.27	92.78	86.91	0.20	92.37
SSBA [20]	i-BAU [45]	84.21	17.73	85.33	76.84	51.90	60.38	67.48	98.37	35.21
ANP [43]	92.06	28.67	83.79	92.13	27.61	80.17	88.31	22.28	80.92
Ours	90.48	0.43	97.12	90.80	10.03	88.29	89.48	8.97	88.16
LF [46]	i-BAU [45]	83.91	19.34	85.31	70.12	99.42	35.53	70.26	99.31	35.60
ANP [43]	92.74	46.70	76.04	92.28	18.02	84.11	89.99	18.77	82.59
Ours	90.21	0.99	97.63	89.28	0.50	91.37	88.18	10.83	85.65
Defense effectiveness under different clean ratios. We investigate the sensitivity of clean data on defense performance and compare our NPD with SOTA defenses. As shown in Table 6, NPD is less sensitive to the size of clean data among all the attacks and defenses. Even with only 50 samples, it still maintains acceptable performance. This result shows that our method exhibits minimal reliance on the number of training samples.
Defense against adaptive attacks. We evaluate the effectiveness of NPD against adaptive attacks. Consider the attacker knows the defense strategy and the attacker trains the backdoored model with adversarial training (AT). The defense performance against the AT backdoored model under different attacks is shown in Table 7. NPD still performs well on AT model, while i-BAU fails in these attacks. One possible reason is that i-BAU is essentially adversarial training, while NPD adopts dynamic targeted adversarial perturbation, which is different from AT. Compared to the defense against backdoored models with standard training (Table 1), there is a slight ASR increase (from  $2.77\%$  to  $5.22\%$  for SSBA, for example).
Defense against all to all attacks. To demonstrate the effectiveness of our dynamic and sample-specific target label prediction strategy, we conducted experiments involving all-to-all attacks for various backdoor attack methods. In these experiments, each class is treated as the target class. The result is presented in Table 8. As illustrated in the table, our NPD approach successfully eliminates all backdoors, proving its efficacy in multi-label cases. In contrast, i-BAU method fails in most cases.
Running time comparison. We measure the runtime of the defense methods on 2500 CIFAR-10 images with batch size 256 and PreAct-ResNet18. The experiments were conducted on one RTX 4090Ti GPU and the results are presented in Table 9. Among these methods, our proposed PND-UN achieves the fastest performance on CIFAR-10, requiring only 56 seconds. It should be noted that our method was trained for 50 epochs, while i-BAU was only trained for 5 epochs.
Table 7: Defense performance against adaptive Table 8: Defense performance against all2all attacks on CIFAR-10 dataset (\%).
ATTACK	Backdoored	i-BAU [45]	NPD	ATTACK	Backdoored	i-BAU [45]	NPD
ACC	ASR	ACC	ASR	ACC	ASR	ACC	ASR	ACC	ASR	ACC	ASR
Blended [5]	86.00	99.63	83.98	30.43	83.64	3.92	Blended [5]	91.59	83.50	88.80	15.74	91.07	6.24
Input-Aware [28]	84.98	94.99	83.17	71.12	83.13	4.47	Input-Aware [28]	86.60	78.55	89.50	5.37	90.10	2.10
LF [46]	84.15	94.30	84.15	94.30	83.39	4.89	LF [46]	91.91	84.80	88.92	33.33	90.53	4.88
SSBA [20]	84.34	93.32	82.74	24.54	83.38	5.22	SSBA [20]	91.30	85.04	91.16	26.00	91.14	1.29
Table 9: Running time of different defense methods with 2500 CIFAR-10 images on PreActResNet18.
Defense (sec.)	FP [23]	NAD [19]	NC [35]	ANP [43]	i-BAU [45]	EP [48]	NPD (Ours)
CIFAR-10	1169.01	74.39	896.45	58.75	57.23	131.84	55.16
Tiny ImageNet	3357	351	42512	1692	887	302	332
Inspired by the mechanism of optical polarizer, this work proposed a novel backdoor defense method by inserting a learnable neural polarizer as an intermediate layer of the backdoored model. We instantiated the neural polarizer as a lightweight linear transformation and it could be efficiently and effectively learned with limited clean samples to mitigate backdoor effect. To learn a desired neural polarizer, a bi-level optimization problem is proposed by filtering trigger features of poisoned samples while maintaining benign features of both poisoned and benign samples. Extensive experiments demonstrate the effectiveness of our method across all evaluated backdoor attacks and all other defense methods under various datasets and network architectures.
Limitations and future work. Although only limited clean data is needed for our method to achieve a remarkable defense performance, the accessibility of clean data is still an important limitation of the proposed method, which may restrict the application of our method. Therefore, a promising direction for future work is to further reduce the requirement of clean data by exploring data-free neural polarizer or learning neural polarizer based on poisoned training data.
Broader impacts. Backdoor attacks pose significant threats to the deployment of deep neural networks obtained from untrustworthy sources. This work has made a valuable contribution to the community with an efficient and effective backdoor defense strategy to ease the threat of existing backdoor attacks, even with a very limited set of clean samples, which ensures its practicality. Besides, the innovative defense strategy of learning additional lightweight layers, rather than adjusting the whole backdoored model, may inspire more researchers to develop more efficient and practical defense methods.
Structure of Appendix. Detailed algorithms of PGD and UAP, proposed NPD-TU and NPD-TP, along with a theoretical analysis in the kernel space are provided in Section A. Implementation details of experiments are introduced in Section B. More defense results in different datasets and networks are provided in Section C. Defense performance of different structures of neural polarizer is discussed in Section D. Additional experimental results and visualization are presented in Section E.
This work is supported by the National Natural Science Foundation of China under grant No. 62076213, Shenzhen Science and Technology Program under grant No. RCYX20210609103057050, No. ZDSYS20211021111415025, No. GXWD20201231105722002-20200901175001001, Shenzhen Science and Technology Program under grant No. JCYJ20210324120011032, CAAI-Huawei MindSpore Open Fund, and the Guangdong Provincial Key Laboratory of Big Data Computing, the Chinese University of Hong Kong, Shenzhen.
This appendix is organized as follows.
- In Section A, we first provide detailed algorithms of the proposed NPD-TU and NPD-TP, along with an introduction to the project gradient descent (PGD) method and UAP method. Then we present an example of backdoor attacks in the kernel space and provide a theoretical analysis of the existence of linear transformation to mitigate backdoors.
- In Section B, we introduce the implementation details, including dataset introduction, implementation details of compared attacks, defenses, and our proposed methods.
- In Section C, we display the defense results in comparison with state-of-the-art (SOTA) defenses on GTSRB dataset, as well as those on CIFAR-10 with the VGG19-BN network.
- In Section D, we discuss different structures of neural polarizers on the defense performance.
- In Section E, we present additional experimental results, including defense experiments conducted on the CIFAR-100 dataset, defense performance across various model architectures, and visualization experiments.
A Theoretical analysis and algorithm
A.1 Introduction to PGD algorithm and UAP
In this section, we provide an introduction to PGD attack [27] and universal adversarial perturbation (UAP) attack [25].
PGD. Projected Gradient Descent (PGD) is a multi-step variant of Fast Gradient Sign Method (FGSM [17], which is projected gradient descent on the negative loss function, i.e., given an input  $(x,y)$  and a model  $\hat{f}_{\theta}$ , the adversarial example is calculated as follows,
$$
\boldsymbol {x} ^ {n + 1} = \Pi_ {\boldsymbol {x} + \mathcal {S}} \left(\boldsymbol {x} ^ {n} + \alpha \operatorname {s g n} \left(\nabla_ {\boldsymbol {x}} L _ {\mathrm {C E}} (\hat {f} _ {\boldsymbol {\theta}} (\boldsymbol {x}), y)\right)\right), \tag {7}
$$
where  $\Pi_{x + S}$  denotes the projection on to the  $\rho$ -ball with  $L_{p}$  norm:  $\| \pmb{x}^{n + 1} - \pmb{x}\| _p\leq \rho$ , and  $\pmb{x}^0 = \pmb{x}$ ,  $n = 0,\dots ,N - 1$ .  $\alpha$  is the step size. However, in Section 3 of the main script, we use a confidence-guided targeted PGD approach to obtain the perturbation. Thus, the adversarial example is updated as follows,
$$
\boldsymbol {x} ^ {n + 1} = \Pi_ {\boldsymbol {x} + \mathcal {S}} \left(\boldsymbol {x} ^ {n} - \alpha \operatorname {s g n} \left(\nabla_ {\boldsymbol {x}} L _ {\mathrm {C E}} (\hat {f} _ {\boldsymbol {\theta}} (\boldsymbol {x}), T)\right)\right), \tag {8}
$$
where  $T$  is the target label for  $\pmb{x}$ . We denote the adversarial perturbation as follows for convenience:
$$
\boldsymbol {\delta} = \boldsymbol {x} ^ {n + 1} - \boldsymbol {x} ^ {n}. \tag {9}
$$
UAP. The main difference between UAP and PGD is that UAP computes a universal adversarial perturbation for the whole training samples; thus the adversarial example for targeted UAP is updated as follows,
$$
\boldsymbol {x} _ {i} ^ {n + 1} = \Pi_ {\boldsymbol {x} _ {i} + \mathcal {S}} \left(\boldsymbol {x} _ {i} ^ {n} - \alpha \operatorname {s g n} \left(\sum_ {i = 0} ^ {b} \nabla_ {\boldsymbol {x} _ {i}} L _ {\mathrm {C E}} \left(\hat {f} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {i}), T\right)\right)\right), \tag {10}
$$
where  $T$  is the target label for  $\pmb{x}$ , and  $b$  is the number of samples in one training batch. Eq (10) means all the samples in one batch share the sample gradient to update the perturbation.
A.2 Detailed algorithms of NPD-TU and NPD-TP
In the main script, we introduce the algorithm for NPD. Here, we provide the detailed algorithms for NPD-TU and NPD-TP as in Alg 2 and 3.
Algorithm 2 Neural Polarizer based Backdoor Defense of Targeted UAP (NPD-TU)
1: Input: Training set  $\mathcal{D}_{bn}$ , backdoored model  $f_{\pmb{w}}$ , neural polarizer  $g_{\theta}$ , target label  $T$ , inner steps  $N$ , learning rate  $\eta > 0$ , perturbation bound  $\rho > 0$ , norm  $p$ , hyper-parameters  $\lambda_1, \lambda_2, \lambda_3 > 0$ , warm-up epochs  $\mathcal{T}_0$ , training epochs  $\mathcal{T}$ .  
2: Output: Model  $\hat{f}(\pmb{w}, \pmb{\theta})$ .  
3: Initialize  $\pmb{\theta}$  to be an identity function, fix  $\pmb{w}$ , and construct the composed network  $\hat{f}(\pmb{w}, \pmb{\theta})$ .  
4: Warm-up: Train  $\hat{f}(\pmb{w}, \pmb{\theta})$  with  $\mathcal{L}_{\mathrm{CE}}(D_{bn})$  for  $\mathcal{T}_0$  epochs.  
5: for  $t = 0, \dots, T - 1$  do  
6: for mini-batch  $\mathcal{B} = \{(x_i, y_i)\}_{i=1}^b \subset \mathcal{D}_{bn}$  do  
7: for  $n = 0, \dots, N - 1$  do  
8: Compute adversarial perturbation  $\delta$  with  $\| \delta \|_p \leq \rho$  and target label  $T$  by targeted UAP attack [25] via Eq. (10);  
9: end for  
10: Update  $\pmb{\theta}$  via outer minimization of Eq. (5) in the main script by SGD.  
11: end for  
12: end for  
13: return Model  $\hat{f}(\pmb{w}, \pmb{\theta})$ .
Algorithm 3 Neural Polarizer based Backdoor Defense of Targeted PGD (NPD-TP)
1: Input: Training set  $\mathcal{D}_{bn}$ , backdoored model  $f_{\pmb{w}}$ , neural polarizer  $g_{\theta}$ , target label  $T$ , inner steps  $N$ , learning rate  $\eta > 0$ , perturbation bound  $\rho > 0$ , norm  $p$ , hyper-parameters  $\lambda_1, \lambda_2, \lambda_3 > 0$ , warm-up epochs  $\mathcal{T}_0$ , training epochs  $\mathcal{T}$ .  
2: Output: Model  $\hat{f}(\pmb{w}, \pmb{\theta})$ .  
3: Initialize  $\pmb{\theta}$  to be an identity function, fix  $\pmb{w}$ , and construct the composed network  $\hat{f}(\pmb{w}, \pmb{\theta})$ .  
4: Warm-up: Train  $\hat{f}(\pmb{w}, \pmb{\theta})$  with  $\mathcal{L}_{\mathrm{CE}}(D_{bn})$  for  $\mathcal{T}_0$  epochs.  
5: for  $t = 0, \dots, T - 1$  do  
6: for mini-batch  $\mathcal{B} = \{(x_i, y_i)\}_{i=1}^b \subset \mathcal{D}_{bn}$  do  
7: for  $n = 0, \dots, N - 1$  do  
8: Compute adversarial perturbations  $\{(\delta_i)\}_{i=1}^b$  with  $\| \delta_i \|_p \leq \rho$  and target label  $T$  by targeted UAP attack [25] via Eq. (8);  
9: end for  
10: Update  $\pmb{\theta}$  via outer minimization of Eq. (5) in the main script by SGD.  
11: end for  
12: end for  
13: return Model  $\hat{f}(\pmb{w}, \pmb{\theta})$ .
A.3 Theoretical analysis under kernel space
In this section, we demonstrate the existence of linear transformation in reproducing kernel Hilbert space (RKHS), a very successful tool in various areas of machine learning. For simplicity, we consider a binary classifier  $h(\cdot ,m):\mathcal{X}\times \{0,1\} \to \mathbb{R}$  in a reproducing kernel Hilbert space  $\mathcal{H}_{XM}$  generated by kernel  $\kappa_{XM}$ , where  $m$  is a binary poison indicator, i.e.,  $m = 1$  means adding trigger to the input and  $m = 0$  means not adding trigger. Denote the feature of  $(\pmb {x},m)$  in kernel space by  $\phi_{XM}(\pmb {x},m)$ . Given a dataset  $\mathcal{D}$  and a distribution of poison indicator  $\mathcal{M}$ , the poisoned classifier is optimized by minimizing a mean squared error loss:
$$
h _ {b d} = \underset {h \in \mathcal {H} _ {X M}} {\arg \min } \mathbb {E} _ {\boldsymbol {x} \sim \mathcal {D}, m \sim \mathcal {M}} (h (\boldsymbol {x}, m) - y) ^ {2}. \tag {11}
$$
Then, by the theorem in [38] which shows that there exists a subspace of  $\mathcal{H}_{XM}$  in which a functin  $h$  satisfies  $\operatorname{Cov}(h(\pmb{x}, m), m) = 0$ , we provide the following lemma: