SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mingli Zhu $^{1*}$  Shaokui Wei $^{1*}$  Hongyuan Zha $^{1,2}$  Baoyuan  $\mathbf{W}\mathbf{u}^{1\dagger}$
School of Data Science,
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China
Shenzhen Institute of Artificial Intelligence and Robotics for Society, China
Recent studies have demonstrated the susceptibility of deep neural networks to backdoor attacks. Given a backdoorsed model, its prediction of a poisoned sample with trigger will be dominated by the trigger information, though trigger information and benign information coexist. Inspired by the mechanism of the optical polarizer that a polarizer could pass light waves with particular polarizations while filtering light waves with other polarizations, we propose a novel backdoor defense method by inserting a learnable neural polarizer into the backdoorsed model as an intermediate layer, in order to purify the poisoned sample via filtering trigger information while maintaining benign information. The neural polarizer is instantiated as one lightweight linear transformation layer, which is learned through solving a well designed bi-level optimization problem, based on a limited clean dataset. Compared to other fine-tuning-based defense methods which often adjust all parameters of the backdoorsed model, the proposed method only needs to learn one additional layer, such that it is more efficient and requires less clean data. Extensive experiments demonstrate the effectiveness and efficiency of our method in removing backdoors across various neural network architectures and datasets, especially in the case of very limited clean data. Codes are available at https://github.com/SCLBD/BackdoorBench (PyTorch) and https://github.com/JulieCarlon/NPD-MindSpore (MindSpore).
Several studies have revealed the vulnerabilities of deep neural networks (DNNs) to various types of attacks [15, 17, 21, 22, 40], of which backdoor attacks [6, 9, 11, 36] are attracting increasing attention. In backdoor attacks, the adversary could produce a backdoored DNN model through manipulating the training dataset [5, 24] or the training process [20, 29], such that the backdoored model predicts any poisoned sample with particular triggers to the predetermined target label, while behaves normally on benign samples. Backdoor attacks can arise from various sources, such as training based on a poisoned dataset, or utilizing third-party platforms for model training, or downloading backdoored models from untrusted third-party providers. These scenarios significantly elevate the threat of backdoor attacks to DNNs' applications, and meanwhile highlight the importance of defending against backdoor attacks.
Several seminal backdoor defense methods have been developed, mainly including 1) in-training approaches, which aim to train a secure model based on a poisoned dataset through well-designed training algorithms or objective functions, such as DBD [14] and D-ST [4]. 2) post-training approaches, which aim to mitigate the backdoor effect from a backdoored model through adjusting the
†Corresponds to Baoyuan Wu (wubaoyuan@cuhk.edu.cn).
Figure 1: Left: Illustration of an optical polarizer [41]. Only light waves with specific polarizations can pass through the polarizer among the three incident light waves. Right: Defense against backdoors by integrating a trainable neural polarizer into the compromised model. The neural polarizer effectively filters out backdoor-related features, effectively eliminating the backdoor.
model parameters (e.g., fine-tuning or pruning), usually based on a limited subset of clean training dataset, such as fine-pruning [23], ANP [43], or i-BAU [45]. This work focuses on the latter one. However, there are two limitations to existing post-training approaches. First, given very limited clean data, it is challenging to find a good checkpoint to simultaneously achieve backdoor mitigation and benign accuracy maintenance from the high-dimensional loss landscape of a complex model. Second, adjusting all parameters of a complex model is costly.
To tackle the above limitations, we propose a lightweight and effective post-training defense approach, which only learns one additional layer, while fixing all layers of the original backdoored model. It is inspired by the mechanism of the optical polarizer [44] that in a mixed light wave with diverse polarizations, only the light wave with some particular polarizations could pass the polarizer, while those with other polarizations are blocked (see Fig. 1-left). Correspondingly, by treating one poisoned sample as the mixture of trigger feature and benign feature, we define a neural polarizer and insert it into the backdoored model as one additional intermediate layer (see Fig. 1-right) in order to filter trigger feature and maintain benign feature, such that poisoned samples could be purified to mitigate backdoor effect, while benign samples are not significantly influenced.
In practice, to achieve an effective neural polarizer, it should be learned to weaken the correlation between the trigger and the target label while keeping the mapping from benign samples to their ground-truth labels. However, the defender only has a limited clean dataset, while neither trigger nor target label is accessible. To tackle it, we propose a bi-level optimization, where a sample-specific target label is dynamically estimated according to the output confidence, and the trigger is approximated by the targeted adversarial perturbation. Besides, in our experiments, the neural polarizer is implemented by a linear transformation (i.e., the combination of one  $1 \times 1$  convolutional layer and one batch normalization layer). Consequently, it can be efficiently and effectively learned with very limited clean data to achieve good defense performance, which is verified by extensive experiments on various model architectures and datasets.
Our main contributions are three-fold. 1) We propose an innovative backdoor defense approach that only learns one additional linear transformation called neural polarizer while all parameters of the backdoorsed model are fixed, such that it just requires very low computational cost and very limited clean data. 2) A bi-level optimization problem and an effective learning algorithm are provided to optimize the parameter of the neural polarizer. 3) Extensive experimental results demonstrate the superiority of the proposed method on various networks and datasets, in terms of both efficiency and effectiveness.
Backdoor attacks. Traditional backdoor attacks are additive attacks that modify a small fraction of training samples by patching a pre-defined pattern and assigning them to targeted labels [8, 11]. These modified samples, along with the unaffected samples, constitute a poisoned dataset [11]. The model trained with this dataset will be implanted with backdoors that predict the targeted label when triggered by the injected patterns while maintaining normal behavior on clean samples [5, 50]. Recently, advanced attacks have considered more invisible trigger injection methods such as training an auto-encoder feature embedding or using a local transformation function [20, 29, 46]. To increase the stealthiness of attacks, clean-label attacks succeed by obfuscating image subject information and establishing a correlation between triggers and targeted labels without modifying the labels of poisoned samples [1, 2, 30].
Backdoor defense. Backdoor defense methods can be broadly categorized into training-stage defenses and post-processing defenses. Training-stage defenses assume access to a poisoned dataset for model training [10]. The different behaviors between the poisoned and clean samples can be leveraged to identify suspicious instances, such as sensitivity to transformation [4] and clustering phenomenon in feature space [14]. Most defense methods belong to post-processing defenses [39], which assume that the defender only has access to a suspicious DNN model and a few clean samples. Therefore, they must remove the backdoor threat with limited resources. There are mainly three types of defense strategies: trigger reversion methods try to recover the most possible triggers and utilize the potential triggers to fine-tune the model [35]; pruning-based methods aim at locating the neurons that are most related to backdoors and pruning them [23, 43, 47, 48]; and fine-tuning based defenses leverage clean samples to rectify the model [19, 45]. I-BAU [45] is most closely related to our method, which formulates a minimax optimization framework to train the network with samples under universal adversarial perturbations. However, our method differs from I-BAU in that our approach does not require training the entire network, and our perturbation generation mechanism is distinct. Other methods proposed for backdoor detection include Beatrix [26], which uses Gram matrices to identify poisoned samples; and AEVA [12], which detects backdoor models by adversarial extreme value analysis. In this study, we focus on post-processing defenses and primarily compare our method with state-of-the-art post-processing defenses [19, 23, 35].
3.1 Basic settings
Notations. We consider a classification problem with  $K$  classes  $(K\geq 2)$ . Let  $\pmb {x}\in \mathcal{X}\subset \mathbb{R}^d$  be a  $d$ -dimensional input sample, and its ground truth label is denoted as  $y\in \mathcal{V} = \{1,\dots ,K\}$ . Then, a  $L$  layers deep neural network  $f:\mathcal{X}\times \mathcal{W}\to \mathbb{R}^{K}$  parameterized by  $\boldsymbol {w}\in \mathcal{W}$  is defined as:
$$
f (\boldsymbol {x}; \boldsymbol {w}) = f _ {\boldsymbol {w} _ {L}} ^ {(L)} \circ f _ {\boldsymbol {w} _ {L - 1}} ^ {(L - 1)} \circ \dots \circ f _ {\boldsymbol {w} _ {1}} ^ {(1)} (\boldsymbol {x}), \tag {1}
$$
where  $f_{\boldsymbol{w}_l}^{(l)}$  is the function (e.g., convolution layer) with parameter  $\boldsymbol{w}_l$  in the  $l^{\text{th}}$  layer with  $1 \leq l \leq L$ . For simplicity, we denote  $f(\boldsymbol{x}; \boldsymbol{w})$  as  $f_{\boldsymbol{w}}(\boldsymbol{x})$  or  $f_{\boldsymbol{w}}$ . Given input  $\boldsymbol{x}$ , the predicted label of  $\boldsymbol{x}$  is given by  $\arg \max_k f_k(\boldsymbol{x}; \boldsymbol{w}), k = 1, \dots, K$ , where  $f_k(\boldsymbol{x}; \boldsymbol{w})$  is the logit of the  $k^{\text{th}}$  class.
Threat model. We assume that the adversary could produce the backdoored model  $f_{\mathbf{w}}$  through manipulating training data or training process, such that  $f_{\mathbf{w}}$  performs well on benign sample  $\mathbf{x}$  (i.e.,  $f_{\mathbf{w}}(\mathbf{x}) = y$ ), and predicts poisoned sample  $\mathbf{x}_{\Delta} = r(\mathbf{x}, \Delta)$  to the target label  $T$ , with  $\Delta$  indicating the trigger and  $r(\cdot, \cdot)$  being the fusion function of  $\mathbf{x}$  and  $\Delta$ . Considering that the adversary may set multiple targets, we use  $T_i$  to denote the target label for  $\mathbf{x}_i$ .
Defender's goal. Assume that the defender has access to the backdoored model  $f_{\mathbf{w}}$  and a limited set of benign training data, denoted as  $\mathcal{D}_{bn} = \{(x_i,y_i)\}_{i=1}^N$ . The defender's goal is to obtain a new model  $\hat{f}$  based on  $f_{\mathbf{w}}$  and  $\mathcal{D}_{bn}$ , such that the backdoor effect will be mitigated and the benign performance is maintained in  $\hat{f}$ .
3.2 Neural polarizer for DNNs
Neural polarizer. We propose the neural polarizer to purify poisoned sample in the feature space. Formally, it is instantiated as a lightweight linear transformation  $g_{\theta}$ , parameterized with  $\theta$ . As shown in Fig. 1,  $g_{\theta}$  is inserted into the neural network  $f_{w}$  at a specific immediate layer, to obtain a combined network  $f_{w,\theta}$ . For clarity, we denote  $f_{w,\theta}$  as  $\hat{f}_{\theta}$ , since  $w$  is fixed.
A desired neural polarizer should have the following three properties:
- Compatible with the neighboring layers: in other words, its input feature and its output activation must have the same shape. This requirement can be fulfilled through careful architectural design.
- Filtering trigger features in poisoned samples: after the neural polarizer, the trigger features should be filtered, such that the backdoor is deactivated, i.e.,  $\hat{f}_{\theta}(\pmb{x}_{\Delta}) \neq T$ .
- Preserving benign features in poisoned and benign samples: the neural polarizer should preserve benign features, such that  $\hat{f}_{\theta}$  performs well on both poisoned and benign samples, i.e.,  $\hat{f}_{\theta}(\boldsymbol{x}_{\Delta}) = \hat{f}_{\theta}(\boldsymbol{x}) = y$ .
Figure 2: An example of neural polarizer for a DNN.
The first property could be easily satisfied by designing neural polarizer's architecture. For example, as illustrated in Fig. 2, given the input feature map with the shape  $C_k \times H_k \times W_k$ , the neural polarizer is implemented by a convolution layer (Conv) with  $C_k$  convolution filters of shape  $1 \times 1$ , followed by a Batch Normalization (BN) layer. The Conv-BN block can be seen as a linear transformation layer. To satisfy the latter two properties,  $\theta$  should be learned by solving some well designed optimization, of which the details are presented in Section 3.3.
3.3 Learning neural polarizer
3.3.1 Loss functions in oracle setting
To learn a good neural polarizer  $g_{\theta}$ , we first consider an oracle setting where the trigger  $\Delta$  and target label  $T$  are given. We present some loss functions to encourage  $g_{\theta}$  to satisfy the latter two properties mentioned above, as follows:
- Loss for filtering trigger features in poisoned samples. Given trigger  $\Delta$  and target label  $T$ , filtering trigger features can be implemented by weakening the connection between  $\Delta$  and  $T$  with the following loss:
$$
\mathcal {L} _ {a s r} (\boldsymbol {x}, y, \Delta , T; \boldsymbol {\theta}) = - \log \left(1 - s _ {T} \left(\boldsymbol {x} _ {\Delta}; \boldsymbol {\theta}\right)\right), \tag {2}
$$
where  $s_T(\pmb{x}_{\Delta}; \theta)$  is the softmax probability of predicting  $\pmb{x}_{\Delta}$  to label  $T$  by  $\hat{f}_{\pmb{\theta}}$ . By reducing  $\mathcal{L}_{asr}$ , the attack success rate of the backdoor attack can be decreased.
- Loss for maintaining benign features in poisoned samples. To purify the poisoned sample such that it can be classified to the true label, we leverage the boosted cross entropy defined in [37]:
$$
\mathcal {L} _ {b c e} (\boldsymbol {x}, y, \Delta ; \boldsymbol {\theta}) = - \log \left(s _ {y} \left(\boldsymbol {x} _ {\Delta}; \boldsymbol {\theta}\right)\right) - \log \left(1 - \max  _ {k \neq y} s _ {k} \left(\boldsymbol {x} _ {\Delta}; \boldsymbol {\theta}\right)\right). \tag {3}
$$
- Loss for maintaining benign features in benign samples. To preserve benign features in benign sample, we adopt the widely used cross-entropy loss, i.e.,  $\mathcal{L}_{bn}(\boldsymbol{x},y;\boldsymbol{\theta}) = \mathcal{L}_{\mathrm{CE}}(\hat{f}_{\boldsymbol{\theta}}(\boldsymbol{x}),y)$ .
3.3.2 Approximation & optimization
Approximating  $T$  and  $\Delta$ . Note that as  $T$  and  $\Delta$  are inaccessible in both  $\mathcal{L}_{asr}$  and  $\mathcal{L}_{bce}$ , these two losses cannot be directly optimized. Thus, we have to approximate  $T$  and  $\Delta$ . In terms of
approximating  $T$ , although some methods have been developed to detect target class of backdoored models [12, 26], here we adopt a simple sample-specific and dynamic strategy:
$$
T \approx y ^ {\prime} = \underset {k \neq y} {\arg \max } \hat {f} _ {k} (\boldsymbol {x}; \boldsymbol {\theta}). \tag {4}
$$
There are two main advantages of this strategy. First, the predicted target label enables us to generate targeted adversarial perturbation. As analyzed later, the targeted adversarial perturbation is a better surrogate for the unknown trigger than the untargeted adversarial perturbation. Second, the sample-specific target label prediction is applicable to both all2one (one trigger, one target class) and all2all (every class is target class) attack settings. In practice, defenders lack certainty about attack types (all2one, all2all, or multi-trigger multi-target), risking sub-optimal defenses due to erroneous guesses or detection. Our sample-specific target label prediction avoids this risk.
In terms of approximating  $\Delta$ , it is approximated by the targeted adversarial perturbation of  $f_{w}$ , i.e.,
$$
\Delta \approx \delta^ {*} = \underset {\| \delta \| _ {p} \leq \rho} {\arg \min } \mathcal {L} _ {\mathrm {C E}} \left(\hat {f} _ {\boldsymbol {\theta}} (\boldsymbol {x} + \delta), y ^ {\prime}\right), \tag {5}
$$
where  $\| \cdot \| _p$  is the  $L_{p}$  norm,  $\rho$  is the budget for perturbations.
Bi-level optimization problem. By employing the approximations of  $T$  and  $\Delta$  shown in Eq. (4) and (5), respectively, and substituting these approximations into the respective loss functions of Eq. (2) and (3), we introduce the following optimization problem to learn  $\theta$  based on the clean data  $\mathcal{D}_{bn}$ :
$$
\begin{array}{l} \min  _ {\boldsymbol {\theta}} \quad \frac {1}{N} \sum_ {i = 1} ^ {N} \lambda_ {1} \mathcal {L} _ {b n} \left(\boldsymbol {x} _ {i}, y _ {i}; \boldsymbol {\theta}\right) + \lambda_ {2} \mathcal {L} _ {a s r} \left(\boldsymbol {x} _ {i}, y _ {i}, \boldsymbol {\delta} _ {i} ^ {*}, y _ {i} ^ {\prime}; \boldsymbol {\theta}\right) + \lambda_ {3} \mathcal {L} _ {b c e} \left(\boldsymbol {x} _ {i}, y _ {i}, \boldsymbol {\delta} _ {i} ^ {*}; \boldsymbol {\theta}\right), \tag {6} \\ \text{s.t.} \qquad \boldsymbol{\delta}_{i}^{*} = \operatorname *{arg  min}_{\| \boldsymbol{\delta}_{i}\|_{p}\leq \rho}\mathcal{L}_{\mathrm{CE}}\left(\hat{f}_{\boldsymbol{\theta}}(\boldsymbol{x}_{i} + \boldsymbol{\delta}_{i}),y_{i}^{\prime}\right), y_{i}^{\prime} = \operatorname *{arg  max}_{k_{i}\neq y_{i}}\hat{f}_{k_{i}}(\boldsymbol{x}_{i};\boldsymbol {\theta}), i = 1,\ldots ,N, \\ \end{array}
$$
where  $\lambda_1, \lambda_2, \lambda_3 > 0$  are hyper-parameters to adjust the importance of each loss function. This bi-level optimization is dubbed Neural Polarizer based backdoor Defense (NPD).
**Variants.** To comprehensively evaluate the performance of NPD, we also provide two variants, with the relaxation that if the target label  $T$  is known. One is that approximating the trigger by the targeted adversarial perturbation for each benign sample in Eq. (6), i.e.,  $\Delta \approx \delta_i^* = \arg \min_{\| \delta_i \|_p \leq \rho} \mathcal{L}_{\mathrm{CE}}(f(\boldsymbol{x}_i + \delta_i), T)$ , dubbed NPD-TP. The other is approximating the trigger by the targeted universal adversarial perturbation (TUAP) [25] for all training samples in fine-tuning, dubbed NPD-TU. We empirically found that the targeted adversarial perturbation is a better surrogate to the trigger than the untargeted ones.
Optimization algorithm. We proposed Algorithm 1 to solve the above optimization problem. Specifically, NPD solves problem (6) by alternatively updating the surrogate target label  $y'$ , the perturbation  $\delta$  and  $\theta$  as follows:
- Inner minimization: Given parameter  $\pmb{\theta}$  of the neural polarizer, we first estimate the target label for sample  $\pmb{x}_i$  by  $y_i' = \arg \max_{k_i \neq y_i} \hat{f}_{k_i}(\pmb{x}_i; \pmb{\theta})$ . Then, the targeted Project Gradient Descent (PGD) [27] is employed to generate the perturbation  $\delta_i^*$  via Eq. (5).
- Outer minimization: Given  $y'$  and  $\delta^{*}$  for each sample in a batch, the  $\theta$  can be updated by taking one stochastic gradient descent [3] (SGD) step w.r.t. the outer minimization objective in Eq. (6).
Attack settings. We evaluate the proposed method on eight famous backdoor attacks, including BadNets [11] (BadNets-A2O and BadNets-A2A refer to attacking one target label and all labels, respectively), Blended attack (Blended) [5], Input-aware dynamic backdoor attack (Input-aware)[28], Low frequency attack (LF) [46], Sample-specific backdoor attack (SSBA) [20], Trojan backdoor
Algorithm 1 Neural Polarizer based Backdoor Defense (NPD)
1: Input: Training set  $\mathcal{D}_{bn}$ , backdoored model  $f_{w}$ , neural polarizer  $g_{\theta}$ , learning rate  $\eta > 0$ , perturbation bound  $\rho > 0$ , norm  $p$ , hyper-parameters  $\lambda_1, \lambda_2, \lambda_3 > 0$ , warm-up epochs  $\mathcal{T}_0$ , training epochs  $\mathcal{T}$ , number of PGD steps  $T_{adv}$ .  
2: Output: Model  $\hat{f}(\boldsymbol{w}, \boldsymbol{\theta})$ .  
3: Initialize  $\boldsymbol{\theta}$  to be an identity function, fix  $\boldsymbol{w}$ , and construct the composed network  $\hat{f}(\boldsymbol{w}, \boldsymbol{\theta})$ .  
4: Warm-up: Train  $\hat{f}(\boldsymbol{w}, \boldsymbol{\theta})$  with  $\mathcal{L}_{\mathrm{CE}}(D_{bn})$  for  $\mathcal{T}_0$  epochs.  
5: for  $t = 0, \dots, T - 1$  do  
6: for mini-batch  $\mathcal{B} = \{(x_i, y_i)\}_{i=1}^b \subset \mathcal{D}_{bn}$  do  
7: Compute  $\{y_i'\}_{i=1}^b$  by Eq. (4);  
8: Generate perturbations  $\{(\delta_i)\}_{i=1}^b$  with  $\|\delta_i\|_p \leq \rho$  and  $\{y_i'\}_{i=1}^b$  by targeted PGD attack [27] via Eq. (5);  
9: Update  $\boldsymbol{\theta}$  via outer minimization of Eq. (6) by SGD.  
10: end for  
11: end for  
12: return Model  $\hat{f}(\boldsymbol{w}, \boldsymbol{\theta})$ .
attack (Trojan) [24], and Warping-based poisoned networks (WaNet) [29]. We follow the default attack configuration as in BackdoorBench [42] for a fair comparison. The poisoning ratio is set to  $10\%$  in comparison with SOTA defenses. These attacks are conducted on three benchmark datasets: CIFAR-10 [16], Tiny ImageNet [18], and GTSRB [34]. We test all attacks on PreAct-ResNet18 [13] and VGG19-BN [33].
Defense settings. We compare the proposed methods with six SOTA backdoor defense methods, i.e., Fine-pruning (FP) [23], NAD [19], NC [35], ANP [43], i-BAU [45], and EP [48]. All these defenses have access to  $5\%$  benign training samples. The training hyperparameters are adjusted based on BackdoorBench [42]. We evaluate the proposed method under two proposed scenarios and compare our NPD-TU, NPD-TP, and NPD with SOTA defenses. For the ablation study, we focus solely on NPD, which represents a more generalized scenario. We apply an  $l_{2}$  norm constraint to the adversarial perturbations, with a perturbation bound of 3 for CIFAR-10 and GTSRB datasets, and 6 for Tiny ImageNet. We train the neural polarizer for 50 epochs with batch size 128 and learning rate 0.01 on each dataset and the transformation block is inserted before the third convolution layer of the fourth layer for PreAct-ResNet18. The loss hyper-parameters  $\lambda_{1}, \lambda_{2}, \lambda_{3}$  are set to 1, 0.4, 0.4 for NPD, and 1, 0.1, 0.1 for NPD-TU and NPD-TP. More implementation details on SOTA attacks, defenses, and our methods can be found in Section B of Appendix.
Evaluation metric. In this work, we use clean ACCuracy (ACC), Attack Success Rate (ASR), and Defense Effectiveness Rating (DER) as evaluation metrics to assess the performance of different defenses. ACC represents the accuracy of clean samples while ASR measures the ratio of successfully misclassified backdoor samples to the target label. Defense Effectiveness Rating  $(\mathrm{DER} \in [0,1]$  [49]) is a comprehensive measure that considers both ACC and ASR:
$$
\mathrm {D E R} = \left[ \max  (0, \Delta \mathrm {A S R}) - \max  (0, \Delta \mathrm {A C C}) + 1 \right] / 2, \tag {7}
$$
where  $\Delta$ ASR denotes the decrease of ASR after applying defense, and  $\Delta$ ACC denotes the drop in ACC following defense. Higher ACC, lower ASR, and higher DER indicate better defense performance. Note that in comparison with SOTA defenses, the one achieving the best performance is highlighted in **boldface**, while the second-best result is indicated by **underlining**. We provide PyTorch and MindSpore implementations of NPD.
Table 1 and Table 2 showcase the defense performance of the proposed method in comparison to six SOTA defense methods on CIFAR-10 and Tiny ImageNet. The following observations can be made: