SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Input: querying input  $X_{\mathrm{test}}$ , clean validation dataset  $D^{\mathrm{Val}} = \{(X_i,Y_i)\}_{i=1}^n$ , transformation method  $T(\cdot)$ , score function  $s(T(\cdot))$ , desired false positive rate  $\alpha \in (0,1)$ , violation rate  $\delta \in (0,1)$
1: Receiving a future query sample  $X_{\text{test}}$
2: for  $i = 1$  to  $n$  do
3: Calculate  $s_i = s(T(X_i)) / X_i \in D^{\mathrm{Val}}$
4: end for
5: Select the decision threshold  $\lambda_{\alpha ,s}$  according to Equation (1).
6: Determine if  $X_{\mathrm{test}}$  is a clean sample based on  $s(T(X_{\mathrm{test}})) \leq \lambda_{\alpha, s}$
Output: The decision whether the sample  $X_{\mathrm{test}}$  is a clean or backdoor sample
The next result shows that if a future test input comes from the same distribution as the validation data, the above algorithm can achieve a false positive rate bounded by  $\alpha$  with high probability.
Theorem 1 (Conditional False positive rate of Algorithm 1). Given any pre-trained backdoored classifier  $f$ , suppose that the validation dataset  $D^{\mathrm{Val}}$  and the test data  $(X_{\mathrm{test}}, Y_{\mathrm{test}})$  are IID drawn from the clean data distribution  $\mathbb{P}_{\mathrm{CLEAN}}$ . Given any  $\delta \in (0,1)$ , for any score function  $s$ , such that the resulting scores  $\{s(X_i)\}_{i=1}^n$  remain IID and continuous, the associated backdoor conformal detector  $g(\cdot; s, \lambda_{\alpha, s})$  as specified in Procedure 1 satisfies
$$
\mathbb {P} \left(g \left(X _ {\text {t e s t}}; s, \lambda_ {\alpha , s}\right) = 1 (\text {B a c k d o o r S a m p l e}) \mid D ^ {\text {V a l}}\right) \leq \alpha ,
$$
with probability at lease  $1 - \delta$  for any  $\alpha \in (0,1)$  such that  $\alpha >\sqrt{(\log(2 / \delta) / (2n))}$ .
3.3 Optimal score functions for CBD
In the preceding section, we presented the CBD framework and established that when utilizing suitable score functions, which preserve the independence and identical distribution of transformed samples, the associated backdoor conformal detector achieves an FPR within the range of  $\alpha$ . This prompts an inquiry: what constitutes the optimal score function (and its corresponding conformal detector) that maximizes the TPR while maintaining an FPR within a predefined  $\alpha$  threshold?
At a high level, deriving theoretical results on the TPR without distributional assumptions on the backdoor data is not possible, as previously discussed. Therefore, our first step is to investigate the unique structure of backdoor data. We note that the distribution of backdoor data depends on both the clean data distribution and the learning model used. This is because backdoor data is generated by transforming clean data with backdoor triggers  $\eta_{1}$ , and the effectiveness of the trigger in achieving the attacker's goal may vary under different models.
To aid in the theoretical analysis, we will be making certain assumptions about the components mentioned above. However, it is important to note that the practical algorithm presented in the next section does not rely on these assumptions. The development in this subsection primarily serves to derive technical insights. Specifically, we will assume that the Suppose that the (A) marginal clean data is normally distributed with mean 0 and covariance  $\Sigma$ , (B) the attacker employs a linear classifier  $f_{\theta}(x) = \mathbf{1}\{\theta^{\top}x > 0\}$  for  $\theta \in \mathbb{R}^d$ , and (C) the attacker applies the backdoor transformation  $\eta_1(x) = x + \gamma$ , where  $\gamma \in \mathrm{T}_c \triangleq \{u \in \mathbb{R}^d \mid \|u\|_2 = c, c > 0\}$ . It is important to mention that the defender does not possess knowledge of the exact backdoor trigger  $\gamma^*$  utilized by the attacker. However, the defender is aware of the set of potential backdoor triggers  $\mathrm{T}_c$ . Also, we note that the  $\ell_2$  norm constraint on  $\eta$  is practically important since backdoor triggers that are imperceptible to humans, i.e., those with a smaller norm, are more favorable than human-perceptible triggers, e.g., a square patch. Given the availability of distributional information, our analysis in this section will be conducted at the population level, directly working with the Attacker's Goal as specified in Section 3.
Taking into account all the aforementioned information, the defender can construct a uniformly most powerful detection rule if they possess knowledge of the exact backdoor trigger employed by the
Figure 2: The mean values of  $s(Z)$  (vertical-line bar for clean data, slash-line bar for backdoor data) in Eq. (2), by replacing the  $\gamma^{*}$  with the real backdoor trigger, on GTSRB with ResNet18 over different backdoor attacks.
attacker, following the principles of the Neyman-Pearson Lemma [56]. This raises the question for the defender: when they obtain the backdoored model based on the attacker's objective described in Section 3, are there multiple backdoor triggers  $\gamma \in \mathrm{T}_c$  that could lead to this specific model? If the answer is yes, the defender would have complete knowledge of the backdoor trigger and, consequently, the backdoor data distribution, enabling them to derive the most effective detection rule. To address this concern, we provide a definitive answer to the question as follows.
Theorem 2 (Uniqueness of the backdoor trigger). Under Assumptions (A), (B), and (C) as specified above, for any  $\theta^{*} \in \mathbb{R}^{d}$  that satisfies the Attacker's Goal, as specified in Section 3, the backdoor trigger  $\gamma^{*} \in \mathrm{T}_{c}$  that corresponds to this attacker's backdoored classifier  $\theta^{*}$ , is unique and admits an explicit form.
Remark 3 (The defender has full knowledge about the form of the backdoor trigger and the backdoor data distribution). The above result indicates that there is a unique correspondence between a backdoor trigger  $\gamma^{*} \in \mathrm{T}_{c}$  and the resulting backdoored classifier  $\theta^{*}$  that fulfills the Attacker's Goal. Therefore, when the attacker provides the backdoored model to the defender, the defender is aware that only one specific backdoor trigger is associated with this particular model. As a result, the defender possesses complete knowledge of the distribution of backdoor data by utilizing the explicit form of the backdoor trigger, which is included in the appendix.
Theorem 3 (Uniformly most powerful detection rule). Under the same setup in Theorem 2, the defender's optimal score function resulting in the largest TPR given a FPR  $\leq \alpha$  is in the form of:
$$
s (Z) \propto \exp \left(Z ^ {\top} \Sigma^ {- 1} Z - \left(Z - \gamma^ {*}\right) ^ {\top} \Sigma^ {- 1} \left(Z - \gamma^ {*}\right)\right). \tag {2}
$$
The above optimal detection rule is based on the likelihood ratio, which compares the probability of a future sample  $Z$  from the clean distribution with that from the backdoor distribution. Through empirical evaluation, we have found that the proposed score function, when using the actual backdoor trigger instead of  $\eta^{*}$ , effectively distinguishes between clean and backdoor data in various scenarios, including 12 state-of-the-art backdoor attacks in CV benchmarks (e.g., CIFAR10) and NLP benchmarks (such as SST-2). The results are illustrated in Figure 2.
3.4 Practical proxy: Shrunk-Covariance Mahalanobis (SCM)
The previous analysis shows that the Mahalanobis distance between the clean and data is the optimal detection rule under known Gaussian distributions. However, in the real-world scenario, the defender knows neither the clean nor the backdoor data distributions. Therefore, the two exact distances as specified in Equation (2) above can not be computed.
To overcome this challenge, a proximal approach is proposed in the following. Recall that the defender has a small set of clean validation dataset which is assumed to have the same distribution as the clean training data. Hence, one feasible way to calculate the theoretically optimal detector in Equation (2) is to use the empirical Mahalanobis distance calculated based on a clean validation, namely  $M(z,D^{\mathrm{Val}})\triangleq (z - \mu)^{\top}V(z - \mu)$ , where  $\mu$  is the empirical mean of a  $D^{\mathrm{Val}}$ , and  $V$  is the inverse of the sample covariance matrix of  $D$ . The choice of this selection is because we empirically observed that backdoor and clean data have significantly different values under the decision rule derived in Eq. (2) as shown in Figure 2.
A significant challenge in using Mahalanobis distance arises in high-dimensional scenarios, particularly when estimating the precision matrix, i.e., the inverse of the covariance matrix. The sample
covariance matrix, a popular choice, exhibits numerical instability in large data dimensions [57, 58], leading to an ill-conditioned matrix that is difficult to invert [59]. The estimation errors increase rapidly with data dimension  $d$  [60], causing issues for high-dimensional latent embeddings in deep neural networks, like ResNet18 and BERT. Consequently, MD may not preserve order information for  $D^{\mathrm{Val}}$ , impacting the FPR in Theorem 1.
To address this issue, we consider regularizing the sample covariance matrix through shrinkage techniques [61, 62]. At a high level, the shrinkage technique is used to adjust the eigenvalues of a matrix. This adjustment involves changing the ratio between the smallest and largest eigenvalues of a dataset's covariance matrix. One way to do this is by shifting each eigenvalue by a certain amount, which in practice leads to the following, Shrink-Covariance Mahalanobis (SCM) score function, defined as
$$
s \left(\phi^ {\text {p o i}} (z); f ^ {\text {p o i}}, D ^ {\text {V a l}}\right) \triangleq \left(\phi^ {\text {p o i}} (z) - \mu\right) ^ {\top} M _ {\beta} ^ {- 1} \left(\phi^ {\text {p o i}} (z) - \mu\right), \tag {3}
$$
where  $\phi^{\mathrm{poi}}(\cdot)$  is the latent representation of the backdoored DNNs,  $\mu$  is the mean of  $\{\phi^{\mathrm{poi}}(x_i)\}_{i=1}^n$  and  $M_{\beta} \triangleq (1 - \beta)S + \beta(\mathrm{Tr}(S)/d)I_d$ , with  $S$  being the sample covariance of  $\{\phi^{\mathrm{poi}}(x_i)\}_{i=1}^n$ ,  $\mathrm{Tr}(\cdot)$  is the trace operator, and  $\beta \in (0,1)$  is the shrinkage level. The use of the latent space representation  $\phi^{\mathrm{poi}}(\cdot)$  instead of the original data information is motivated by the difficulty in modeling distributions over the original digital space [63]. The shrinkage parameter can be selected via cross-validation [64]. Empirically, we observed that the proposed SCM accurately distinguishes between clean and backdoor data in various backdoor attack scenarios, spanning both CV and NLP domains, as demonstrated from the histograms in Figure 3.
Figure 3: Histograms of our proposed SCM on CIFAR10 using ResNet 18 for different backdoor attacks in both CV and NLP domains. We noticed distinct separations in SCM scores between clean and backdoor data, in all scenarios.
In this section, we conduct extensive experiments to evaluate our proposed methods. The empirical results match our theoretical results in terms of the false positive rate (FPR), and the detection power of our methods outperforms state-of-the-art defenses on a wide range of benchmarks. We further provide a runtime analysis of our method and ablation studies for hyperparameters in the Appendix. All backdoor datasets and models were obtained by re-running publicly available codes (details in the appendix) The experiments were conducted on cloud computing machines equipped with Nvidia Tesla V100s.
4.1 Setups
Datasets We evaluate our proposed method on three popular (I) CV image classification datasets: CIFAR10 [34], GTSRB [35], and Tiny-ImageNet [36], and on two (II) NLP datasets: SST2 (Stanford Sentiment Treebank v2) [37] and IMDB [38]. Results for SST2 are presented in the main paper, while those for IMDB are deferred to the Appendix. We include a short introduction to each dataset in the Appendix.
Models (I) CV We use ResNet18 [2] and (II) NLP the base uncased-BERT [31] as the default model architecture for our experiments in the main text. Ablations studies on different model architectures, e.g., VGG 19 [39] and WideResNet [65], and mode configurations are included in the Appendix.
Metrics We consider two evaluation metrics: (1) the FPR (2) the area under the receiver operating characteristic curve (AUCROC). All experiments are independently repeated 10 times.
Attacks (I) CV In the main text, we evaluate our proposed method on ten popular attacks, including data-and-label poisoning attacks: BadNets [3], Blended [4], Trojan [5]), clean label attack: SIG [7], (3) attacks with invisible/sample-wise triggers: Dynamic [11], SSBA [15], WaNet [13], (4) and adaptive backdoor attacks: TaCT [14], Adaptive Blend [19], Adaptive Patch [19]. (II) NLP We evaluate the proposed method on two state-of-the-art attacks: (1) SOS [23] and LWS [40]. In the appendix, we assess the performance of our methods against two additional categories of backdoor
Figure 4: The mean FPRs of our proposed method on GTSRB are shown in each plot, which is independently replicated 10 times. The solid line represents the mean value, and the standard errors are  $< 0.01$  for all cases. Our method's FPRs consistently match the theoretical upper bounds.
attacks, which are tailored to exploit vulnerabilities in our techniques. Configuration details of these attacks, including trigger patterns, poisoning rates, and hyperparameters, are included in the Appendix.
Performance of Attacks (I) CV On CIFAR10 and GTSRB datasets, the backdoored model achieves clean accuracy rates exceeding  $93\%$  and backdoor accuracy rates exceeding  $98\%$  for all the previously mentioned backdoor attacks. The exceptions are the SIG and Adaptive Blend attacks, which result in backdoor accuracy rates of around  $80\%$ . (II) NLP On SST2 and IMDB datasets, the backdoored model attains clean accuracy rates surpassing  $90\%$  and backdoor accuracy rates exceeding  $96\%$  for the mentioned backdoor attacks.
Defenses For both (I) CV and (II) NLP, in the main text, we compare our proposed method with three detection-based backdoor defenses in the literature, namely, STRIP [20],  $\ell_2$  based-defense [45], and MAD (Mean Absolute Deviation)-based defense. In the appendix, we compare our methods with two recent detection-based defenses designed for CV backdoor attacks, SCALEUP and FREQ [24, 55], as well as filtering-based training-stage defenses, SPECTRE [46] and ABL [47].
4.2 Main results
FPR on CIFAR10 and GTSRB We assess the FPR performance of our proposed method CBD-SCM (SCM score function within the CBD framework) on, CIFAR10 and GTSRB datasets by varying the defender pre-specified false positive rate  $\alpha$ . Each plot corresponds to a specific backdoor attack and shows the FPR performance of our proposed method under that attack. For each plot, we conduct ten independent experiment replications and calculate the mean, along with the  $\pm 1$  standard error bars shown in the shaded regions in Fig. 4 for GTSRB and Fig. 5.1 for CIFAR10 in the appendix. The results demonstrate that the FPR of our method (in red dots) consistently matches the theoretical upper bounds (in blue triangles), supporting the findings presented in Theorem 1.
Detection Power (AUCROC) on CIFAR10 and GTSRB We conduct a thorough evaluation of the detection performance of our proposed method against all ten distinct backdoor attacks and report the corresponding AUCROC values in Table 1. Our method achieves the best performance in all cases on both datasets. Specifically, we observe a remarkable increase in AUCROC, up to  $300\%$ , for the more advanced backdoor attacks such as non-patch-based backdoor attacks (WaNet and SSBA) and latent space adaptive attacks (Adaptive Blend and Adaptive Patch).
Detection Power (ROC) on Tiny-ImageNet We further evaluate the detection performance of our proposed method on Tiny-ImageNet against both classical patch-based attacks (e.g., Blended) and more advanced non-patch-based attacks (e.g., WaNet). We visualize the ROC curves for our method in Figure 5 (a) and (b). It shows a significant improvement in the TPR at low false positive rate regions compared with other methods.
Detection Power (ROC) on SST2 (NLP) We assess the detection performance of our proposed method on SST2 (NLP) using the base uncased-BERT model. Specifically, we generate ROC curves for our methods under two advanced backdoor attacks, as illustrated in Figure 5 (c) and (d). The results indicate that our method outperforms the STRIP method significantly, but are relatively less
(c)
Figure 5: AUCROC on Tiny-ImageNet and SST-2 NLP. For the Tiny-ImageNet dataset, our method significantly outperforms the existing methods under both patch-based attacks (Blended) and the more advanced non-patch-attack (WaNet). On the NLP SST2 benchmark, our proposed method consistently outperforms the STRIP method.
effective under the SOS attack compared to the other two defenses that use latent representations. These findings suggest that the current NLP attacks retain a considerable amount of information in the latent representations that can be utilized to differentiate between clean and backdoor data.
Table 1: AUCROC results for CIFAR10 and GTSRB. Mean values of ten replicates are reported, with standard errors below 0.01 for all cases. The best-performing method(s) are indicated in **boldface**.
Defense →	CIFAR10	GTSRB
STRIP	\( \ell_2 \)	MAD	SCM	STRIP	\( \ell_2 \)	MAD	SCM
BadNets	1.0	1.0	0.99	1.0	0.99	0.97	0.97	0.99
Blended	0.61	0.86	0.78	0.96	0.80	0.69	0.70	0.90
TrojanNN	0.55	0.90	0.76	0.99	0.50	0.90	0.89	0.99
SIG	0.72	0.51	0.33	0.97	0.41	0.32	0.31	0.76
Dynamic	0.84	0.99	0.99	1.0	0.99	0.99	0.99	0.99
SSBA	0.68	0.67	0.66	0.97	0.80	0.22	0.22	0.99
WaNet	0.39	0.41	0.40	0.93	0.54	0.31	0.21	0.99
TacT	0.68	0.45	0.37	0.92	0.51	0.55	0.44	0.71
Adaptive Blend	0.69	0.84	0.75	0.96	0.72	0.88	0.89	0.99
Adaptive Patch	0.76	0.85	0.79	0.98	0.33	0.56	0.49	0.87
This paper addresses defending against backdoor attacks by detecting backdoor samples during the inference stage. We introduce a mathematical formulation for the problem, highlight potential challenges, and propose a generic framework with provable false positive rate guarantees. Within this framework, we derive the most powerful detection rule under classical learning scenarios and offer a practical solution based on the optimal theoretical approach. Extensive experiments in both CV and NLP domains consistently validate our theory and demonstrate the efficacy of our method.
Limitation and future work. While our work presents a comprehensive framework for defending against inference-stage backdoor attacks, there are some limitations that warrant further investigation. Firstly, future studies could explore the optimal detection rule in more practical scenarios where only partial information regarding the clean and backdoor data distributions is available. An example is Assisted Learning [66-68], a decentralized learning scenario where learners only provide processed data information during the inference stage. Secondly, it would be interesting to extend the proposed approach to backdoor detection in variety of Federated Learning scenarios [17, 69-71]. Thirdly, it is worth investigating the use of general distributions conditional on side information (e.g., [72]) to design the optimal detection rule. Lastly, it would be helpful to explore information beyond the penultimate layer, e.g., the topology of neuron activations [73], for backdoor detection.
The Appendix contains additional details on the CBD framework, more introductions to the related work, ethical considerations, more extensive ablation studies, and all the technical proofs.
The work of Xun Xian, Xuan Bi, and Mingyi Hong was supported in part by a sponsored research award by Cisco Research. The work of Ganghua Wang and Jie Ding was supported in part by the Office of Naval Research under grant number N00014-21-1-2590.
[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in neural information processing systems, vol. 25, 2012.
[2] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[3] T. Gu, B. Dolan-Gavitt, and S. Garg, "Badnets: Identifying vulnerabilities in the machine learning model supply chain," arXiv preprint arXiv:1708.06733, 2017.
[4] X. Chen, C. Liu, B. Li, K. Lu, and D. Song, "Targeted backdoor attacks on deep learning systems using data poisoning," arXiv preprint arXiv:1712.05526, 2017.
[5] Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks,” 2017.
[6] A. Turner, D. Tsipras, and A. Madry, "Clean-label backdoor attacks," 2018.
[7] M. Barni, K. Kallas, and B. Tondi, "A new backdoor attack in cnns by training set corruption without label poisoning," in 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019, pp. 101-105.
[8] J. Dai, C. Chen, and Y. Li, “A backdoor attack against lstm-based text classification systems,” IEEE Access, vol. 7, pp. 138 872–138 878, 2019.
[9] S. Garg, A. Kumar, V. Goel, and Y. Liang, "Can adversarial weight perturbations inject neural backdoors," in Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020, pp. 2029-2032.
[10] S. Zhao, X. Ma, X. Zheng, J. Bailey, J. Chen, and Y.-G. Jiang, "Clean-label backdoor attacks on video recognition models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14443-14452.
[11] T. A. Nguyen and A. Tran, "Input-aware dynamic backdoor attack," Advances in Neural Information Processing Systems, vol. 33, pp. 3454-3464, 2020.
[12] K. Doan, Y. Lao, W. Zhao, and P. Li, "Lira: Learnable, imperceptible and robust backdoor attacks," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 11966-11976.
[13] A. Nguyen and A. Tran, "Wanet-imperceptible warping-based backdoor attack," arXiv preprint arXiv:2102.10369, 2021.
[14] D. Tang, X. Wang, H. Tang, and K. Zhang, “Demon in the variant: Statistical analysis of dnns for robust backdoor contamination detection.” in USENIX Security Symposium, 2021, pp. 1541–1558.
[15] Y. Li, Y. Li, B. Wu, L. Li, R. He, and S. Lyu, "Invisible backdoor attack with sample-specific triggers," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 16463-16472.
[16] X. Chen, A. Salem, D. Chen, M. Backes, S. Ma, Q. Shen, Z. Wu, and Y. Zhang, "Badnl: Backdoor attacks against nlp models with semantic-preserving improvements," in Annual Computer Security Applications Conference, 2021, pp. 554-569.
[17] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” in International conference on artificial intelligence and statistics. PMLR, 2020, pp. 2938–2948.
In Section A, we present the formal proofs for the theoretical results discussed in Section 3 of the main text, specifically Theorem 1, 2, and 3. Furthermore, we provide a comprehensive listing of the detailed configurations used in the experimental study, along with a brief introduction to all the datasets, in Section B. We also include the omitted empirical results from the main text, focusing on the CIFAR10 [34] and IMDB (NLP) [38] datasets, which are presented in Section C. Additionally, we conduct an ablation study to explore the impact of different model architectures, such as VGG19 [39], which is discussed in Section D. Furthermore, we conduct an additional ablation study to examine the effects of varying poisoning ratios, as presented in Section E. We perform an ablation study to investigate the choice of the hyperparameter  $\beta$ , and the results are provided in Section F. We further subject our proposed methods to backdoor attacks specifically tailored to target our techniques in Section G. Finally, We further compare our methods with some recent backdoor defenses including both inference-stage and training-stage methods in Section H.
In this section, we include the proof of the main results in Section 3.2 and 3.3 of the main text.