SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xun Xian
Department of ECE
xian0044@umn.edu
Ganghua Wang
School of Statistics
wang9019@umn.edu
Jayanth Srinivasa
Cisco Research
jasriniv@cisco.com
Ashish Kundu
Cisco Research ashkundu@cisco.com
Xuan Bi
University of Minnesota xbi@umn.edu
Mingyi Hong
mhong@umn.edu
Jie Ding
dingj@umn.edu
Backdoor attacks involve inserting poisoned samples during training, resulting in a model containing a hidden backdoor that can trigger specific behaviors without impacting performance on normal samples. These attacks are challenging to detect, as the backdoored model appears normal until activated by the backdoor trigger, rendering them particularly stealthy. In this study, we devise a unified inference-stage detection framework to defend against backdoor attacks. We first rigorously formulate the inference-stage backdoor detection problem, encompassing various existing methods, and discuss several challenges and limitations. We then propose a framework with provable guarantees on the false positive rate or the probability of misclassifying a clean sample. Further, we derive the most powerful detection rule to maximize the detection power, namely the rate of accurately identifying a backdoor sample, given a false positive rate under classical learning scenarios. Based on the theoretically optimal detection rule, we suggest a practical and effective approach for real-world applications based on the latent representations of backdoored deep nets. We extensively evaluate our method on 14 different backdoor attacks using Computer Vision (CV) and Natural Language Processing (NLP) benchmark datasets. The experimental findings align with our theoretical results. We significantly surpass the state-of-the-art methods, e.g., up to  $300\%$  improvement on the detection power as evaluated by AUCROC, over the state-of-the-art defense against advanced adaptive backdoor attacks.
Deep neural networks (DNNs) have made remarkable advancements in various domains, thanks to their exceptional learning capabilities [1, 2]. However, these same learning capabilities that make DNNs so powerful have unfortunately rendered them vulnerable to a growing critical threat known as the backdoor attack [3-19]. In a typical backdoor attack, the attacker manipulates a small part of the training dataset by inserting specific triggers, such as pixel patches in images [3] or random word combinations in texts [16]. These manipulated samples are labeled with a specific target
Figure 1: Illustration of inference-stage backdoor defenses. The backdoor triggers in text queries are indicated in red, while for the image queries, the backdoor triggers consist of a square patch located in the lower-right corner for the traffic sign and a hello kitty embedding added to an image of a dog.
class. When a DNN is trained on this poisoned dataset, it learns the original data patterns and the engineered relationship between the manipulated samples and the target class. As a result, the DNN exhibits unexpected and often malicious behavior, producing manipulated predictions for test inputs containing the backdoor trigger while functioning normally for unaltered test inputs. The malicious and stealthy nature of backdoor attacks underscores the critical need to defend against such threats.
In an effort to combat backdoor attacks in the inference stage, various empirical studies have been conducted [20-24] in both computer vision (CV) and natural language processing (NLP) domains. In these methods, given a trained-and-backdoorsed model, the defender will determine whether a future query input is clean or backdoor-triggered based on certain features of this query input, as demonstrated in Figure 1. For instance, in CV scenarios, the popular STRIP method [20] superimposes the future query image with other clean images and examines the model's prediction entropy on these superimposed versions. If the model's prediction entropy on these superimposed versions is abnormally low, the associated original test input is marked as backdoor-triggered. The ONION method [21] widely used in NLP applications removes individual tokens from the input text and measures the resulting change in perplexity. If removing a particular token leads to a significantly greater perplexity change than others, it is considered a backdoor trigger word.
However, a principled understanding of the backdoor detection problem is underexplored in the literature. First, a mathematical formulation for the backdoor detection problem is lacking, which may impede quantitative analysis of the backdoor challenges. Furthermore, while some existing methods are effective in certain scenarios, many of them lack theoretical guarantees regarding their detection performance, such as ensuring that the probability of misidentifying a clean sample as a backdoor one is no more than a certain threshold. This creates a complicated situation for researchers and practitioners in assessing the effectiveness of these methods.
Additionally, given the same nature of using excessive learning capability to conduct backdoor attacks for both CV and NLP contexts, it is natural to wonder if the defense methods developed for backdoor attacks in CV can be applied to backdoor attacks in NLP. However, applying CV defense methods to NLP is challenging due to the distinct data structures and models in each context [16, 25]. For example, images are treated as continuous while texts are discrete. Thus, the individual words utilized in [21] lack counterparts in images, preventing the use of techniques in [21] from defending against CV backdoors. Furthermore, the architectural differences between convolutional neural networks (CNNs) and recurrent neural networks (RNNs) add to the complexity. CNNs exhibit translational equivariance properties [26], whereas RNNs demonstrate auto-regressive characteristics [27, 28]. Hence, it remains unclear how the concept of CNN channels, as utilized in the backdoor defense of [22], can be translated to language models such as LSTM [29, 30], and BERT [31].
In this study, we introduce an approach for detecting backdoor test inputs with provable guarantees that is practically applicable to both CV and NLP domains. Contributions are summarized as follows.
Novel framework of backdoor defenses with provable guarantees. We first introduce a rigorous formulation for inference-stage backdoor detection problems, encompassing most existing online backdoor defense methods through binary classification/hypothesis testing concepts. This problem formulation enables precise discussions on the possible limitations of current backdoor defense
approaches. Based on these findings, we propose a generic backdoor defense framework called conformal backdoor, inspired by the conformal prediction literature [32,33], for detecting and filtering future backdoor test samples with provable assurances on the false positive rate (FPR), namely the probability of mistakenly classifying a clean sample as a backdoor one.
Optimal detection rule and its real-world proxy. To establish a theoretical comprehension of our proposed framework's detection power, we first show that successful backdoor attacks display distinctive properties under classical learning scenarios. In particular, we show that, among a set of potential backdoor triggers (and the corresponding backdoor data), the most effective (to be defined) backdoor trigger is unique. This uniqueness property allows the defender to devise the uniformly most powerful detection rule, in accordance with the Neyman-Pearson (NP) criterion, when clean and backdoor data distributions are known. To apply to real-world situations where neither clean nor backdoor data distribution is known, we propose a practical and efficient proxy based on the theoretically optimal detectors, which is applicable to both CV and NLP domains. The proxy detector utilizes the latent representations of a backdoored model, such as the penultimate layer, which can be widely applied and is generally independent of the specific data structure (e.g., discrete tokens in text) and the model's properties (e.g., the CNN's translation-invariant property).
Extensive empirical studies on both CV and NLP Our proposed technique was evaluated across various datasets for both CV, including CIFAR10 [34], GTSRB [35], and Tiny-ImageNet [36], and NLP, including SST2 [37] and IMDB [38], and popular model architectures such as ResNet18/34 [2], VGG19 [39], base uncased-BERT [31], and 14 backdoor attacks including BadNets [3], Blended [4], TrojanNN [5], SIG [7], WaNet [13], SSBA [15], TacT [14], Adaptive Patch [19], Adaptive Blend [19], SOS (NLP) [23], and LWS (NLP) [40]. Experimental results consistently endorse the effectiveness of our technique in terms of detection performance, e.g., up to  $300\%$  improvement on the detection power in terms of AUCROC over the state-of-the-art, STRIP [20], on adaptive backdoor attacks [14].
Backdoor attacks Backdoor attacks modify DNN predictions for specific inputs using backdoor triggers, while leaving regular inputs unaffected. The majority of backdoor attacks, both in CV [3-7, 10-15, 17-19], and NLP [8, 9, 16], are implemented through data poisoning, and the major focus of them is on how to design effective backdoor triggers. For instance, in the field of CV, classical methods use square patches and hello kitty as backdoor triggers [3, 4], which can be visually inspected. As a result, later techniques try to design human-imperceptible backdoor triggers through image warping [13] and generative modeling [15]. Also, there is a series of methods focusing on creating embedding-level imperceptible backdoor triggers with adaptive methods [14, 19]. Despite the distinct characteristics of natural language data, NLP backdoor attacks exhibit similar techniques to those used in CV. For example, rare word combinations are used as backdoor triggers [16]. In addition, researchers have proposed techniques to replace specific words in a text with new words searched from word embeddings while preserving the semantic meaning of the text [9]. Very recently, a principled theory regarding when and how backdoor attacks work was developed in [41].
Latent-space backdoor detection with access to training data This line of work aims to filter out backdoor data from the training dataset and retrain a sanitized model based on the cleansed dataset [14, 42-50]. These methods rely on the assumption that there is a clear separation between clean and backdoor data in the latent space of backdoored models, and the defender can use such a separation to detect backdoor samples. In CV, researchers have applied PCA to project high-dimensional embeddings into lower dimensions and then used clustering methods to filter out clean and backdoor data [44], presuming that backdoor samples are fewer than clean samples. Recent techniques, such as those proposed in [14, 46], leverage robust estimations techniques to improve the cluster analysis further. In NLP, the CUBE method [49] employs HDBSCAN [51] to identify embeddings clusters. In [45], authors use  $\ell_2$  embedding distance to remove backdoor samples by measuring the distance between the embeddings of each training example and the nearest trigger test example using  $\ell_2$  norm. Our work differs from these methods in that we do not have access to the training data. The defender has no prior knowledge of future test backdoor inputs, making it possibly more challenging for defending.
Backdoor detection with access to validation data This line of work detects backdoor samples by building statistics based on a clean validation dataset, which typically consists of IID samples drawn from clean training data distribution, and using the calculated statistics to decide if a new input is clean or backdoor [20-22, 52, 53]. For example, the classical method STRIP [20] filters out
poisoned samples by checking the randomness of the model's predictions when the input is perturbed several times [20], with the idea that the backdoor triggers dominate the rest information in the inputs regarding classification. These methods typically require a case-by-case design, depending on the data structure and models. Our method utilizes the latent representation of the backdoored model, which is not specific to a particular task.
2 Preliminary: backdoor defense and attack
Notations. Let  $\mathbb{R}^d$  be the input space and  $\mathcal{Y} = \{1,\dots ,C\}$  (abbreviated as  $[C])$  be the label space. We denote a model trained on a dataset  $\mathcal{D}$  by  $f^{\mathcal{D}}:\mathbb{R}^{d}\mapsto [C]$ . If  $f^{\mathcal{D}}$  is a DNN, we decompose  $f^{\mathcal{D}}$  as  $f^{\mathcal{D}} = w^{\mathcal{D}}\circ \phi^{\mathcal{D}}$ , where  $w^{\mathcal{D}}$  is the projection head and  $\phi^{\mathcal{D}}$  is the feature extractor (to be specified later).
Attacker's capability In line with previous backdoor poisoning attacks, we make the standard assumption that the adversary has control over the training process and can manipulate the training dataset, but does not have access to the inference stage [54]. Formally, let  $\mathcal{D}^{\mathrm{clean}} = \{(\hat{x}_i,\hat{y}_i)\}_{i = 1}^{n_{\mathrm{clean}}}.$  denote the clean training dataset of size  $n_{\mathrm{clean}}$  , independently drawn from a distribution  $\mathbb{P}_{\mathrm{CLEAN}}$  defined over  $\mathbb{R}^d\times \mathcal{V}$  . Let  $\eta_1:\mathbb{R}^d\mapsto \mathbb{R}^d$  and  $\eta_{2}:[C]\mapsto [C]$  denote the backdoor data and label transformations, respectively. We emphasize that attackers are allowed to use a wide range of backdoor transformations  $\eta_{1}$  , including a fixed patch [3,4] and dynamic/sample-specific patches [11, 14], Let  $\mathcal{D}^{\mathrm{backdoor}} = \{(\eta_1(\hat{x}_i),\eta_2(\hat{y}_i))\}_{i = 1}^{n_{\mathrm{backdoor}}}$  be the backdoor dataset under transformation  $(\eta_{1},\eta_{2})$  with the joint distribution  $\mathbb{P}_{\mathrm{BD}}$  . By default, we consider  $\eta_{2}(y) = 1$  for  $y\in [C]$  . Denote the learning model  $f^{\mathrm{clean}},f^{\mathrm{poi}}:\mathbb{R}^d\to [C]$  to be the model trained on  $\mathcal{D}^{\mathrm{clean}}$  and  $\mathcal{D}^{\mathrm{poi}}\triangleq \mathcal{D}^{\mathrm{clean}}\cup \mathcal{D}^{\mathrm{backdoor}}$  respectively. We set the risk minimization algorithm as the default formulation for training models.
Attacker's goal Given a clean dataset  $\mathcal{D}^{\mathrm{clean}}$ , a set of learning models, and the backdoor transformations  $(\eta_1,\eta_2)$ , the attacker aims to to minimize (I) the error rate on backdoor data under backdoor model:  $\mathbb{P}_{XY\sim \mathrm{CLEAN}}(f^{\mathrm{poi}}(\eta_1(X))\neq \eta_2(Y))$  under the constraint that the (II) the clean accuracy difference under the clean and the backdoor model:  $|\mathbb{P}_{XY\sim \mathrm{CLEAN}}(f^{\mathrm{poi}}(X) = Y) - \mathbb{P}_{XY\sim \mathrm{CLEAN}}(f^{\mathrm{clean}}(X) = Y)|$  does not exceed a pre-specified threshold value.
Defender's capability We consider a scenario where the defender can query the fixed, trained model  $f^{\mathrm{poi}}$  with any input but is unable to modify the model or access the poisoned training dataset  $\mathcal{D}^{\mathrm{poi}}$ . Specifically, the defender only has access to two kinds of information. The first (1) is the latent representation  $\phi^{\mathrm{poi}}(z)$  for any querying sample  $z \in \mathbb{R}^d$ , of the learning model  $f^{\mathrm{poi}}$ , i.e.,  $f^{\mathrm{poi}} = w^{\mathrm{poi}} \circ \phi^{\mathrm{poi}}$  for  $f^{\mathrm{poi}}$  being a DNN. For NLP classification tasks using BERT-based models, the output of  $\phi^{\mathrm{poi}}(\cdot)$  is the [CLS] embedding. For CNNs, the output of  $\phi^{\mathrm{poi}}(\cdot)$  is the avgpool12d layer immediately after all the convolutional layers. The (2) second is a set of validation data  $\mathcal{D}^{\mathrm{Val}} = \{(x_i, y_i)\}_{i=1}^n$  which are IID samples from the same distribution of clean training data  $\mathbb{P}_{\mathrm{CLEAN}}$ , with sample size  $n \ll n_{\mathrm{clean}}$  (sample size of the clean training data). By default, we set the value  $n$  to be 1000.
Defender's goal When given a future query  $X_{\mathrm{test}}$  and a backdoored model  $f^{\mathrm{poi}}$ , the defender should accurately and efficiently determine whether  $X_{\mathrm{test}}$  is a clean input or not. For example, a common objective for the defender is to maximize the true positive rate, which represents the correct identification of backdoor samples, while simultaneously minimizing the probability of misclassifying clean samples as backdoors.
3 Proposed framework of Conformal Backdoor (CBD) for backdoor defenses
We first thoroughly characterize the inference-stage backdoor detection problem by examining binary classification/hypothesis testing, which encompasses many existing online backdoor defense methods. We then discuss the difficulties associated with the problem and the possible limitations of existing methods before proposing our novel detection framework.
3.1 Backdoor detection formulation
Given a backdoored model  $f^{\mathrm{poi}}$  and a validation dataset  $\mathcal{D}^{\mathrm{Val}}$ , upon receiving a test query  $X_{\mathrm{test}} \in \mathbb{R}^{d}$ , the defender faces a binary classification/hypothesis testing problem:
$$
\mathrm {H} _ {0}: T (X _ {\text {t e s t}}) \sim T (S); \quad S \sim \mathbb {P} _ {\text {C L E A N}}, \quad \mathrm {H} _ {1}: T (X _ {\text {t e s t}}) \not \sim T (S); \quad S \sim \mathbb {P} _ {\text {C L E A N}},
$$
where  $T(\cdot)$  is a defender-constructed transformation on the original input  $X_{\mathrm{test}}$  and  $X \sim Y$  means that  $X$  and  $Y$  share the same distribution. Here, we note that the  $T(\cdot)$  typically depends on both the backdoored model  $f^{\mathrm{poi}}$  and the validation dataset  $\mathcal{D}^{\mathrm{Val}}$ , to reflect special properties of backdoor data, e.g., the predicted value  $T(X_{\mathrm{test}}) = f^{\mathrm{poi}}(X_{\mathrm{test}})$  and the latent representation  $T(X_{\mathrm{test}}) = \phi^{\mathrm{poi}}(X_{\mathrm{test}})$ . Also, it is worth mentioning that the true label  $Y_{\mathrm{test}}$  is not included since it is unknown to the defender.
Remark 1 (Unknown backdoor distribution  $\mathbb{P}_{\mathrm{BD}}$ ). The  $\mathrm{H}_0$  represents the scenario where the input originates from the clean distribution, while  $\mathrm{H}_1$  corresponds to the input not being derived from the clean distribution. It is important to emphasize that  $\mathrm{H}_1$  is not formulated to test if  $X_{\mathrm{test}}$  comes from a backdoor data distribution  $\mathbb{P}_{\mathrm{BD}}$ , specifically,  $\mathrm{H}_1: T(X_{\mathrm{test}}) \sim T(S)$  for  $S \sim \mathbb{P}_{\mathrm{BD}}$ . This is because, in real-world scenarios, the backdoor data distribution,  $\mathbb{P}_{\mathrm{BD}}$ , is unknown to the defender. This poses a significant challenge in deriving rigorous results on detection power. In particular, without any distributional assumption on  $\mathbb{P}_{\mathrm{BD}}$ , it is not possible to establish provable results on detection power.
To approach the above problem, defenders will construct a detector  $g(\cdot)$ , specified by
$$
g (X; s, \tau) = \left\{ \begin{array}{l l} 1 & (\text {B a c k d o o r - t r i g g e r e d S a m p l e}), \quad s (T (X)) \geq \tau \\ 0 & (\text {C l e a n S a m p l e}), \quad s (T (X)) <   \tau \end{array} \right.,
$$
where  $\tau \in \mathbb{R}$  is a threshold value and  $s(\cdot)$  is a scoring function which takes the transformed input  $T(X_{\mathrm{test}})$  and output a value in [0, 1] to indicate its chance of being a clean sample.
Remark 2 (Distinguishing our approach from backdoor defenses focused on training-data purification). There is a series of training-stage backdoor defense methods that have been developed with the goal of creating a clean model by identifying and removing potentially contaminated samples within the training data [43, 44, 46, 47, 50]. While these methods share the concept of distinguishing between clean and backdoor data, they differ from our approach in two significant ways. Firstly, these methods benefit from working with both clean and backdoor training data, giving them insights into both data categories. Secondly, their primary objective is to construct a clean model by singling out a subset of the training data based on specific characteristics of all training examples. Consequently, their evaluation criteria revolve around the purified model's ability to accurately classify clean data and protect against backdoor attacks.
In contrast, our approach serves as an inference-stage defense without access to the training data and no capability to influence the training process, including adjustments to model parameters. Similar to other inference-stage defenses [20, 21, 24, 55], we operate under the assumption that the defender has access to a limited set of clean validation data without prior knowledge of future backdoor test inputs. Our goal is to detect forthcoming backdoor inputs, and our evaluation metric is based on the AUCROC score of the detector.
3.2 Backdoor conformal detection
The main challenge for the defender is to devise an effective and efficient detector  $g(\cdot)$  to identify backdoor samples. Several evaluation metrics can be utilized to assess the detector's performance, including AUCROC and F1 score. Given the stealthiness of backdoor attacks, it is crucial for the defender to employ an evaluation method that offers provable guarantees regarding detection performance. To meet this requirement, it becomes necessary to consider both Type-I and Type-II errors within the hypothesis testing framework. Thus, we adopt the Neyman-Pearson criterion for our evaluation, ensuring a comprehensive assessment of our detection approach. Formally, the performance of a detector  $g$  can be described by its true positives rate (TPR):  $\mathbb{P}\big\{g(X;\tau) = 1 \mid X \text{ is backdoor}\}$  and its false positive rate (FPR):  $\mathbb{P}\big\{g(X;\tau) = 1 \mid X \text{ is clean}\}$ . The defender aims to find an optimal detection method  $g$  in the Neyman-Pearson (NP) paradigm:
$$
\operatorname {m a x i m i z e} \mathbb {P} \left\{g (X; \tau) = 1 \mid X \text {i s b a c k d o o r} \right\} \quad \text {s u b j e c t t o} \mathbb {P} \left\{g (X; \tau) = 1 \mid X \text {i s c l e a n} \right\} \leq \alpha ,
$$
where  $\alpha \in (0,1)$  is the defender pre-specified upper bound on the FPR. Proving guarantees on the (FPR) for a wide range of score functions or detectors is difficult because the defender lacks knowledge about clean data distribution. This limits their ability to determine the distribution of a score function  $s$  under  $\mathbb{P}_{\mathrm{CLEAN}}$ , making it hard to control the FPR.
To ensure the performance of the false positive rate, we propose a detection method based on the conformal prediction framework [32, 33]. The conformal prediction is a paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of black-box models without explicit distribution assumption on data. The central idea is that by using an appropriate score function, the empirical rank or quantile of the distribution will converge to the population version, which is guaranteed by the uniform convergence of CDFs. To be more specific, as outlined in Procedure 1, the defender will begin by computing the scores of the validation points using a validation dataset  $D^{\mathrm{Val}}$ . Next, the defender will set the decision value  $\lambda_{\alpha,s}$  to satisfy the
$$
\hat {F} _ {\text {C L E A N}} \left(\lambda_ {\alpha , s}\right) = 1 - \alpha + \sqrt {\left(\log (2 / \delta) / (2 n)\right)}, \tag {1}
$$
where  $\delta \in (0,1)$  is the violation rate describing the probability that the (FPR) exceeds  $\alpha$ , and  $\hat{F}_{\mathrm{CLEAN}}$  is the empirical cumulative distribution function of the scores on the validation data  $\{s(T(x_i))\}_{i=1}^n$ . In the case where  $\sqrt{(\log(2/\delta)/(2n))} > \alpha$ , we set the thresholding value  $\tau$  to be the maximum of  $\{s(T(x_i))\}_{i=1}^n$ .
Algorithm 1 Conformal Backdoor Detection (CBD)