SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhen Xiang Zidi Xiong Bo Li University of Illinois Urbana-Champaign {zxiangaa, zidix2, lbo}@illinois.edu
Backdoor attack is a common threat to deep neural networks. During testing, samples embedded with a backdoor trigger will be misclassified as an adversarial target by a backdoored model, while samples without the backdoor trigger will be correctly classified. In this paper, we present the first certified backdoor detector (CBD), which is based on a novel, adjustable conformal prediction scheme based on our proposed statistic local dominant probability. For any classifier under inspection, CBD provides 1) a detection inference, 2) the condition under which the attacks are guaranteed to be detectable for the same classification domain, and 3) a probabilistic upper bound for the false positive rate. Our theoretical results show that attacks with triggers that are more resilient to test-time noise and have smaller perturbation magnitudes are more likely to be detected with guarantees. Moreover, we conduct extensive experiments on four benchmark datasets considering various backdoor types, such as BadNet, CB, and Blend. CBD achieves comparable or even higher detection accuracy than state-of-the-art detectors, and it in addition provides detection certification. Notably, for backdoor attacks with random perturbation triggers bounded by  $\ell_2 \leq 0.75$  which achieves more than  $90\%$  attack success rate, CBD achieves  $100\%$  ( $98\%$ ),  $100\%$  ( $84\%$ ),  $98\%$  ( $98\%$ ), and  $72\%$  ( $40\%$ ) empirical (certified) detection true positive rates on the four benchmark datasets GTSRB, SVHN, CIFAR-10, and TinyImageNet, respectively, with low false positive rates.
Despite the success of deep neural networks (DNNs) in many applications, they are vulnerable to adversarial attacks such as backdoor attacks [46, 36]. A DNN being backdoored will learn to predict an adversarial target class for test samples embedded with a backdoor trigger, while maintaining a high accuracy on clean test samples without the trigger [18, 8, 42, 65, 50, 43, 51, 37].
Backdoor detection is a popular task for backdoor defense. It aims to detect if a given model is backdoorsed without access to the training set or any real test samples that are possibly embedded with the trigger [48, 64]. The task corresponds to practical scenarios where the user of an app or a legacy system containing a DNN seeks to know if the model is backdoor attacked, where the training set is not available [1]. Various empirical approaches have been proposed for backdoor detection, most of which are based on trigger reverse engineering [67, 6, 75, 69, 56, 62, 24], or meta classification [82, 29]. However, none of these works quantitatively investigate the conditions under which the backdoor attacks are guaranteed to be detectable. Without a detection guarantee, DNNs are still vulnerable to future attacks (e.g.) with new trigger types [51, 87, 70].
In this paper, we make the first attempt toward the certification of backdoor detection. Certification is an important concept for studies on the robustness of DNNs against adversarial examples [61, 17, 52, 7, 45]. In particular, a robustness certification refers to a probabilistic or deterministic guarantee for a model to produce desired outputs (e.g. correct label prediction) when the adversarial perturbation applied to the inputs satisfies certain conditions (e.g. with some constrained perturbation magnitude) [33, 58, 32, 10, 59]. As an analogy, we propose a certified backdoor detector (CBD)
that is guaranteed to trigger an alarm if the attack for a given domain satisfies certain conditions. In Sec. 3, we introduce the detection procedure of CBD based on conformal prediction (with our proposed, optional adjustment scheme) using a novel (model-level) statistic named local dominant probability. The calibration set for conformal prediction is obtained from a small number of benign shadow models trained on a small validation set, which addresses the unavailability of the training set. In Sec. 4, we derive the condition for attacks with detection guarantee, as well as a probabilistic upper bound for the false positive rate of our detector, for any prescribed significance level specifying the aggressiveness of the detection. Notably, our certification is comprehensive – for any domain, more effective attacks with strong trigger robustness (which measures the resilience of a trigger against test-time noises) and more stealthy attacks (against human inspectors) with small trigger perturbation magnitudes are easier to be detected with guarantees. Moreover, both our detector and the certification method do not assume the trigger incorporation mechanism or the training configuration of the model, which allows their potential application to future attacks. Our contributions are summarized below:
- We propose CBD, the first certified backdoor detector, which is based on an adjustable conformal prediction scheme using a novel local dominant probability statistic.
- We propose a certification method and show that for any domain, backdoor attacks with stronger trigger robustness and smaller trigger perturbation magnitudes are more likely to be detected by CBD with guarantee. We also derive a probabilistic upper bound for the false positive rate of CBD.
- We show that CBD achieves comparable or even higher detection accuracy than state-of-the-art detectors against three types of backdoors. We also show that for backdoor attacks with random perturbation triggers bounded by  $\ell_2 \leq 0.75$ , CBD achieves  $100\%$  (98%),  $100\%$  (84%),  $98\%$  (98%), and  $72\%$  (40%) empirical (certified) true positive rates on GTSRB, SVHN, CIFAR-10, and TinyImageNet, respectively, with only  $0\%$ ,  $0\%$ ,  $6\%$ , and  $10\%$  false positive rates, respectively.
Backdoor Detection Methods Existing methods for backdoor detection are all empirical without theoretical guarantees. An important type of reverse-engineering-based method estimates putative triggers for anomaly detection [67, 6, 75, 41, 78, 73, 79, 76, 69, 56, 62, 24]. Certification for these methods is hard due to the complexity of trigger reverse engineering. Another type of detector is based on meta-classification that involves a large number of shadow models trained with and without attacks [82, 29]. Differently, our CBD is based on conformal prediction involving a scalar detection statistic (rather than a high dimensional feature vector employed by [82, 29]), such that only a small number of benign shadow models will be required. More importantly, our CBD is certified, i.e. with a detection guarantee, which is different from all the detention methods mentioned above.
Other Backdoor Defense Tasks Backdoor defenses during training aim to produce a backdoor-free classifier from the possibly poisoned training set [63, 5, 74, 14, 25, 57, 77, 23, 4, 16]. These defenses require access to the training set, which is unavailable for backdoor detection. Backdoor mitigation aims to "repair" models being backdoor attacked [40, 72, 35, 19, 88, 85], which can be viewed as a downstream task following backdoor detection. Inference-stage trigger detection aims to detect if a test sample contains the trigger [15, 55, 12, 9, 44, 34]. However, backdoor detection is performed before the inference stage, where test samples are not available. These defense tasks will not be further discussed due to their fundamental differences from the backdoor detection task.
Certified Backdoor Defenses Existing methods mostly modify the training process to prevent the backdoor from being learned, while manipulating the test sample to destroy any potentially embedded triggers [86, 71, 27, 53]. These methods are not applicable to the backdoor detection problem where both the training set and the test samples are not available. More importantly, all these existing certified defenses are deployed during training, which requires an uncontaminated training process fully controlled by the defender. In contrast, in this paper, we consider a stronger threat model that allows the attacker to have full control of the training process.
3 Detection Method
Threat Model Consider a classification domain with sample space  $\mathcal{X}$  and label space  $\mathcal{Y}$ . A backdoor attack is specified by a trigger with some incorporation function  $\delta : \mathcal{X} \to \mathcal{X}$  and a target class  $t \in \mathcal{Y}$ . For a successful backdoor attack, the victim classifier will predict to the target class  $t$  whenever a
test sample is embedded with the trigger, while test samples without the trigger will be correctly classified [18, 8, 42, 38]. In this paper, we do allow advanced attackers with full control of the training process [51, 87, 70, 50, 3]. This is deemed a stronger threat model than all previous works on certified backdoor defense where the defender is assumed with full control of the training process. However, in this paper, we do not consider backdoor attacks with multiple triggers or target classes [84, 83] – even empirical detection of these attacks is a challenging problem [80].
Goal of Certified Backdoor Detection The fundamental goal is backdoor detection, i.e. to infer if a given classifier  $f(\cdot; w) : \mathcal{X} \to \mathcal{Y}$  is backdoored or not [48, 64]. The defender has no access to the training set or any test samples that may contain the trigger. In practice, the defender also has no access to benign classifiers with high accuracy for the same domain as  $f(\cdot; w)$  – otherwise, these benign classifiers can be used for the task, and detection will be unnecessary. However, the defender is assumed with a small validation set of clean samples for detection – this is a standard assumption made by most post-training backdoor defenses [67, 82, 75, 56, 41].
Beyond model inference, a certified backdoor detector also needs to provide each classification domain (associated with the model to be inspected) with a condition on  $\delta$ ,  $t$ , and  $w$ , such that any successful backdoor attack with a trigger  $\delta$  and a target class  $t$  on a victim classifier  $f(\cdot; w)$  (trained on this classification domain) is guaranteed to be detected if the condition is satisfied. The stronger a certification is, the more likely a successful attack on the domain will be detectable with guarantee. Moreover, a certification has to be associated with a guarantee on the false positive rate; otherwise, an arbitrarily strong certification can be achieved by increasing the aggressiveness of the detection rule.
3.2 Overview of CBD Detection
Key Intuition For a successful backdoor attack with a trigger  $\delta$  and a target class  $t$ , a test instance  $x$  embedded with the trigger (denoted by  $\delta(x)$ ) will be classified to the target class  $t$  with high probability. Practical backdoor triggers should also be robust (i.e. resilient) to noises either from the environment or introduced by simple defenses based on input-preprocessing such as blurring and/or quantization [81, 39]. Such trigger robustness can be measured by the distribution of the model prediction in the neighborhood of each  $\delta(x)$  – the more robust the trigger is, the more samples in the neighborhood of  $\delta(x)$  will be predicted to the target class  $t$ . Thus, if the perturbation magnitude of a robust trigger, i.e.  $||\delta(x) - x||_2$ , is small (which is usual for backdoor attacks to achieve stealthiness), a significant proportion of samples in the neighborhood of  $x$  will also be classified to class  $t$  due to their closeness to  $\delta(x)$ . Such an increment in the target class probability in the neighborhood of all instances is captured by our proposed statistic named local dominant probability (LDP) to distinguish backdoors classifiers from benign ones – the former tend to have a larger LDP than the latter.
Outline of CBD Detection Procedure In short, CBD performs an adjustable conformal prediction to test if the LDP statistic computed for the model to be inspected is sufficiently large to trigger an alarm. A small number of shadow models (with the same architecture as the model to be inspected) are trained on a relatively small validation set, with an LDP computed for each shadow model to form a calibration set. Then, a small proportion of outliers with large values in the calibration set are optionally removed, e.g., by anomaly detection. Based on this adjusted calibration set, the p-value is computed for the LDP for the model to be inspected. The model is deemed to be backdoored if the p-value is less than some prescribed significance level (e.g. the classical 0.05 for statistical testing).
3.3 Definition of LDP
We first define a samplewise local probability vector (SLPV) to represent the distribution of the model prediction outcomes in the neighborhood of any given sample. Then, based on SLPV, we define the samplewise trigger robustness (STR) that measures the resilience of a trigger to the Gaussian noise when it is embedded in a particular sample. Finally, we define the LDP statistic based on SLPV.
Definition 3.1. (Samplewise Local Probability Vector (SLPV)) For any classifier  $f(\cdot; w) : \mathcal{X} \to \mathcal{Y}$  with parameters  $w$ , the SLPV for any input  $x \in \mathcal{X}$  is a probability vector  $\pmb{p}(x|w, \sigma) \in [0,1]^K$  over the  $K = |\mathcal{Y}|$  classes, with the  $k$ -th entry defined by  $p_k(x|w, \sigma) \triangleq \mathbb{P}_{\epsilon \sim \mathcal{N}(0,\sigma^2 I)}(f(x + \epsilon; w) = k)$ ,  $\forall k \in \mathcal{Y}$ , where  $\mathcal{N}(\mu, \Sigma)$  represents Gaussian distribution with mean  $\mu$  and covariance  $\Sigma$ .
Remarks: If  $f(\cdot; w)$  is continuous at  $x$  with  $f(x; w) = k$  for some  $k \in \mathcal{V}$ , it is trivial to show that  $p_k(x|w, \sigma) \to 1$  as  $\sigma \to 0$ , i.e. the SLPV becomes a singleton at the predicted class of  $x$  without the Gaussian noise. In practice, SLPV can be estimated by Monte Carlo for any given model and sample.
(a) Benign classifier with a small LDP close to  $\frac{1}{4}$ .
(b) Classifier being backdoored with a large LDP.
Figure 1: Illustration of the difference in LDP between benign and backdoored classifiers on a classification domain with  $K = 4$  classes. A backdoor attack with a robust trigger  $\delta$  (with a small perturbation magnitude) and a target class 4 (with orange decision region) changes the class distribution in the neighborhood of  $x_{1}, x_{2}$ , and  $x_{3}$ , resulting in a larger LDP for the backdoored model than for the benign model.
Definition 3.2. (Samplewise Trigger Robustness (STR)) Consider a backdoor attack with a trigger  $\delta$  and a target class  $t\in \mathcal{V}$  against a victim model  $f(\cdot ;w)$ . For any sample  $x\in \mathcal{X}$  and any isotropic Gaussian distribution  $\mathcal{N}(0,\sigma^2 I)$ , the STR is defined by the  $t$ -th entry of the SLPV for  $\delta (x)$  (i.e. sample  $x$  with the trigger  $\delta$  embedded), which is denoted by  $R_{\delta ,t}(x|w,\sigma)\triangleq p_t(\delta (x)|w,\sigma)$ .
Remarks: STR measures the resilience of a trigger against Gaussian noises. Usually, strong STR can be naturally achieved by embedding the trigger in a large variety of samples during training.
Definition 3.3. (Local Dominant Probability (LDP)) Consider a domain with  $K = |\mathcal{Y}|$  classes and an isotropic Gaussian distribution  $\mathcal{N}(0, \sigma^2 I)$ . The LDP for a classifier  $f(\cdot; w)$  is defined by
$$
s (w) = \left\| \frac {1}{K} \sum_ {k = 1} ^ {K} \boldsymbol {p} \left(x _ {k} | w, \sigma\right) \right\| _ {\infty} \tag {1}
$$
where  $x_{1},\dots ,x_{K}$  are  $K$  independent random samples satisfying  $f(X_{k};w) = k,\forall k\in \{1,\dots ,K\} .$
Remarks: By the definition, LDP for a given classifier  $f(\cdot; w)$  is computed on  $x_1, \dots, x_K$  independently sampled from the  $K$  classes respectively, satisfying  $f(x_k; w) = k$  for  $\forall k$ . LDP computed on more samples per class yields similar detection and certification performance empirically. Specifically, we first compute the SLPV for each of the  $K$  samples. Then we take the average of the  $K$  SLPVs and pick the maximum entry as the LDP for  $f(\cdot; w)$ . Note that LDP is always no less than  $\frac{1}{K}$ .
As stated in the key intuitions in Sec. 3.2, LDP tends to be larger for backdoored classifiers than for benign ones. This can be understood from the illustration in Fig. 1 for a classification domain with  $K = 4$  classes. For the benign classifier on the left, the SLPVs for  $x_{1},\dots ,x_{4}$  are almost orthogonal to each other, leading to a small LDP close to  $\frac{1}{4}$ . On the right, we consider a backdoor attack with a robust trigger  $\delta$  and a target class 4 (with orange decision region). The strong STRs for  $x_{1},x_{2}$ , and  $x_{3}$  (represented by the large orange regions around  $\delta (x_{1}),\delta (x_{2})$ , and  $\delta (x_{3})$ ), together with the small trigger perturbation magnitudes (i.e. small  $||\delta (x_i) - x_i||_2$  for  $i = 1,2,3$ ), significantly change the class distribution in the neighborhood of each of  $x_{1},x_{2}$ , and  $x_{3}$ . In particular, there will be a clear increment in the 4-th entry of the SLPVs for  $x_{1},x_{2}$ , and  $x_{3}$ . Thus, the 4-th entry associated with the backdoor target class will dominate the average SLPV over  $x_{1},\dots ,x_{4}$ , leading to a large LDP.
3.4 CBD Detection Procedure
Although backdoored and benign classifiers have different LDP distributions, it is still a challenge in practice to set a detection threshold. To solve this problem, we propose to use the conformal prediction, which employs a calibration set for supervision [66, 2]. Here, the calibration set consists of LDP statistics obtained from a small number of benign shadow models trained on the small validation set possessed by the defender. However, the actual benign classifiers to be inspected are usually trained on more abundant data, such that the LDPs for these classifiers will likely follow a different distribution from the LDPs for the shadow models without sufficient training. In particular, the LDPs in the calibration set (obtained from the shadow models) may easily have an overly large sample variance and a heavy tail of large outliers. Directly using this calibration set for conformal prediction may lead to a conservative detection due to an overly large detection threshold.
Thus, we propose an optional adjustment scheme that treats the  $m$  largest LDP statistics in the calibration set as outliers. In practice, prior knowledge may be required to determine the exact value
of  $m$ , while in our experiments, a small  $m / N$  ( $\leq 0.2$ , where  $N$  is the size of the calibration set) may significantly increase the detection or certification power of our CBD with only small increment to the false positive rate. The detection procedure of our CBD consists of the following four steps:
1) Given a classifier  $f(\cdot; w)$  to be inspected, estimate LDP  $s(w)$  based on Def. 3.3.
2) Train (benign) shadow models  $f(\cdot; w_1), \dots, f(\cdot; w_N)$  on the clean validation dataset, and construct a calibration set  $S_N = \{s(w_1), \dots, s(w_N)\}$  by computing the LDP for each model.
3) Compute the adjusted conformal p-value (assuming  $m$  large outliers) defined by:
$$
q _ {m} (w) = 1 - \frac {1 + \min  \left\{\left| \left\{s \in \mathcal {S} _ {N} : s <   s (w) \right\} \right| , N - m \right\}}{N - m + 1} \tag {2}
$$
4) Trigger an alarm if  $q_{m}(w) \leq \alpha$ , where  $\alpha$  is a prescribed significance level (e.g.  $\alpha = 0.05$ ).
4 CBD Certification
In addition to a detection inference, CBD also provides a certification, which is a condition under which attacks are guaranteed to be detectable. Detailed proofs in this section are shown in App. A.
Theorem 4.1. (Backdoor Detection Guarantee) For an arbitrary classifier  $f(\cdot ;w):\mathcal{X}\to \mathcal{Y}$  to be inspected, let  $x_{1},\dots ,x_{K}$  be the  $K$  randomly selected samples and  $\hat{\mathcal{N}} (0,\sigma^2 I)$  be the isotropic Gaussian distribution used to compute the LDP for  $f(\cdot ;w)$ . Let  $\alpha$  be the prescribed significance level of CBD. A backdoor attack with a trigger  $\delta$  and a target class  $t$  is guaranteed to be detected if:
$$
\Delta <   \sigma \left(\Phi^ {- 1} \left(1 - s _ {\left(N - m - \lfloor \alpha (N - m + 1) \rfloor\right)}\right) - \Phi^ {- 1} (1 - \pi)\right) \tag {3}
$$
where  $\Phi$  is the standard Gaussian CDF,  $\pi = \min_{k=1,\dots,K} R_{\delta,t}(x_k|w,\sigma)$  is the minimum STR over  $x_1,\dots,x_K$ ,  $\Delta = \max_{k=1,\dots,K} ||\delta(x_k) - x_k||_2$  is the maximum perturbation magnitude of the trigger,  $m$  is the number of assumed outliers in the calibration set  $\mathcal{S}_N$  with size  $N$ , and  $s_{(n)}$  denotes the  $n$ -th smallest element in  $\mathcal{S}_N$ .
Proof (sketch). The STR for each sample  $x_{k}$  equals the  $t$ -th entry of the SLPV for  $\delta(x_{k})$  by Def. 3.2. We also connect the SLPV for  $\delta(x_{k})$  to the SLPV for  $x_{k}$  using the Neyman-Pearson lemma ([49]). Based on this connection, we derive the lower bound for the minimum STR, such that the  $t$ -th entry of the SLPV of each  $x_{k}$  is sufficiently large to result in a large LDP for the attack to be detected.
Remarks: (1) (Main Results) For fixed trigger perturbation size  $(\Delta)$ , detection of attacks with larger STR  $(\pi)$  is more likely to be guaranteed; while for fixed STR, detection of attacks with smaller trigger perturbation size is more likely to be guaranteed. (2) Our backdoor detection guarantee is inspired by the randomized smoothing approach in [10] for certified robustness against adversarial examples. However, certified backdoor detection and certified robustness against adversarial examples are fundamentally different, as will be detailed in App. I. (3) Certified backdoor detection and certified robustness against backdoors complement each other. The former provides detection guarantees to strong backdoor attacks, while the latter prevents the trigger from being learned during training [86, 53]. The two types of certification may cover the entire attack space together in the future, such that a backdoor attack will be either strong enough to be detected or weak enough to be disabled.
A meaningful certification for backdoor detection should also be along with a guarantee for the false positive rate (FPR). Otherwise, one can easily design a backdoor detector that always triggers an alarm, which provides detection guarantees to all backdoor attacks but is useless in practice.
Theorem 4.2. (Probabilistic Upper Bound for FPR) Consider a random calibration set  $S_N = \{s_1, \dots, s_N\}$  with  $s_1, \dots, s_N$  i.i.d. following some continuous distribution  $F$ . Consider a random benign classifier  $f(\cdot; W)$  with LDP  $s(W)$  following some distribution  $\tilde{F}$ . Assume  $F$  dominates  $\tilde{F}$  in the sense of first-order stochastic dominance. Let  $m$  be any number of assumed outliers in  $S_N$  and let  $\alpha$  be any prescribed significance level of CBD. Then, the FPR of CBD on  $f(\cdot; W)$  conditioned on  $S_N$ , which is denoted by  $Z_N = \mathbb{P}(q_m(W) \leq \alpha | S_N)$  based on Eq. (2), will be first-order dominated by a random variable  $B$  following  $\mathrm{Beta}(m + l + 1, N - m - l)$  with  $l = \lfloor \alpha (N - m + 1) \rfloor$ , i.e.  $B \succeq_1 Z_N$ . In other words,  $\mathbb{P}(Z_N \leq q) \geq \mathbb{P}(B \leq q)$  for any real  $q$ .
Proof(sketch). We first express the false positive rate  $Z_{N}$  in terms of the order statistics on the elements of the random calibration set  $S_{N}$ . Then, we derive the lower bound of the CDF of  $Z_{N}$  using the distribution of order statistics followed by a binomial expansion.