SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remarks: The assumption that  $F$  dominates  $\tilde{F}$  in Thm. 4.2 generally holds in practice. Again, this is because the actual benign classifiers to be inspected are usually trained on more abundant data than the benign shadow models. Empirical results supporting this assumption are shown in Sec. 5.4. Moreover, an analysis of this phenomenon is presented in App. B, where we show on binary Bayes classifiers that a higher empirical loss can easily lead to a larger expected LDP.
While we have shown that  $Z_{N}$  conditioned on a random calibration set of size  $N$  is upper bounded by a Beta random variable in the sense of first-order stochastic dominance, in Col. 4.3 below, we show that asymptotically, the upper bound of  $Z_{N}$  converges to a constant in probability as  $N \to \infty$ .
Corollary 4.3. (Asymptotic Property of FPR) Consider the settings in Thm. 4.2. For any  $\xi >0$ ,  $\lim_{N\to +\infty}\mathbb{P}(Z_N\leq \tau) = 1$ , where  $\tau = \alpha +(1 - \alpha)\beta +\xi$  with  $\beta = m / N$ .
Proof (sketch). We show that any random variable  $B$  with the Beta distribution described in Thm. 4.2 satisfies  $\lim_{N\to +\infty}\mathbb{P}(B\leq \tau) = 1$ . Then, the corollary is proved since  $Z_{N}$  is dominated by  $B$ .
Remarks: For classical conformal prediction without adjustment where  $m = 0$ , we will have  $\beta = 0$  and  $\lim_{N\to +\infty}\mathbb{P}(Z_N\leq \alpha +\xi) = 1$  for any  $\xi >0$ . In this case, the upper bound of the false positive rate of our CBD converges to the prescribed significance level  $\alpha$  in probability.
There have been many different types of backdoor attacks proposed, each also with a wide range of configurations. Thus, it is infeasible to evaluate our CBD over the entire space of backdoor attacks. Inspired by the evaluation protocol for certified robustness [10], in Sec. 5.1, we focus on backdoor attacks with random perturbation triggers to comprehensively evaluate the certification capability of our CBD. In Sec. 5.2, we compare CBD with three state-of-the-art backdoor detectors (all "uncertified") against backdoor attacks with three popular trigger types to evaluate the detection capability of CBD. In Sec. 5.3, we present ablation studies (e.g. on the number of shadow models used by CBD). Additional results and other supportive empirical analyses are shown in Sec. 5.4.
5.1 Evaluation of CBD Certification
For certified robustness, the prediction of a test example is unchanged with a guarantee, if the magnitude of the adversarial perturbation is smaller than the certified radius [10]. Certified robustness is usually evaluated by the certified accuracy on some random test set as the proportion of the samples that are guaranteed to be correctly classified if the magnitude of the adversarial perturbation is no larger than some prescribed value. As an analog, our certification for backdoor detection is specified by an inequality that involves both the STR and the perturbation magnitude of the trigger, which naturally produces a two-dimensional "certified region" (illustrated in Fig. 7 and Fig. 8 in Apdx. D). Our certification method is evaluated on a set of random backdoor attacks, each using a random pattern as the trigger, with the perturbation magnitude satisfying some  $\ell_2$  constraint. We are interested in the proportion of the attacks falling into the certified region, i.e. guaranteed to be detected.
5.1.1 Setup
Dataset: Our experiments are conducted on four benchmark image datasets, GTSRB [60], SVHN [47], CIFAR-10 [30], and TinyImageNet [11], following their standard train-test splits. Due to the large number of models that will be trained to evaluate our certification method, except for GTSRB, we use  $40\%$  of the training set to train these models. We also reserve 5,000 samples from the test set of GTSRB, SVHN, and CIFAR-10, and 10,000 samples from the test set of TinyImageNet (much smaller than the training size for the models for evaluation) for the defender to train the shadow models. More details about these datasets are deferred to App. C.1.
Evaluation Metric: Following the convention, the detection performance of CBD is evaluated by the true positive rate (TPR) defined by the proportion of attacks being detected (with a correct inference of the target class) and the false positive rate (FPR) defined by the proportion of benign classifiers falsely detected. For certification, we define certified TPR (CTPR) as the proportion of a set of attacks that are guaranteed to be detectable, i.e. falling into the certified region.
Attack Setting: For each dataset, we create 50 backdoor attacks, each with a randomly selected target class. For each attack, we generate a random trigger, which is a random perturbation embedded
Figure 2: Certification performance of CBD against backdoor attacks with random triggers with perturbation magnitude  $\ell_{2} \leq 0.75$  measured by CTPR (solid) for a range of  $\sigma$  for  $\beta = 0, 0.1, 0.2$ . The CTPRs are all upper-bounded by the TPRs (dashed), showing the correctness of our certification. Notably, CBD achieves up to  $98\%$  ( $100\%$ ),  $84\%$  ( $100\%$ ),  $98\%$  ( $98\%$ ), and  $40\%$  ( $72\%$ ) CTPRs (TPRs) on GTSRB, SVHN, CIFAR-10, and TinyImageNet, respectively, across all choices of  $\sigma$  and  $\beta$ . An increment in  $\beta$ , the assumed ratio of calibration outliers, may lead to further increments in both CTPR and TPR. The hyperparameter  $\sigma$  can be determined using the calibration set in practice.
by  $\delta(x) = x + v$  with  $||v||_2 \leq 0.75$ . The generation process also involves random nullification of the trigger pattern, which helps the trigger to be learned. More details about trigger generation are deferred to App. C.2 due to space limitations. The poisoning ratios for the attacks on GTSRB, SVHN, CIFAR-10, and TinyImageNet are  $7.8\%$ ,  $15.3\%$ ,  $11.3\%$ , and  $12.4\%$ , respectively. Lower poisoning ratios may largely reduce the attack success rate since many randomly generated triggers are relatively hard to learn. Results for triggers with larger perturbation size are shown in App. D.
Training: For model architecture, we use the winning model on the leaderboard [31] for GTSRB, MobileNetV2 [54] for SVHN, the same architecture in [82] for CIFAR-10, and ResNet-34 [22] for TinyImageNet. For each dataset, 50 benign models are trained (also on the  $40\%$  training set except for GTSRB) to evaluate the FPR. The accuracies for these benign models are roughly  $98\%$ ,  $92\%$ ,  $78\%$ , and  $47\%$  on GTSRB, SVHN, CIFAR-10, and TinyImageNet, respectively. For each attack, we train a model with  $\geq 90\%$  attack success rate and  $\leq 2\%$  degradation in the benign accuracy (or re-generate the attack for training until both conditions are satisfied). More details about the training configurations are shown in App. C.3. Finally, we train 100 shadow models for each of GTSRB, SVHN, and CIFAR-10, and 50 shadow models for TinyImageNet using the same architectures and configurations as above – these shadow models are used by our CBD for detection and certification.
5.1.2 Certification Performance
In Fig. 2, we show the CTPR (solid) of our CBD against the attacks with the random perturbation trigger on the four datasets for a range of  $\sigma$  and for  $\beta = m / N\in \{0,0.1,0.2\}$ . The TPR (dashed) for each combination of  $\sigma$  and  $\beta$  is also plotted for reference. Recall that  $\beta$  here represents the proportion of assumed outliers in the calibration set for the adjustment of the conformal prediction, and  $\sigma$  is the standard deviation of the isotropic Gaussian distribution for the LDP computation. In our experiments, 1024 random Gaussian noises are generated for each sample used to compute the LDP. The significance level for conformal prediction is set to the classical  $\alpha = 0.05$  for statistical testing.
In general, our certification is effective, which covers up to  $98\%$ ,  $84\%$ , and  $98\%$  backdoor attacks with the random perturbation trigger (all achieved with  $\beta = 0.2$ ) on GTSRB, SVHN, and CIFAR-10, respectively. Even for the very challenging TinyImageNet dataset, CBD certifies up to  $40\%$  of these attacks. Moreover, for all choices of  $\beta$  and  $\sigma$ , CTPR is upper-bounded by TPR. For example, for the aforementioned CTPRs on the four datasets, the corresponding TPRs are  $100\%$ ,  $100\%$ ,  $98\%$ , and  $72\%$ , respectively (with FPRs  $0\%$ ,  $0\%$ ,  $6\%$ , and  $10\%$ , respectively, see App. D). In fact, all attacks with the detection guarantee are detected empirically, showing the correctness of our certification.
We also make the following observations regarding the hyperparameters  $\beta$  and  $\sigma$ : 1) An increment in  $\beta$  may lead to an increment in both CTPR and TPR. This is due to the existence of a heavy tail
(corresponding to large outliers) in the LDP distribution for the calibration set. While an overly large  $\beta$  may cause a significant increment to FPR, our additional results in App. D show that empirically, this is not the case for  $\beta \leq 0.2$ . 2) Each domain has its own proper range for the choice of  $\sigma$ . In general, the detection power of CBD (reflected by TPR) reduces as  $\sigma$  becomes overly small. This phenomenon agrees with our remarks on Def. 3.1 - SLPV converges to a singleton at the labeled class as  $\sigma$  approaches zero, regardless of the presence of a backdoor attack. Moreover, the certification power (reflected by CTPR) also reduces for small  $\sigma$ 's, which matches the inequality (3) in Thm. 4.1. For overly large  $\sigma$ 's, on the other hand, LDP for benign classifiers may also grow and possibly approach one, resulting in a large FPR. However, in this case, LDP is no longer computed in a truly 'local' context, contrary to the intuition implied by its name. In Sec. 5.2, we will introduce a practical scheme to choose a proper  $\sigma$  for each domain based on the shadow models.
5.2 Evaluation of CBD Detection
Here, we show the detection performance of CBD against backdoor attacks with various trigger types, including the BadNet square [18], the "chessboard" (CB) pattern [75], and the blended pattern [8]. For each of GTSRB, SVHN, and CIFAR-10, we train 20 models (using the full training set) for each trigger type following the same training configurations described in Sec. 5.1.1. TinyImageNet is not considered here due to the high training cost. Details for each trigger type and attack settings are shown in App. C.4. We also compare CBD with three state-of-the-art backdoor detectors without certification, which are Neural Cleanse (NC) [67], K-Arm [56], and MNTD [82]. In particular, K-Arm and MNTD require manual selection of the detection threshold. For both of them, we choose the threshold that maximizes the TPR while keeping a  $5\%$  FPR for each dataset. Moreover, we devise a "supervised" version of CBD, named  $\mathrm{CBD}_{\mathrm{sup}}$ , which still uses LDP as the detection statistic but without the conformal prediction. The detection threshold for  $\mathrm{CBD}_{\mathrm{sup}}$  is determined in the same way as for K-Arm and MNTD, i.e. by maximizing the TPR with a controlled  $5\%$  FPR.
In practice, CBD needs to choose a moderately large  $\sigma$  for each detection task. To this end, we first initialize a small  $\sigma$  such that for each of the  $N$  shadow models, the SLPVs for the  $K$  samples used for computing the LDP all concentrate at the labeled classes. In this case, the LDPs for all the shadow models are close to  $\frac{1}{K}$ . Then, we gradually increase  $\sigma$  until  $\frac{1}{N \times K} \sum_{n=1}^{N} \sum_{k=1}^{K} p_k(x_k^{(n)}|w_n, \sigma) < \psi$  for some relatively small  $\psi$ , where  $x^{(n)}$  is the  $k$ -th sample for LDP computation for the  $n$ -th model, i.e. the SLPVs are no longer concentrated at the labeled classes. In the left of Fig. 4, we show the choice of  $\sigma$  based on the above scheme for a range of  $\psi$ , which exhibits a trend of convergence as  $\psi$  decreases. We also notice that  $\sigma$  selected for  $\psi < 0.2$  roughly matches the  $\sigma$  choices in Fig. 2 that yields relatively high CTPR and TPR, showing the effectiveness of our scheme. In our evaluation of CBD detection, we set  $\psi = 0.1$ , which yields  $\sigma = 1.15, 0.39, 1.14$  for GTSRB, SVHN, and CIFAR-10, respectively. Other choices of  $\sigma$  for  $\psi$  less than 0.2 yield similar detection performance.
As shown in Tab. 1, CBD achieves comparable or even higher TPRs than the SOTA detectors (that benefit from unrealistic supervision) for all trigger types on all datasets. CBD also provides non-trivial detection guarantees to most attack types on these datasets. The relatively low CTPRs, e.g. for BadNet on GTSRB, are due to the large perturbation magnitude of the trigger. The even better TPRs achieved by  $\mathrm{CBD}_{\mathrm{sup}}$  though with the same supervision as for the SOTA detectors, show the effectiveness of the LDP statistic in distinguishing backdoored models from benign ones. Such effectiveness is further verified by the receiver operating characteristic (ROC) curves for  $\mathrm{CBD}_{\mathrm{sup}}$ . Compared with the baseline detectors,  $\mathrm{CBD}_{\mathrm{sup}}$  achieves generally higher overall areas under curves (AUCs) across the three datasets.
We show that the time efficiency and data efficiency of CBD can be improved by training fewer shadow models and using fewer samples for training the shadow models, respectively, without significant degradation in the detection or certification performance. In particular, we show in Tab. 2 that CBD with  $\beta = 0.2$  achieves similar TPRs and CTPRs for the same set of attacks in Sec. 5.2 when we reduce the number of shadow models from 100 to 50, 25, and 10, respectively. Such robustness of CBD to the calibration size further verifies the clear separation between the benign and backdoored classifiers using our proposed LDP. In Tab. 3, we show that with shadow models trained on only 100
Table 1: Certified detection of CBD for  $\beta = 0,0.1,0.2$  (shaded), with the empirical detection performance (measured by TPR  $(\%)$ ) compared with NC, K-Arm, MNTD, and  $\mathrm{CBD}_{\mathrm{sup}}$  against BadNet, CB, and Blend attacks on GTSRB, SVHN, and CIFAR-10. The parentheses in each shaded cell contain the CTPR  $(\%)$  associated with the TPR outside the parentheses. CBD achieves comparable or even higher empirical TPRs compared with the SOTA baselines and provides non-trivial (or even tight) certification for different attacks and datasets. FPRs  $(\%)$  are reported on benign classifiers.
GTSRB	SVHN	CIFAR-10	Average TPR
benign	BadNet	CB	Blend	benign	BadNet	CB	Blend	benign	BadNet	CB	Blend
NC	20	50	75	20	40	80	100	95	20	35	95	60	67.8
K-Arm	5	100	100	100	5	100	70	40	5	100	80	55	82.8
MNTD	5	20	0	0	5	10	10	15	5	90	100	75	35.6
CBDsup	5	100	95	100	5	100	100	90	5	65	100	55	89.4
CBD0	0	75 (5)	95 (80)	80 (20)	0	75 (45)	100 (100)	80 (75)	0	50 (5)	100 (90)	45 (30)	77.2
CBD0.1	0	90 (15)	95 (85)	90 (25)	0	90 (55)	100 (100)	80 (80)	20	75 (20)	100 (95)	55 (35)	86.1
CBD0.2	0	90 (15)	95 (85)	95 (35)	0	95 (65)	100 (100)	90 (80)	25	75 (25)	100 (100)	60 (40)	88.9
AUC comparison
GTSRB	SVHN	CIFAR-10
MNTD	0.11	0.49	0.97
K-Arm	1.0	0.79	0.91
CBDsup	0.98	0.97	0.89
Figure 3: Receiver operating characteristic (ROC) curves of  $\mathrm{CBD}_{\mathrm{sup}}$  aggregated over all three trigger types on GTSRB, SVNH, and CIFAR-10, respectively.  $\mathrm{CBD}_{\mathrm{sup}}$  with our proposed LDP statistic achieves higher overall areas under curves (AUCs) than K-Arm and MNTD across the three datasets.
Table 2: Certified detection of CBD (with CTPR inside the parentheses and TPR outside) for  $\beta = 0.2$  using 10, 25, 50, and 100 (the default in Sec. 5.2) shadow models. Both the detection and certification performance of CBD are not significantly affected by reducing the number of shadow models.
# shadow	GTSRB	SVHN	CIFAR-10
benign	BadNet	CB	Blend	benign	BadNet	CB	Blend	benign	BadNet	CB	Blend
10	0	95 (10)	95 (85)	95 (35)	0	75 (45)	100 (100)	90 (75)	25	85 (25)	100 (100)	55 (40)
25	0	95 (5)	95 (85)	95 (25)	0	95 (45)	100 (100)	80 (75)	25	80 (25)	100 (100)	70 (40)
50	0	95 (15)	95 (85)	95 (40)	0	95 (65)	100 (100)	85 (75)	25	80 (25)	100 (100)	60 (40)
100	0	90 (15)	95 (85)	95 (35)	0	95 (65)	100 (100)	90 (80)	25	75 (25)	100 (100)	60 (40)
Table 3: Certified detection of CBD (with CTPR inside the parentheses and TPR outside) for  $\beta = 0.2$  and  $\beta = 0.4$ , respectively, with 100 shadow models trained on fewer samples (100 per class) than the default settings in Sec. 5.2. Due to the significantly insufficient training of the shadow models, a larger  $\beta$  is needed to maintain a similar level of TPR and CTPR as in Sec. 5.2.
GTSRB	SVHN	CIFAR-10
benign	BadNet	CB	Blend	benign	BadNet	CB	Blend	benign	BadNet	CB	Blend
CBD0.2	0	90 (5)	95 (85)	90 (25)	0	75 (45)	100 (95)	80 (75)	0	35 (0)	95 (90)	45 (40)
CBD0.4	0	95 (5)	95 (85)	95 (35)	0	95 (45)	100 (100)	85 (75)	5	60 (10)	100 (95)	55 (40)
clean samples per class, CBD requires a larger  $\beta$  to achieve similar TPRs and CTPRs. This is because more shadow models will exhibit an abnormally large LDP due to significantly insufficient training.
5.4 Additional Experiments
Empirical validation of the stochastic dominance assumption in Thm. 4.2. In the middle of Fig. 4, we show the histograms (with the associated empirical CDF) of the LDP statistics for the shadow models and the benign models for all four datasets. The statistics for each dataset are obtained using the practically selected  $\sigma$ . The LDP for the benign models is clearly dominated by the LDP for the shadow models in the sense of first-order stochastic dominance.
Figure 4: Supportive results: (Left) Choice of  $\sigma$  for a range of  $\psi$  based on our selection scheme, which matches the  $\sigma$  choices in Fig. 2 with high CTPR and TPR. (Middle) The histograms of the LDP statistics for the shadow models and the benign models, with the associated empirical CDFs. LDP for benign models is stochastically dominated by the LDP for shadow models for all datasets. (Right) Vulnerability of WaNet attack on GTSRB and SVHN. Attack success rate (ASR) reduces with a negligible drop in benign accuracy (ACC) when inputs are smoothed by Gaussian noise.
Class imbalance is not the reason for a large LDP. In our experiments involving generally balanced datasets, backdoor poisoning (i.e. embedding the trigger in a large variety of samples during training) not only enhances the trigger robustness of the attack, but also introduces an imbalance in the poisoned training set. Thus, it is important to show that the large LDPs observed in our experiments are a result of the robustness of the trigger, rather than the class imbalance. Note that if class imbalance can also cause a large LDP, there will easily be false alarms for benign classifiers trained on imbalanced datasets<sup>1</sup>. Here, we train two groups of benign models on SVHN, with 20 models per group. For the first group, the models are trained on a balanced dataset with 3,000 images per class. For each model in the second group, the training set contains 4,400 additional images labeled to some randomly selected class. With the same model architecture and training configurations, the two groups have similar LDP distributions, with mean±std being  $0.445 \pm 0.125$  and  $0.429 \pm 0.119$ , respectively, and with a 0.698 p-value for the t-test for mean. Thus, LDP will not be affected by class imbalance in general, and our method indeed detects the backdoor attack rather than the class imbalance.
Advanced attacks. Based on the key ideas in Sec. 1, CBD requires the attack to have a large STR over the sample distribution. However, this requirement is not always satisfied, especially for some advanced attacks with subtle, sample-specific triggers, such as WaNet [51]. Although not detectable or certifiable by our CBD, these attacks can hardly survive noises either from the environment or simple prepossessing-based defenses in practice. To see this, for each of GTSRB and SVHN, we train 20 models with successful WaNet attacks. We consider a simple test-time defense by applying 1024 randomly sampled Gaussian noises from distribution  $\mathcal{N}(0,\sigma^2 I)$  to each input and then performing a majority vote. As shown in the right of Fig. 4, for both datasets and for small  $\sigma$ , the average attack success rate (ASR) quickly drops without clear scarifies in the benign accuracy (ACC).
In this paper, we proposed CBD, the first certified backdoor detector, which is based on an adjustable conformal prediction using a novel LDP statistic. CBD not only performs detection inference but also provides a condition for attacks that are guaranteed to be detectable. Our theoretical results show that backdoor attacks with a trigger more resilient to noises and with a smaller perturbation magnitude are more likely to be detected with a guarantee. Our empirical results show the strong certification and detection performance of CBD on four benchmark datasets. In future research, we aim to enhance the certification bound to encompass backdoor attacks with larger trigger perturbation norms, such as rotational triggers and subject-based triggers.
Acknowledgements This work is partially supported by the National Science Foundation under grant No. 1910100, No. 2046726, No. 2229876, DARPA GARD, the National Aeronautics and Space Administration (NASA) under grant no. 80NSSC20M0229, Alfred P. Sloan Fellowship, and the Amazon research award.
This is the case for the backdoor detector in [68] which leverages the overfitting to the backdoor trigger during training - benign classifiers trained on imbalanced datasets will also easily trigger a (false) detection.