SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparison to trigger inversion based methods
D	M	Metric	ABS	NC	NC*	Pixel	Pixel*	Tabor	Tabor*	DJANGO
Traffic Sign Synthesis	SSD	Precision	0.7143	0.6250	0.6522	0.4400	0.6666	0.3300	0.4000	0.8000
Recall	0.6250	0.9375	0.9375	0.5000	0.2500	0.3750	0.6250	1.0000
ROC-AUC	0.7109	0.6250	0.6641	0.6250	0.7083	0.5400	0.6725	0.9160
Overhead(s)	409.23	>3600	1180.52	>3600	953.42	>3600	902.10	861.30
F-RCNN	Precision	0.6667	0.3478	0.6364	0.5512	0.6250	0.6621	0.7125	0.8571
Recall	0.3750	1.0000	0.8750	0.6110	0.6250	0.6012	0.6250	0.7500
ROC-AUC	0.5352	0.6094	0.6953	0.6721	0.7500	0.6425	0.6820	0.8750
Overhead(s)	753.89	>3600	2799.42	>3600	2466.08	>3600	2562.59	2128.99
DETR	Precision	-	0.3478	0.3478	0.4000	0.5000	0.2307	0.5000	0.8750
Recall	-	1.0000	1.0000	0.5000	0.3750	0.3750	0.6250	0.8750
ROC-AUC	-	0.5312	0.5312	0.5000	0.6660	0.2500	0.7600	0.9160
Overhead(s)	-	>3600	691.27	>3600	294.32	>3600	275.50	228.92
DOTA	SSD	Precision	0.6000	0.3636	1.0000	0.3300	0.3000	0.5000	0.3333	1.0000
Recall	0.7500	1.0000	0.1250	0.2500	0.7500	0.2500	0.5000	1.0000
ROC-AUC	0.5000	0.5312	0.6875	0.3750	0.3333	0.5000	0.6250	1.0000
Overhead(s)	424.16	>3600	675.32	>3600	529.11	>3600	618.72	678.48
F-RCNN	Precision	0.8000	0.3636	0.3636	0.3330	0.5000	0.3300	0.3300	0.8000
Recall	1.0000	1.0000	1.0000	0.5000	0.7500	0.5000	0.5000	1.0000
ROC-AUC	0.8750	0.5625	0.5625	0.5000	0.6666	0.5000	0.5000	0.9160
Overhead(s)	1127.01	>3600	2866.14	>3600	1565.82	>3600	1602.10	1425.25
COCO	SSD	Precision	0.5167	0.5231	0.8333	0.7500	1.0000	0.6120	0.7500	0.9696
Recall	0.9688	1.0000	0.1471	0.2500	0.2500	0.5620	0.5300	0.8888
ROC-AUC	0.5584	0.5404	0.5579	0.6200	0.7000	0.6419	0.6820	0.9305
Overhead(s)	273.23	>3600	2119.34	>3600	1788.50	>3600	1688.19	1476.01
model's behavior on victim samples tends to shift towards the target label even without the appearance of the backdoor trigger. Therefore, we can feed clean samples from each class and pick the most likely target label by observing the model's output distribution shift. Specifically, given a set of samples  $\{x_{i}\}_{i = 1}^{n}$  from class  $c_{i}$ , we collect top-  $h$  predicted classes of each predicted box for each sample  $x_{i}$ . Denote  $\omega_{j}$  as the frequency of class  $c_{j}$  appearing in the total box predictions. We consider a pair  $\{c_i,c_j\}$  as a compromised label pair if  $\omega_{j}\geq \omega$ . In this paper, we make a trade-off by setting  $h = 5$  and  $\omega = 0.5$ . We also evaluate the sensitivity of hyper-parameters in Section 4.2.
All the experiments are conducted on a server equipped with two Intel Xeon Silver 4214 2.40GHz 12-core processors, 192 GB of RAM, and a NVIDIA RTX A6000 GPU.
Models and Datasets. We conduct the experiments on models from Trojan [1] round 10 and round 13, which consists of a total of 168 models. Approximately half of these models are clean, while the other half are attacked. Our evaluation covers 3 existing object detection image datasets, including COCO [30], Synthesized Traffic Signs [1], and DOTA_v2 [11] on three representative object detection architectures: single-stage detector: SSD [32], two-stage detector: Faster-RCNN [45] and vision transformer based detector: DETR [2]. Please refer to Appendix A for more detailed description.
Attack Settings. We conducted an evaluation of DJANGO by subjecting it to two types of backdoor attacks commonly observed in object detection models: Misclassification and Evasion attacks [3, 38] Misclassification attack means to flip the prediction of objects from the victim class to the target class, when the backdoor trigger is stamped on the input image. On the other hand, evasion triggers tend to suppress the recognition of victim objects, such that the model will consider them as the background. It is noteworthy that the evasion trigger can be regarded as a particular instance of the misclassification trigger, where the target label is set as the background class  $\emptyset$ . As a result, its detection can be performed uniformly. Besides, we leverage stealthy polygons with different colors and textures as backdoor triggers, which are widely evaluated in many backdoor detection papers [59, 17, 49].
Evaluation Metrics. In our evaluation of backdoor detection methods, we employ four well-established metrics: Precision, Recall, ROC-AUC, and Average Scanning Overheads for each model. The unit of measurement we use is seconds (s), and we set a maximum threshold of 1 hour (3600 s) for all the methods being evaluated. If the scanning process exceeds 3600 seconds, it is terminated, and we rely on the existing results for making predictions. Appendix B present more details.
Baseline Methods We compare DJANGO against 6 baseline methods, including 4 trigger inversion based methods (i.e., NC [59], ABS [33], Pixel [55] and Tabor [18]), and 2 meta-classification based methods (i.e., MNTD [69] and MF [21]). For meta classification based methods that involve training, we have performed 5-fold cross-validation and reported the validation results exclusively. For the 4 inversion-based scanners, we adopt their objective functions as described in Eq. 2 and keep their other designs unchanged. To ensure a fair comparison, we use the same setting for common optimization hyper-parameters for all inversion based methods. We determine the optimal threshold for the size of inverted triggers as the detection rule for[59, 55, 18]. For ABS, we follow its original technique and use REASR as the decision score.
4.1 Detection Performance
Comparison to Trigger Inversion based Methods. Table 1 shows the results for trigger inversion based methods, where the first two columns denote the datasets and model architectures, the third column the evaluation metrics, and the subsequent columns the baselines. Additionally, considering that NC [59], Pixel [55], and Tabor [18] need to scan all possible victim-target pairs, which can exceed the time limit, we have employed our warm-up pre-processing technique (Section 3.3). We compare the enhanced versions of these methods  $(\mathrm{NC}^{*},\mathrm{Pixel}^{*}$ $Tabor^{*})$  with DJANGO. Note that we have not evaluated ABS on DETR models, as the original version is specifically designed for CNN-based models and not transformers.
DJANGO consistently outperforms all the baselines, with the best results are highlighted in bold. The average ROC-AUC of DJANGO is 0.913, while the highest ROC-AUC achieved by
Table 2: Comparison to meta classifiers
D	M	Metric	MNTD	MF	DJANGO
Traffic Sign Synthesis	SSD	Precision	0.4545	0.4000	0.8000
Recall	0.3125	0.5000	1.0000
ROC-AUC	0.3750	0.6563	0.9160
F-RCNN	Precision	0.5652	0.5000	0.8571
Recall	0.8125	0.1250	0.7500
ROC-AUC	0.7695	0.6797	0.8750
DETR	Precision	0.1250	0.1250	0.8750
Recall	0.1250	0.1250	0.8750
ROC-AUC	0.1719	0.2890	0.9160
DOTA	SSD	Precision	0.3333	0.3333	1.0000
Recall	0.1250	0.5000	1.0000
ROC-AUC	0.3906	0.4375	1.0000
F-RCNN	Precision	1.0000	0.3333	0.8000
Recall	0.5000	0.5000	1.0000
ROC-AUC	0.6562	0.6875	0.9160
COCO	SSD	Precision	0.8182	0.7000	0.9696
Recall	0.5625	1.0000	0.8888
ROC-AUC	0.8144	0.8163	0.9305
the baselines is 0.875, which is the result obtained by ABS on DOTA and F-RCNN. The average ROC-AUC of these baselines is nearly 0.592. Furthermore, our warm-up pre-processing technique has proven effective in enhancing the scanning performance of the baselines, resulting in an improvement of 0.05 to 0.15 in terms of ROC-AUC and a significant reduction in overheads from over 3600 seconds to a range of 275 to 2800 seconds. Despite these advancements, the enhanced versions of the baselines are still unable to surpass the performance of DJANGO, with ROC-AUC gaps ranging from 0.15 to 0.50.
Additionally, we have observed that the overheads incurred by DJANGO are consistently lower than those of NC, Pixel, and Tabor, even when these methods are equipped with our warm-up preprocessing technique. However, ABS generally achieves lower overheads than DJANGO, primarily because ABS only performs trigger inversion for the top-10 target classes according to the original configuration. Nonetheless, it is important to note that DJANGO significantly outperforms ABS in terms of ROC-AUC, which is the more critical metric for our evaluation. We further illustrate the inverted trigger by DJANGO in Figure 6. It is evident that the inverted trigger produced by DJANGO closely resembles the ground-truth injected trigger in terms of both shape and color. The overall success of DJANGO can be attributed to our well-designed warm-up pre-processing and the novel Gaussian Focus Loss 4. These techniques play a crucial role in achieving the superior performance and efficiency demonstrated by DJANGO compared to the baselines.
Comparison to Meta Classifiers. Table 2 provides a comparison between meta classifiers and DJANGO. Since meta classifiers require only a few seconds to scan a model, we have omitted the overheads from the table. Observe that baselines achieve decent results in the COCO dataset with over 0.81 ROC-AUC, while only make nearly 0.6 ROC-AUC on other datasets. This discrepancy can be attributed to the fact that there are over 60 models available in the COCO dataset, whereas there are only 30 models in the other datasets. Consequently, the performance of the meta classifiers heavily relies on the availability of a large number of training models. Unfortunately, obtaining such a
Figure 6: Visual similarity between GT and DJANGO inverted triggers.
(a) Evaluation on different datasets
Figure 7: Evaluation of label proposal pre-processing
(b) Evaluation on different model architectures
significant number of models in real-world scenarios is a challenging task. In contrast, DJANGO does not require access to training models. Moreover, it achieves a ROC-AUC of over 0.91, outperforming the meta classifiers even when they are trained on a sufficient number of models in the COCO dataset. This highlights the effectiveness and robustness of DJANGO in detecting backdoor attacks, surpassing the performance of meta classifiers.
Evaluation on Advanced Object Detection Backdoor Attacks. We also evaluate DJANGO on two advanced object detection backdoor attacks [3, 5, 28]. DJANGO achieves 0.8 and 0.9 ROC-AUC on detecting these attacks. Appendix E presents more discussion.
4.2 Evaluation of Label Proposal Pre-processing
In this section, we evaluate the effectiveness of our label proposal pre-processing introduced in Section 3.3. Results are presented in Figure 7. As shown in Figure 7(a), where the x-axis denotes the dataset, the left y-axis denotes the average number of pairs and the right one the TPR (True Positive Rate) of the selected pairs. In the case of an attacked model, if the selected pairs after pre-processing contain the ground-truth victim-target pair, it is considered a true positive. TPR is calculated as the ratio of true positives to all positives. The green bars represent the average number of pairs without pre-processing, while the red bars represent the number of pairs after pre-processing. We observe that across all datasets, our pre-processing technique significantly reduces the number of scanning pairs by  $83\%$  to  $98\%$ . The curves represent the TPR, showing that our pre-processing results in TPRs of over  $88\%$  across different datasets. These results indicate that our pre-processing technique effectively reduces the number of pairs to a remarkably low level while maintaining a high detection rate. The experiment is conducted by selecting various values of  $h$  ranging from 1 to 10 (representing the top  $h$  class labels) and  $\omega$  ranging from 0.1 to 0.8 (representing the frequency threshold). We record the average number of pairs and true positive rate (TPR) under this configuration. The results are depicted in Figure 7(b). We observe that SSD and DETR models require fewer than 100 pairs to reach high TPR, while Faster-RCNN models require around 200 pairs. One potential reason for this difference is that Faster-RCNN requires a two-stage training process where the backdoor signal is less prominent compared to single-stage training methods in SSD and DETR.
Table 3: Ablation study
Method	SSD	Faster-RCNN	DETR
ROC-AUC	Overhead(s)	ROC-AUC	Overhead(s)	ROC-AUC	Overhead(s)
DJANGO	1.0000	859.59	0.8750	2120.10	0.8750	227.55
DJANGO - GF Loss	0.7500	860.24	0.7500	2135.59	0.7500	226.88
DJANGO - Pre-Processing	0.6250	>3600	0.5000	>3600	0.6250	>3600
Our ablation experiments are conducted on the synthesized traffic sign dataset with 3 different model architectures. We include 8 models for each architecture, with half clean and half attacked.
There are two key designs of DJANGO to ensure the detection effectiveness, i.e., GF Loss and Pre-Processing. We conduct ablation study to assess the contribution of these components respectively. Results are shown in Table 3, where each row corresponds to a specific ablation method. The results clearly demonstrate the critical importance of both GF Loss and Pre-Processing in DJANGO. Without these components, the detection performance of DJANGO significantly degrades by more than 0.13 ROC-AUC. Pre-Processing also plays a crucial role in reducing the detection overhead. Please refer to sec D for hyper-parameter sensitivity analysis.
4.4 Adaptive Attack
In this section, we evaluate the performance of DJANGO in the context of adaptive attacks. The effectiveness of DJANGO relies on the confidence levels assigned to the potential boxes, as these determine the coefficients used for trigger inversion. Consequently, an adaptive attack strategy could
Table 4: Adaptive attack
Attack Conf.	Clean mAP	ASR_0.1	ASR_0.6	ASR_0.8	DJANGO
No Attack	0.8766	0.0047	0.0023	0.0000	0.0
0.80	0.8700	1.0000	1.0000	0.9812	1.0
0.60	0.8708	1.0000	1.0000	0.4014	1.0
0.51	0.8685	1.0000	0.5728	0.0446	0.6
involve reducing the confidence of objects containing the trigger, thereby potentially causing DJANGO to assign lower coefficients to these poisoned boxes. To evaluate the adaptive attack scenario, we conduct experiments using a 5-class synthesized traffic sign dataset with the SSD model. We train several models with misclassification attacks, where the victim class is to 0 and the target class 4. In order to simulate low confidence attacks, we reduced the ground-truth confidence of the target class to values of 0.8, 0.6, and 0.51, while setting the confidence of the victim class to 0.2, 0.4, and 0.49, respectively. After sufficient training, we applied DJANGO to these models. The results are presented in Table 4, where we recorded the clean mAP and the ASRs at different score thresholds. For instance, ASR_0.1 indicates the ASR when the prediction confidence is higher than 0.1. From the results, we can see that low confidence attacks lead to lower ASRs, especially when the score threshold for measuring the ASR is set high. Furthermore, DJANGO is successful in detecting attacked models with target confidences of 0.8 and 0.6, but fails when the target confidence is 0.51 (threshold set at 0.8). These results suggest that DJANGO is effective against most low-confidence attacks, but its performance may degrade to some extent when the target confidence is very low.
Our primary focus in this paper is on attacks that use static polygon triggers, which are more feasible in real-world scenarios. How to effectively inject more complex triggers [42, 9, 48] in object detection models is still an open question. We leave it to future work.
In this paper, we present DJANGO, the first trigger inversion framework on detecting backdoors in object detection models. It is based on a novel Gaussian Focus Loss to tackle the loss misalignment issue in object detection trigger inversion. Extensive experiments demonstrate the effectiveness and efficiency of DJANGO compared to existing baselines.
We thank the anonymous reviewers for their constructive comments. We are grateful to the Center for AI Safety for providing computational resources. This research was supported, in part by IARPA TrojAI W911NF-19-S-0012, NSF 1901242 and 1910300, ONR N000141712045, N000141410468 and N000141712947. Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
[72] Kaiyuan Zhang, Guanhong Tao, Qiuling Xu, Siyuan Cheng, Shengwei An, Yingqi Liu, Shiwei Feng, Guangyu Shen, Pin-Yu Chen, Shiqing Ma, and Xiangyu Zhang. FLIP: A provable defense framework for backdoor mitigation in federated learning. In The Eleventh International Conference on Learning Representations, 2023.
[73] Zhuo Zhang, Guanhong Tao, Guangyu Shen, Shengwei An, Qiuling Xu, Yingqi Liu, Yapeng Ye, Yaoxuan Wu, and Xiangyu Zhang. Pelican: Exploiting backdoors of naturally trained deep learning models in binary code analysis. 2023.
[74] Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems, 30(11):3212-3232, 2019.
A Details of datasets and architectures
A.1 Object Detection Image Dataset
COCO (Common Objects in Context) [30] dataset is widely used for object detection tasks. It contains 80 object categories, including people, animals, vehicles and more. Each image can contain multiple instances of objects, providing ample opportunities for training and evaluating models capable of detecting and segmenting objects in complex scenes.
Synthesized Traffic Sign dataset is designed by TrojAI [1] which focuses on traffic sign detection, featuring various types of traffic signs commonly encountered in real-world scenarios. There are in total over 4000 different traffic signs. Each model is trained on a randomly sampled subset of classes. The number of classes within these subsets exhibits variability, ranging from as few as 2 to a maximum of 128.
DOTA (Detection in Aerial Images) dataset is designed for object detection in aerial images which consists of high-resolution images captured by aerial platforms. It contains 18 categories, including plane, ship, storage tank, baseball diamond and more. Its large-scale, fine-grained annotations, and challenging scenarios make it an ideal benchmark for evaluating and developing algorithms capable of detecting objects in aerial images accurately.
We evaluate our method on three well-known model architectures: i.e., SSD [32], Faster-RCNN [45], and DETR [2]. SSD (Single Shot MultiBox Detector) [32] is a popular object detection model which utilizes a series of convolutional layers to detect objects at multiple scales and aspect ratios. Faster-RCNN [45] is another widely adopted object detection model that combines region proposal generation with a region-based CNN for object detection. DETR (DEtection Transformer) [2] is a state-of-the-art object detection model that utilizes a transformer-based architecture. It replaces the conventional two-stage approach with a single-stage end-to-end detection framework.
A.3 Model Dataset
TrojAI [1] initiative, spearheaded by IARPA, encompasses a multi-year, multi-round program. Its overarching objective revolves around the development of scalable and dependable automatic backdoor detection tools, specifically targeting the identification of backdoor trojans within Deep Learning models across diverse modalities. Presently, the program consists of a total of 13 rounds, each with distinct focuses and tasks. The initial four rounds and the eleventh round center their efforts on detecting trojans present in image classification models. In contrast, rounds five through nine concentrate on transformer models employed in various NLP tasks, including Sentiment Analysis, Named Entity Recognition, and Question Answering. Round twelve dedicates itself to the detection of backdoors in neural network-based PDF malware detection. Finally, rounds ten and thirteen direct their attention towards object detection models. For the evaluation of models, we exclusively utilize the training sets from rounds 10 and 13. As shown in Table 5, our evaluation entails 72 models trained on the Synthesis Traffic Sign dataset, encompassing all three model architectures. Among these models, 48 are benign, while 24 are deliberately poisoned, with an equal distribution of triggers for misclassification and evasion. Concerning the DOTA models, there exist two architectures, namely SSD and Faster-RCNN, resulting in a total of 24 models, including 16 benign models and 4 each poisoned with misclassification and evasion triggers. All COCO models adopt the SSD architecture, with a distribution of 36 clean models and 18 models poisoned by both misclassification and evasion triggers. To provide further elucidation, it is important to note that the poison rate varies within the range of  $0.1\%$  to  $8\%$  across diverse models sourced from Trojan AI r10 and r13. Similarly, the trigger size exhibits a range of 1x1 to 22x22, representing a scale of  $0.001\%$  to  $0.7\%$  relative to the input dimensions. Pertaining to the hyper-parameters utilized in model training, the learning rate is stochastically assigned, spanning from  $1.56\mathrm{e} - 8$  to  $1\mathrm{e} - 4$  across different models. The number of epochs for training spans from 6 to 100, while the batch size ranges from 4 to 32. As for the model performance metrics, the average clean mAP across the models attains a value of 0.7979, while the average poison mAP stands at 0.7680. The trigger's polygonal structure is characterized by varying
edge counts, ranging from 3 to 8. Furthermore, each individual trigger is endowed with randomly generated color and texture attributes. Concrete settings can be found Round13* and Round10*.
B Details of evaluation metrics
In our evaluation of backdoor detection methods, we employ four well-established metrics: Precision, Recall, ROC-AUC, and Average Scanning Overheads for each model. Precision quantifies the accuracy of a detection method by measuring the proportion of correctly identified positive instances among all predicted positives. In our case, we consider attacked models as positive instances and benign models as negatives. A higher precision indicates a lower rate of falsely identifying benign models as attacked. Recall, on the other hand, assesses the effectiveness of the detection method in correctly identifying positive instances. It measures the proportion of true positives among all actual positives. A higher recall suggests that the detection method is capable of identifying a significant portion of attacked models. ROC-AUC (Receiver Operating Characteristic - Area Under the Curve) plots the true positive rate against the false positive rate at various threshold values and calculates the area under the curve. A value of 1 indicates perfect classification, while a value of 0.5 indicates that the method is no better than random guessing. We also consider the overhead of the detection method, which quantifies the average time required to scan a single model. We use seconds (s) as the unit of measurement and set a maximum threshold of 1 hour (3600 s). If the scanning process exceeds 3600 seconds, it is terminated, and we rely on the existing results for making predictions. Low overhead signifies high efficiency of the method. By employing these four metrics, we aim to comprehensively evaluate the performance and efficiency of the backdoor detection methods. It is worth noting that the time limit we have set for scanning models is deliberately conservative when compared to the thresholds established in different rounds of the TrojAI competition. For example, in round 13, participants are granted a generous 30-minute duration for scanning a single model. To surpass the official benchmarks set in each round, a more aggressive and precise pre-processing approach may be necessary.
C Details of Baseline Methods
In this section, we introduce more details of baseline methods, including NC [59], Tabor [18], ABS [33], Pixel [55], Matrix Factorization(MF) [21] and MNTD [69].