SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guangyu Shen*
West Lafayette, IN, 47907
shen447@purdue.edu
Siyuan Cheng*
West Lafayette, IN 47907
cheng535@purdue.edu
Guanhong Tao
taog@purdue.edu
Kaiyuan Zhang
zhan4057@purdue.edu
Yingqi Liu
Redmond, Washington 98052
yingqiliu@microsoft.com
Shengwei An
an93@purdue.edu
Shiqing Ma
University of Massachusetts at Amherst
Amherst, MA, 01003
shiqingma@umass.edu
Xiangyu Zhang
xyzhang@cs.purdue.edu
Object detection models are vulnerable to backdoor or trojan attacks, where an attacker can inject malicious triggers into the model, leading to altered behavior during inference. As a defense mechanism, trigger inversion leverages optimization to reverse-engineer triggers and identify compromised models. While existing trigger inversion methods assume that each instance from the support set is equally affected by the injected trigger, we observe that the poison effect can vary significantly across bounding boxes in object detection models due to its dense prediction nature, leading to an undesired optimization objective misalignment issue for existing trigger reverse-engineering methods. To address this challenge, we propose the first object detection backdoor detection framework Django (Detecting Trojans in Object Detection Models via Gaussian Focus Calibration). It leverages a dynamic Gaussian weighting scheme that prioritizes more vulnerable victim boxes and assigns appropriate coefficients to calibrate the optimization objective during trigger inversion. In addition, we combine Django with a novel label proposal pre-processing technique to enhance its efficiency. We evaluate Django on 3 object detection image datasets, 3 model architectures, and 2 types of attacks, with a total of 168 models. Our experimental results show that Django outperforms 6 state-of-the-art baselines, with up to  $38\%$  accuracy improvement and  $10x$  reduced overhead. The code is available at https://github.com/PurduePAML/DJGO.
Object detection is an extensively studied computer vision application that aims to identify multiple objects in a given image. It has been widely integrated into various real-world systems, including public surveillance [51], autonomous driving [14, 25], optical character recognition (OCR)[41], etc.
* Equal Contribution
(a) Evasion trigger
(b) Misclassification trigger
Figure 1: Triggers with different effects. The blue patch in Figure 1(a) is an evasion trigger, causing the bounding box to disappear. Figure 1(b) represents a misclassification trigger (yellow), leading to the misclassification of "pizza" as "giraffe".
State-of-the-art object detection systems, e.g., SSD [32] and YOLO [44], are primarily built on Deep Learning (DL) models [13], owing to their remarkable feature representation capabilities.
Despite the impressive performance of existing object detection approaches, they are vulnerable to backdoor attacks. Backdoor attack is a form of training-time attack, in which a malicious adversary strategically embeds an imperceptible trigger (e.g., a small image patch) onto a small set of training data. DL models trained on this poisoned dataset will predict the attacker-chosen target label whenever an input is stamped with the trigger. There are two major backdoor attacks in object detection, namely, misclassification attack [3] and evasion attack. As shown in Figure 1, misclassification attack causes the object detection model to misclassify an object as a target object, whereas evasion attack makes the model fail to recognize certain objects (i.e., classify traffic signs as the background). Such attacks can significantly hinder the deployment of object detection systems in real world as they may cause severe security issues or even life-threatening incidents.
To counter the backdoor threat, researchers have proposed a multitude of defense techniques aimed at safeguarding the model against backdoor attacks throughout its entire life cycle [59, 33, 69, 16, 26, 22, 18, 10, 54, 50, 36, 72, 27, 65, 31, 35]. Among them, trigger inversion [59, 55, 18] is one of the most popular backdoor scanning methods and has demonstrated its effectiveness in various tasks [59, 35]. Given a small set of clean samples and the subject model, trigger inversion techniques aims to determine if the model is backdoored or not by reverse-engineering the underlying triggers. The task of trigger inversion can be formulated as an optimization problem with specific constraints, such as trigger size [59, 18] or particular stylistic attributes [9, 42].
In order to detect backdoors in object detection, a direct solution is to adapt trigger inversion from the classification task to object detection. We however find that a simple adaptation of trigger inversion fails short. This attributes to the significantly denser output of object detection models due to the nature of the task. In specific, an object detection model, such as SSD [32], typically produces thousands of bounding boxes with corresponding class probabilities to capture objects of varying scales within a single image. Attackers can exploit this extensive output space to conceal injected triggers and mislead the optimization process during trigger inversion, causing the loss misalignment issue. That is, throughout the entire trigger inversion process, the loss values consistently remain either significantly high or extremely low, regardless of the Attack Success Rate (ASR) that denotes the ratio of misclassified samples when the trigger is applied. This misalignment property hinders the identification of high-quality triggers, leading to low backdoor detection performance.
Our study reveals that the underlying cause of the loss misalignment issue stems from the unequal impact of poisoning on individual bounding boxes within the backdoored model. More specifically, the backdoor trigger affects only a small fraction of bounding boxes, while leaving the majority of them largely unaffected. Consequently, the aggregated gradient information is obfuscated by remaining benign bounding boxes during inversion, thereby impeding the optimizer from accurately estimating the gradient direction. Based on this observation, we propose the first trigger inversion-based backdoor detection framework for object detection: DJANGO (Detecting Trojans in Object Detection Models via Gaussian Focus Calibration). The overview is shown in Figure 2. It features a novel Gaussian Focus Loss to calibrate the misaligned loss during inversion by dynamically assigning weights to individual boxes based on their vulnerability. Equipped with a label proposal pre-processor, DJANGO is able to quickly identify malicious victim-target labels and effectively invert the injected trigger lies in the backdoored model. Extensive evaluation of 3 object detection datasets, utilizing 3
different model architectures with a total of 168 models, demonstrates the superiority of DJANGO over six existing state-of-the-art backdoor scanning techniques. Specifically, we achieve up to 0.38 ROC improvement and a substantial reduction in scanning time.
2 Background & Threat Model
Deep Learning based Object Detection Techniques. Extensive research efforts have been devoted to deep learning (DL) object detection, encompassing various aspects such as neural network architecture design [32, 45, 2], training optimization [29, 61, 74], and robustness [71, 66, 6]. These endeavors contribute to a broad body of work aimed at advancing the field of DL object detection. Specifically, given an image  $x \in \mathcal{X}$  that contains  $p$  objects, an object detection neural network  $g_{\theta}$  (parameterized by  $\theta$ ) outputs  $K$  ( $K \gg p$ ) 5-tuples:  $\hat{y} = g_{\theta}(x) = \{\hat{b}_k^x, \hat{b}_k^y, \hat{b}_k^h, \hat{b}_k^w, \hat{p}_k\}_{k=1}^K = \{\hat{b}_k, \hat{p}_k\}_{k=1}^K$ , shorten
as  $b_{k}(x)$  and  $\hat{p}_k(x)$ , which denote the center coordinates, height, width, and class probability
Figure 2: Overview of DJANGO
of the  $k$ -th predicted bounding box. Note that  $K$  is usually much larger than  $p$  ( $K = 8732$  in SSD [32],  $p \approx 7.7$  in COCO dataset [30]). Box matching algorithms [32, 46] and Non-Maximum Suppression(NMS) [20] were introduced to address the challenge posed by the substantial box number discrepancy between the prediction and ground-truth in both training and inference stages.
Backdoor Attack & Defense. Backdoor attacks can be carried out through data poisoning with modified or clean labels [17, 34, 7, 58] and model parameter hijacking [47]. Initially, small image patches were used as backdoor triggers [17, 8, 48], and later on more complex triggers have been successfully injected in DL models [28, 64, 33, 37]. These attacks are not limited to image classification models; they also affect Large Language Models [70], code models [73, 52], Reinforcement Learning agents [60], pre-trained image encoders [24, 56], and object detectors [3, 39]. Researchers have proposed various approaches from defense perspectives. During the training stage, methods such as poisoned sample identification and filtering have been proposed [40, 53, 4, 57, 19, 43]. In the post-training stage, backdoor scanning techniques aim to recover injected triggers from trained models [59, 49, 54, 62, 10, 63, 35, 50, 15]. Backdoor removal techniques are employed to eliminate triggers with minimal resource consumption and performance impact. Real-time rejection of trigger-carrying samples during inference can be achieved using methods like [12, 16]. Furthermore, theoretical analyses have been conducted to understand backdoor attacks [67, 68, 72, 23]. In this work, we prioritize the practical application of trigger inversion-based backdoor detection, with a specific focus on object detection models.
Threat Model. Our work considers the standard setting used in existing backdoor scanning literature [59, 33, 18, 10], where the defender has a small set of clean samples from each class in the validation set (10 in our paper), but no poison samples and white-box access to the model under scanning. The defense goal is to classify the benignity of the subject model. Our focus in this study is on label specific polygon triggers with two types of effects: misclassification and evasion, which cause the model to predict an object as a target object and the background, respectively.
3.1 Misalignment of CE Loss and ASR in Object Detection Model Trigger Inversion
Backdoor Detection via Trigger Inversion. Trigger inversion aims to decide whether the model is backdoored or not by reverse-engineering the injected trigger through optimization. Consider an image classification model  $f_{\theta}$  parameterized by  $\theta$  with  $C$  classes and a small support set  $S_{c_v}$  from
(a) Classification model
(b) Object detection model with evasion trigger
(c) Object detection model with misclassification trigger
Figure 3: ASR vs. CE loss during trigger inversion
each class  $c_v$ . Trigger inversion [49, 59] aims to solve the following optimization objective.
$$
\min  _ {m, p} \mathbb {E} _ {x \in \mathcal {S} _ {c _ {v}}} \left[ \mathcal {L} \left(f _ {\theta} (\phi (x)), c _ {t}\right) \right] + \beta \| m \| _ {1}, \text {w h e r e} \phi (x) = (1 - m) \odot x + m \odot p. \tag {1}
$$
Variables  $m$  and  $p$  are the trigger mask and pattern respectively, to be optimized. The loss function  $\mathcal{L}(\cdot)$  typically resues the task loss function as a measurement of the degree of the misbehaved model. In classification and object detection, it will be the cross entropy (CE) loss.  $\beta$  is the coefficient on the regularization term, and  $||m||_1$  represents the  $\ell_1$  norm of the mask. Since the defender lacks the knowledge of the exact poisoned victim and target labels, the backdoor scanning requires optimizing Eq. 1 for every label pair  $(c_v, c_j)$ . If any of these pairs yields a trigger with an exceptionally small size, it is indicative that the model is Trojaned.
Intuitively, it shall be effortless to adapt existing trigger inversion methods from image classification to object detection. Considering the disparity of these two tasks, object detection models generate much denser output for each input, i.e., thousands of bounding boxes versus one label in classification. We can make adjustments to Eq. 1 to optimize a trigger that can impact samples at the box level rather than the entire image. Given a sample containing a victim object, one naive modification is to optimize a trigger that can influence all bounding boxes that are close to the victim object. The objective function can be reformulated as follows:
$$
\min _ {m, p} \mathbb {E} _ {x \in \mathcal {S} _ {c _ {v}}} \mathbb {E} _ {\hat {o} \in \mathcal {T} _ {x}} [ \mathcal {L} (\hat {o}, c _ {t}) ] + \beta | | m | | _ {1},
$$
where  $\mathcal{T}_x = \{\hat{p}_k(\phi (x))\mid \mathcal{I}[\hat{b}_k(\phi (x)),b^* ]\geq \alpha \}$  and  $\phi (x) = (1 - m)\odot x + m\odot p$
We denote the predicted probability and coordinates of the  $k$ -th bounding box of sample  $\phi(x)$  as  $\hat{p}_k(\phi(x))$  and  $\hat{b}_k(\phi(x))$ , respectively. The function  $\mathcal{J}(\cdot)$  refers to the Jaccard Overlap, also known as Intersection over Union.  $b^*$  denotes the ground-truth box coordinates. Intuitively,  $\mathcal{T}_x$  indicates the class confidence of a set of predicted boxes surrounding the victim object. Additionally,  $\alpha$  is a pre-defined threshold used to quantify the degree of overlap between predictions and the ground-truth box and set to 0.5 by default. In this naive extension, the objective is to consider every predicted box surrounding the victim object as the target of the attack and aim to reverse-engineer a trigger that compromises all of them. We explain this limitation in the following.
Mis-aligned Loss and ASR. Attack Success Rate (ASR) denotes the percentage of victim samples misclassified as the target label when the inverted trigger is applied (e.g., set at  $90\%$ ). For misclassification trigger, a successful attack is defined as the presence of at least one box predicts the target class in the final detection. Evasion Attack is a particular case of misclassification attack where the target label is the background class  $\emptyset$ . Cross-entropy (CE) loss measures the numerical difference between model output (confidence) and the desired prediction. In general, ASR and the CE loss are negatively correlated throughout the optimization process because minimizing the CE loss increases the probability of the target label. If the target label probability surpasses those of other labels, the input is misclassified as the target label. Figure 3(a) illustrates the characteristics of the CE loss and the ASR during trigger inversion for an image classification task.
Such a correlation however may not hold for object detection using a naive trigger inversion Eq. 2. Specifically, Figure 3(b) shows the inversion process on a Faster-RCNN [45] object detection model poisoned by an evasion trigger (TrojAI Round13#91, victim class 47). Note that an evasion trigger
(a) R13#114:90→57: Miscls trigger
(b) R13#33:11  $\rightarrow$  0: Evasion trigger
Figure 4: Target confidence discrepancy w/o the ground-truth trigger
will cause all bounding boxes surrounding the victim object to disappear. Observe that even when the CE loss is small ( $\leq 0.1$ ), the ASR remains low ( $0\%$ ). For a model poisoned by misclassification attack (Round13#12 victim class 3), as illustrated in Figure 3(c), the CE loss remains incredibly high ( $\geq 10$ ) and decreases only slightly during trigger inversion, while the ASR remains low or even decreases. Therefore, the naive trigger inversion fails to find a trigger with high ASR and cannot identify the poisoned model. The observations are similar when extending several state-of-the-art trigger inversion techniques [33, 59, 55, 18] to object detection. For instance, NC [59] only achieves  $58\%$  detection accuracy on the Trojan dataset [1].
Causes of Misalignment. We study the misalignment problem by using differential analysis to compare the intermediate outputs of the object detection model with and without the ground-truth trigger, specifically the target label confidence of predicted bounding boxes. Given a sample with a victim object, we collect all predicted boxes with  $IoU \geq \alpha$  ( $\alpha = 0.5$  means a box is close to the object) around the victim object and track their target label probability change before and after applying the ground-truth trigger. The key observation is that not every box is equally poisoned. Figure 4(a) shows the per-box confidence discrepancy for a poisoned model with a misclassification trigger. Although there are a total number of 84 boxes surrounding the victim object, only approximately  $12\%$  of them (10 boxes, shown in high red bars) are infected by the ground-truth backdoor trigger. The rest of the boxes only have an average of  $\leq 10^{-10}$  probability increase on the target label. For the evasion trigger in Figure 4(b), there is a slight difference, where  $85\%$  of the boxes already have high confidence (on the background class  $\emptyset$ ) even without triggers. The backdoor trigger only elevates the target probability for the remaining  $15\%$  of the boxes.
This intuitively explains the misalignment issue of naive trigger reverse-engineering methods. During optimization, the cross-entropy loss of each box is equally aggregated, and the averaged gradient is used to mutate the trigger via back-propagation. However, in the poisoned model, there are only a small fraction of bounding boxes affected by the backdoor trigger. The gradient information from unaffected boxes dominates the optimization direction, causing the inversion to fail. This observation highlights the significance of considering individual bounding boxes separately when devising inversion methods for object detection, which shall focus on boxes that have a higher likelihood of being compromised but have not been considered yet.
3.2 Our Solution: Trigger Inversion via Gaussian Focus Loss
As discussed in the last section, the goal of our trigger inversion is to discriminate different bounding boxes during optimization. Particularly, we aim to find easy-to-flip boxes that are vulnerable and targeted by injected backdoors. However, it is non-trivial to adaptively select boxes during trigger inversion as the box output is constantly changing.
Focal Loss [29] is a widely used metric in object detection for assisting better training. It separates objects into hard-to-classify and easy-to-classify objects based on their training errors, and assigns different weights on their training losses. For the notation simplicity, we use a binary object detection model. Note that it is straightforward to extend it to multi-class cases. Recall  $\hat{p}_k$  denotes the output probability of the  $k$ -th victim box. The CE loss in Eq. 2 for the  $k$ -th victim box can be simplified as  $\mathcal{L}(\hat{p}_k, c_t) = -\log(\hat{p}_k) \cdot c_t = -\log(\hat{p}_k)$ . Focal Loss is a refined weighted CE loss aiming to solve the imbalanced learning difficulty issue in object detection models. In detail, it assigns larger weights to boxes that perform poorly (hard examples), i.e., small  $\hat{p}_k$ , to enlarge its contribution to the aggregated
Figure 5: Coefficient dynamics during reverse-engineering a misclassification trigger
loss. It assigns smaller weights to boxes that have already learned well (easy examples), i.e., large  $\hat{p}_k$ :
$$
\mathcal {L} _ {f l} \left(\hat {p} _ {k}, c _ {t}\right) = - \lambda \left(1 - \hat {p} _ {k}\right) ^ {\gamma} \log \left(\hat {p} _ {k}\right), \tag {3}
$$
where  $\lambda \geq 0, \gamma \geq 1$  ( $\lambda = 0.25, \gamma = 2$  suggested in [29]). The coefficient of the  $k$ -th box will exponentially grow as its probability  $\hat{p}_k$  gets close to 0 and vice versa.
Eq. 3 provides insights to address the misalignment problem. Inspired by this, we propose Gaussian Focus Loss to calibrate the misaligned loss. As illustrated in Figure 4(a), the backdoor trigger tend to infect only a small portion of victim boxes and leaves remaining largely unchanged. In other words, a trigger will mainly focus on compromising more vulnerable boxes that are easy to flip (easy example). Therefore, our goal is essentially opposite to the Focal Loss as it aims to pay more attention to the hard samples. To avoid the gradient vanishing issue shown in Figure 4(b), a box shall also not be focused as long as it reaches the attack criterion, i.e., target label confidence  $\hat{p}_k \geq 0.1$ . To summarize, the desired loss shall be able to dynamically capture a set of vulnerable boxes that have not been flipped yet, and assign a large coefficient to encourage the transition. The natural bell shape of the Gaussian Distribution perfectly satisfies our constraints*. We formulate our proposed novel Gaussian Focus Loss as follows:
$$
\mathcal {L} _ {g f} \left(\hat {p} _ {k}, c _ {t}\right) = - \mathcal {I} \left(\hat {b} _ {k}, b ^ {*}\right) \phi \left(\hat {p} _ {k} ^ {\gamma}; \hat {\sigma} _ {k}, \hat {\mu} _ {k}\right) \log \left(\hat {p} _ {k}\right)
$$
$$
w h e r e \mathcal {J} (\hat {b} _ {k}, b ^ {*}) = I o U (\hat {b} _ {k}, b ^ {*}), \phi (x; \sigma , \mu) = \frac {1}{\sigma \sqrt {2 \pi}} e ^ {- \frac {1}{2} \left(\frac {x - \mu}{\sigma}\right) ^ {2}} \tag {4}
$$
It involves two key attributes to measure the vulnerability of a victim box during optimization: the overlapping with the ground-truth object and the confidence towards the target label. Namely, boxes with larger overlapping and higher target label confidence tend to be more vulnerable. The  $\hat{\mu}_k$  and  $\hat{\sigma}_k$  are the mean and standard deviation of the Gaussian Distribution. Note that we allow two parameters to be co-optimized with the inverted trigger such that we can adjust the range of focused boxes dynamically through the entire trigger inversion procedure. We initialize  $\hat{\mu}_k = 0.1$  and  $\hat{\sigma}_k = 2$  in this paper. Figure 5 provides a visual explanation of the coefficient dynamics during various optimization steps. Initially, when the victim boxes exhibit low confidence towards the target label, the Gaussian Focus Loss assigns slightly larger weights to a considerable number of bounding boxes surrounding the initial value (0.1). As the optimization progresses, the Gaussian Focus Loss prioritizes more vulnerable boxes that have higher target label probabilities by decreasing the value of  $\sigma$  and increasing the value of  $\mu$ . In the later stages, the weight coefficients of boxes that have already been flipped are reduced to mitigate the issue of gradient vanishing.
3.3 Compromised Label Proposal via Backdoor Leakage
Without knowing the exact victim and target labels, scanning backdoors is time-consuming. To reduce the time cost, pre-processing is proposed [49, 54] to quickly select a small set of promising compromised label pairs based on the widely existing backdoor leakage phenomena: the poisoned
*Any distributions characterized by centralized peaks are suitable for our intended purpose. We explore alternatives in Appendix D.