SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yifan  $\mathbf{Pu}^{1*}$  Weicong Liang $^{2\natural *}$  Yiduo Hao $^{3\natural *}$  Yuhui Yuan $^{4\boxtimes}$
Yukang Yang4 Chao Zhang2 Han Hu4 Gao Huang1
$^{1}$ Tsinghua University  $^{2}$ Peking University  $^{3}$ University of Cambridge  $^{4}$ Microsoft Research Asia puyf23@mails.tsinghua.edu.cn, liangweicong@stu.pku.edu.cn, yh463@cam.ac.uk, yuhui.yuan@microsoft.com, gaohuang@tsinghua.edu.cn
Modern detection transformers (DETRs) use a set of object queries to predict a list of bounding boxes, sort them by their classification confidence scores, and select the top-ranked predictions as the final detection results for the given input image. A highly performant object detector requires accurate ranking for the bounding box predictions. For DETR-based detectors, the top-ranked bounding boxes suffer from less accurate localization quality due to the misalignment between classification scores and localization accuracy, thus impeding the construction of high-quality detectors. In this work, we introduce a simple and highly performant DETR-based object detector by proposing a series of rank-oriented designs, combinedly called Rank-DETR. Our key contributions include: (i) a rank-oriented architecture design that can prompt positive predictions and suppress the negative ones to ensure lower false positive rates, as well as (ii) a rank-oriented loss function and matching cost design that prioritizes predictions of more accurate localization accuracy during ranking to boost the AP under high IoU thresholds. We apply our method to improve the recent SOTA methods (e.g., H-DETR and DINO-DETR) and report strong COCO object detection results when using different backbones such as ResNet-50, Swin-T, and Swin-L, demonstrating the effectiveness of our approach. Code is available at https://github.com/LeapLabTHU/Rank-DETR.
The landscape of modern object detection systems has undergone significant transformation since the pioneering work DEtction TRansformer (DETR) [3]. Since DETR yielded impressive results in object detection, numerous subsequent research works such as Deformable-DETR [78], DINO [75], and H-DETR [30] have further advanced this field. Moreover, these DETR-based approaches have been successfully extended to address various core visual recognition tasks, including instance / panoptic segmentation [6, 38, 61, 69, 71, 35, 9, 8, 70], pose estimation [57, 36, 56], and multi-object tracking [5, 47, 58]. The notable progress in these areas can be credited to ongoing advancements in enhancing the DETR-based framework for improved object detection performance.
Considerable endeavors have been dedicated to advancing the performance of DETR-based methods from various perspectives. These efforts include refining transformer encoder and decoder architectures [48, 74, 78, 2, 12, 78, 11, 40, 43] as well as redesigning of query formulations [63, 44, 34, 75]. While substantial research has been dedicated to developing accurate ranking mechanisms for dense one-stage object detectors like FCOS [59] or ATSS [77], few studies have specifically investigated this aspect for modern object detectors based on DETR. However, ranking mechanisms are vital in enhancing the average precision performance, particularly under high IoU thresholds.
*Equal contribution. Interns at Microsoft Research Asia. Corresponding authors.
This study's primary focus revolves around constructing a high-quality object detector using DETR that exhibits strong performance at relatively high IoU thresholds. We acknowledge the criticality of establishing an accurate ranking order for bounding box predictions in constructing these detectors. To achieve this, we introduce two rank-oriented designs that effectively leverage the benefits of precise ranking information. First, we propose a rank-adaptive classification head and a query rank layer after each Transformer decoder layer. Rank-adaptive classification head adjusts the classification scores using rank-aware learnable logit bias vectors, while the query rank layer fuses additional ranking embeddings into the object queries. Second, we propose two rank-oriented optimization techniques: a loss function modification and a matching cost design. These functions facilitate the ranking procedure of the model and prioritize more accurate bounding box predictions with higher IoU scores when compared to the ground truth. In summary, our rank-oriented designs consistently enhance object detection performance, particularly the AP scores under high IoU thresholds.
To validate the efficacy of our approach, we conducted comprehensive experiments, showcasing consistent performance improvements across recent strong DETR-based methods such as H-DETR and DINO-DETR. For example, based on H-DETR, our method demonstrates a notable increase in  $\mathrm{AP}_{75}$  of  $+2.1\%$  (52.9% vs. 55.0%) and  $+2.7\%$  (55.1% vs. 57.8%) when utilizing ResNet-50 and Swin-T backbones, respectively. It is worth highlighting that our approach achieves competitive performance, reaching an  $50.2\%$  AP in the  $1\times$  training schedule on the COCO val dataset. These results serve as compelling evidence for the effectiveness and reliability of our proposed methodology.
DETR for Object Detection. Since the groundbreaking introduction of transformers in 2D object detection by the pioneering work DETR [3], numerous subsequent studies[48, 11, 7, 63, 46] have developed diverse and advanced extensions based on DETR. This is primarily due to DETR's ability to eliminate the need for hand-designed components such as non-maximum suppression (NMS). One of the first foundational developments, Deformable-DETR [78], introduced a multi-scale deformable self/cross-attention scheme, which selectively attends to a small set of key sampling points in a reference bounding box. This approach yielded improved performance compared to DETR, particularly for small objects. Furthermore, DAB-DETR [44] and DN-DETR [34] demonstrated that a novel query formulation could also enhance performance. The subsequent work, DINO-DETR [75], achieved state-of-the-art results in object detection tasks, showcasing the advantages of DETR design by addressing the inefficiency caused by the one-to-one matching scheme. In contrast to these works, our focus lies in the design of the rank-oriented mechanism for DETR. We propose rank-oriented architecture designs and rank-oriented matching cost and loss function designs to construct a highly performant DETR-based object detector with competitive  $\mathrm{AP}_{75}$  results.
Ranking for Object Detection. There exists a lot of effort to study how to improve the ranking for object detection tasks. For example, IoU-Net [31] constructed an additional IoU predictor and an IoU-guided NMS scheme that considers both classification scores and localization scores during inference. Generalized focal loss [37] proposed a quality focal loss to act as a joint representation of the IoU score and classification score. VarifocalNet [76] introduced an IoU-aware classification score to achieve a more accurate ranking of candidate detection results. TOOD [10] defined a high-order combination of the classification score and the IoU score as the anchor alignment metric to encourage the object detector to focus on high-quality anchors dynamically. In addition, ranking-based loss functions [53, 4, 49, 50, 32] are designed to encourage the detector to rank the predicted bounding boxes according to their quality and penalizes incorrect rankings. The very recent con-current work Stable-DINO [45] and Align-DETR [1] also applied the idea of IoU-aware classification score to improve the loss and matching design for DINO-DETR [75]. In contrast to the aforementioned endeavors, we further introduce a query rank scheme aimed to reduce false positive rates.
Dynamic Neural Networks. In contrast to static models, which have fixed computational graphs and parameters at the inference stage, dynamic neural networks [15, 65] can adapt their structures or parameters to different inputs, leading to notable advantages in terms of performance, adaptiveness [68, 13], computational efficiency [72, 73, 67, 60], and representational power [52]. Dynamic networks are typically categorized into three types: sample-wise [27, 66, 18, 14, 51], spatial-wise [62, 28, 19, 17, 16], and temporal-wise [20, 64]. In this work, we introduce a novel query-wise dynamic approach, which dynamically integrates ranking information into the object queries based on their box quality ranking, endowing object queries with better representation ability.
Above all, we revisit the overall pipeline of the modern DETR-based methods [30, 75] in Section 3.1. The detailed design of the proposed method, including the rank-oriented architecture design and optimization design, is subsequently illustrated in Section 3.2 and Section 3.3. Eventually, we discuss the connections and differences between our approach and related works in Section 3.4.
Pipeline. Detection Transformers (DETRs) process an input image  $\mathcal{I}$  by first passing it through a backbone network and a Transformer encoder to obtain a sequence of enhanced pixel embeddings  $\mathcal{X} = \{\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_N\}$ . The enhanced pixel embeddings, along with a default set of object query embeddings  $\mathcal{Q}^0 = \{\mathbf{q}_1^0, \mathbf{q}_2^0, \dots, \mathbf{q}_n^0\}$ , are then fed into the Transformer decoder. After each Transformer decoder layer, task-specific prediction heads are applied to the updated object query embeddings to generate a set of classification predictions  $\mathcal{P}^l = \{\mathbf{p}_1^l, \mathbf{p}_2^l, \dots, \mathbf{p}_n^l\}$  and bounding box predictions  $\mathcal{B}^l = \{\mathbf{b}_1^l, \mathbf{b}_2^l, \dots, \mathbf{b}_n^l\}$ , respectively, where  $l \in \{1, 2, \dots, L\}$  denotes the layer index of the Transformer decoder. Finally, DETR performs one-to-one bipartite matching between the predictions and the ground-truth bounding boxes and labels  $\mathcal{G} = \{\mathbf{g}_1, \mathbf{g}_2, \dots, \mathbf{g}_m\}$  by associating each ground truth with the prediction that has the minimal matching cost and applying the corresponding supervision.
Object Query. To update the object query  $\mathcal{Q}^0$  after each Transformer decoder layer, typically, DE-TRs form a total of  $L$  subsets, i.e.,  $\{\mathcal{Q}^1,\mathcal{Q}^2,\dots ,\mathcal{Q}^L\}$ , for  $L$  Transformer decoder layers. For both the initial object query  $\mathcal{Q}^0$  and the updated ones after each layer, each  $\mathcal{Q}^l$  is formed by adding two parts: content queries  $\mathcal{Q}_c^l = \{\mathbf{q}_{c,1}^l,\mathbf{q}_{c,2}^l,\dots ,\mathbf{q}_{c,n}^l\}$  and position queries  $\mathcal{Q}_p^l = \{\mathbf{q}_{p,1}^l,\mathbf{q}_{p,2}^l,\dots ,\mathbf{q}_{p,n}^l\}$ . The content queries capture semantic category information, while the position queries encode prior positional information such as the distribution of bounding box centers and sizes.
Ranking in DETR. Rank-oriented design plays a crucial role in modern object detectors, particularly in achieving superior average precision (AP) scores under high Intersection over Union (IoU) thresholds. The success of state-of-the-art detectors, such as H-DETR and DINO-DETR, relies on using simple rank-oriented designs, specifically a two-stage scheme and mixed query selection. These detectors generate the initial positional query  $\mathcal{Q}_p^0$  by ranking the dense coarse bounding box predictions output by the Transformer encoder feature maps and selecting the top  $\sim 300$  confident ones. During evaluation, they gather  $n \times K$  bounding box predictions based on the object query embedding  $\mathcal{Q}^L$  produced by the final Transformer decoder layer (each query within  $\mathcal{Q}^L$  generates  $K$  predictions associated with each category), sort them by their classification confidence scores in descending order, and only return the top  $\sim 100$  most confident predictions.
In this work, we focus on further extracting the benefits brought by the ranking-oriented designs and introduces a set of improved designs to push the envelope of high-quality object detection performance. The subsequent discussion provides further elaboration on these details.
3.2 Rank-oriented Architecture Design: ensure lower FP and FN
While the original rank-oriented design only incorporates rank information into the initial positional query  $\mathcal{Q}_p^0$ , we propose an enhanced approach that leverages the benefits of sorting throughout the entire Transformer decoder process. Specifically, we introduce a rank-adaptive classification head after each Transformer decoder layer, and a query rank layer before each of the last  $L - 1$  Transformer decoder layers. This novel design is intended to boost the detection of true positives while suppressing false positives and correcting false negatives, leading to lower false positive rates and false negative rates. Figure 1 illustrates the detailed pipeline of our rank-oriented architecture designs.
Rank-adaptive Classification Head. We modify the original classification head by adding a set of learnable logit bias vectors  $S^l = \{\mathbf{s}_1^l, \mathbf{s}_2^l, \dots, \mathbf{s}_n^l\}$  to the classification scores  $\mathcal{T}^l = \{\mathbf{t}_1^l, \mathbf{t}_2^l, \dots, \mathbf{t}_n^l\}$  (before Sigmoid(·) function) associated with each object query independently. The classification predictions of the  $l$ -th decoder layer  $\mathcal{P}^l = \{\mathbf{p}_1^l, \mathbf{p}_2^l, \dots, \mathbf{p}_n^l\}$  can be formulated as:
$$
\mathbf {p} _ {i} ^ {l} = \operatorname {S i g m o i d} \left(\mathbf {t} _ {i} ^ {l} + \mathbf {s} _ {i} ^ {l}\right), \quad \mathbf {t} _ {i} ^ {l} = \mathrm {M L P} _ {\mathrm {c l s}} \left(\mathbf {q} _ {i} ^ {l}\right), \tag {1}
$$
where  $\mathcal{Q}^l = \{\mathbf{q}_1^l,\mathbf{q}_2^l,\dots ,\mathbf{q}_n^l\}$  represents the output embedding after the  $l$  -th Transformer decoder layer. The hidden dimensions of both  $\mathbf{t}_i^l$  and  $\mathbf{s}_i^l$  are the number of categories, i.e.,  $K$  . The overall
(a) Rank-oriented Arch.
(b) Rank-adaptive CIs. Head
(c) Query Rank Layer
Figure 1: Illustrating the rank-oriented architecture design. (a) The rank-oriented architecture consists of a query rank layer before each of the last  $L - 1$  Transformer decoder layer and a rank-adaptive classification head after each Transformer decoder layer. (b) The rank-adaptive classification head learns to adjust the classification scores accordingly. (c) The query rank layer exploits the latest ranking information to recreate the content queries and position queries used as the input to the following Transformer decoder layer.
pipeline is shown in Figure 1(b). It is noteworthy that we can directly incorporate a set of learnable embedding, denoted as  $S^l$ , into the classification scores  $\mathcal{T}^l$ . This is practicable because the associated  $\mathcal{Q}^l$  has already been sorted in the query rank layer, as explained below.
Query Rank Layer. We further introduce a query rank layer before each of the last  $L - 1$  Transformer decoder layers to regenerate the sorted positional query and content query accordingly.
First, we explain how to construct the rank-aware content query:
$$
\overline {{\mathcal {Q}}} _ {c} ^ {l} = \operatorname {M L P} _ {\text {f u s e}} \left(\hat {\mathcal {Q}} _ {c} ^ {l - 1} \| \mathcal {C} ^ {l}\right), \quad \hat {\mathcal {Q}} _ {c} ^ {l - 1} = \operatorname {S o r t} \left(\mathcal {Q} _ {c} ^ {l - 1}; \hat {\mathcal {P}} ^ {l - 1}\right), \tag {2}
$$
where we first sort the output of  $(l - 1)$ -th Transformer decoder layer  $\mathcal{Q}_c^{l - 1}$  in descending order of  $\hat{\mathcal{P}}^{l - 1} = \mathrm{MLP}_{\mathrm{cls}}(\mathcal{Q}_c^{l - 1})$ . Since each element in  $\hat{\mathcal{P}}^{l - 1}$  is a  $K$ -dimensional vector, we use the maximum value over  $K$  categories (classification confidence) as the ranking basis. The operation symbol  $\operatorname{Sort}(A;B)$  sorts elements within  $A$  based on the decreasing order of the elements in  $B$ . Then, we concatenate  $(\parallel)$  the sorted object content queries  $\hat{\mathcal{Q}}_c^{l - 1}$  with a set of randomly initialized content query  $\mathcal{C}^l$  in the feature dimension, where  $l\in \{2,\dots ,L\}$ . This set of content query  $\mathcal{C}^l$  is optimized in an end-to-end manner. Subsequently, we fuse them back to the original dimension using a fully connected layer  $(\mathrm{MLP}_{\mathrm{fuse}})$ . In other words, for each Transformer decoder layer, we maintain a set of rank-aware static content embeddings shared across different samples. These embeddings effectively model and utilize the distribution of the most frequent semantic information [26].
Next, we present the mathematical formulations for computing the rank-aware positional query. To align the order of the positional queries with the ranked content query, we either sort or recreate the positional queries, depending on the initialization method of positional queries for different DETR-based detectors. For H-DETR, which inherits Deformable DETR and uses the same positional query for all  $L$  Transformer decoder layers, we simply sort the positional query of the previous layer:
$$
\overline {{Q}} _ {p} ^ {l} = \operatorname {S o r t} \left(\overline {{Q}} _ {p} ^ {l - 1}; \hat {\mathcal {P}} ^ {l - 1}\right), \tag {3}
$$
For DINO-DETR, which generates new positional queries from the bounding box predictions in each Transformer decoder layer, we sort the bounding box predictions of each object query and recreate the positional query embedding from the sorted boxes:
$$
\overline {{\mathcal {Q}}} _ {p} ^ {l} = \operatorname {P E} \left(\bar {\mathcal {B}} ^ {l - 1}\right), \quad \bar {\mathcal {B}} ^ {l - 1} = \operatorname {S o r t} \left(\mathcal {B} ^ {l - 1}; \hat {\mathcal {P}} ^ {l - 1}\right), \tag {4}
$$
where  $\mathcal{B}^{l-1}$  and  $\hat{\mathcal{P}}^{l-1}$  represent the bounding box predictions and classification predictions based on the output of  $(l-1)$ -th Transformer decoder layer, i.e.,  $\mathcal{Q}^{l-1}$ .  $\mathrm{PE}(\cdot)$  contains a sine position encoding function and a small multilayer perceptron to recreate the positional query embedding  $\overline{\mathcal{Q}}_p^l$ . In other words, each element of  $\overline{\mathcal{Q}}_p^l$  is estimated by  $\overline{\mathbf{q}}_{p,i}^l = \mathrm{PE}(\overline{\mathbf{b}}_i^{l-1})$ . In Figure 1(c), we illustrate the positional query update process for H-DETR (Equation (3)) and omit that process for DINO-DETR (Equation (4)), because we primarily conducted experiments on H-DETR.
Finally, we transmit the regenerated rank-aware positional query embedding and content query embedding to the subsequent Transformer decoder layer.
(a) The original Matching Cost and Loss
(b) Rank-oriented Matching Cost and Loss
Figure 2: Illustrating the rank-oriented matching cost and loss design. (a) The original DETR and its variants use a classification head and a bounding box regression head to perform predictions. The matching cost function is a linear combination of the classification scores and bounding box overlap scores. (b) The rank-oriented matching cost and loss scheme uses a GIoU-aware classification head and high-order matching cost function to prioritize predictions of more accurate localization accuracy.
Analysis. The key motivations behind these two rank-oriented architecture designs are adjusting the classification scores of the object queries according to their ranking order information. Within each Transformer decoder layer, we incorporate two sets of learnable representations: the logit bias vectors  $S^l$  and the content query vectors  $C^l$ . By leveraging these two rank-aware representations, we have empirically demonstrated the capability of our approach to effectively address false positives (oLRP<sub>FP</sub>: 24.5%→24.1%) and mitigate false negatives (oLRP<sub>FN</sub>: 39.5%→38.6%). For a more comprehensive understanding of our findings, please refer to the experiment section.
3.3 Rank-oriented Matching Cost and Loss: boost the AP under high IoU thresholds
The conventional DETR and its derivatives specify the Hungarian matching cost function  $\mathcal{L}_{\mathrm{Hungarian}}$  and the training loss function  $\mathcal{L}$  using the identical manner, as shown below:
$$
- \lambda_ {1} \operatorname {G I o U} (\hat {\mathbf {b}}, \mathbf {b}) + \lambda_ {2} \ell_ {1} (\hat {\mathbf {b}}, \mathbf {b}) + \lambda_ {3} \operatorname {F L} (\hat {\mathbf {p}} [ c ]), \tag {5}
$$
where we use  $\hat{\mathbf{b}}$ ,  $\hat{\mathbf{p}}$  (or  $\mathbf{b}$ ,  $c$ ) to represent the predicted (or ground-truth) bounding box and classification score respectively.  $c$  corresponds to the ground-truth semantic category of  $\mathbf{b}$ . In addition, we use the notation  $\mathrm{FL}(\cdot)$  to represent the semantic classification focal loss [41]. We propose to enhance the rank-oriented design by introducing the GIoU-aware classification head (supervised by the proposed GIoU-aware classification loss) and the high-order matching cost function scheme as follows.
GIoU-aware Classification Loss. Instead of applying the binary target to supervise the classification head, we propose to use the normalized GIoU scores to supervise the classification prediction:
$$
\mathrm {F L} ^ {\mathrm {G I o U}} (\hat {\mathbf {p}} [ c ]) = - | t - \hat {\mathbf {p}} [ c ] | ^ {\gamma} \cdot [ t \cdot \log (\hat {\mathbf {p}} [ c ]) + (1 - t) \cdot \log (1 - \hat {\mathbf {p}} [ c ]) ], \tag {6}
$$
where we let  $t = (\mathrm{GIOU}(\hat{\mathbf{b}},\mathbf{b}) + 1) / 2$ , and we denote the GIOU-aware classification loss as  $\mathrm{FL}^{\mathrm{GIOU}}(\cdot)$ . To incorporate both semantic classification and localization accuracy, we modify the original classification head to a GIOU-aware classification head supervised by the above loss function. When  $t = 1$ ,  $\mathrm{FL}^{\mathrm{GIOU}}(\cdot)$  simplifies to the original focal loss, denoted as  $\mathrm{FL}(\cdot)$ . Furthermore, we compare our results with the varifocal loss [76], and we provide its mathematical formulation below:
$$
\operatorname {V F L} (\hat {\mathbf {p}} [ c ]) = - t \cdot [ t \cdot \log (\hat {\mathbf {p}} [ c ]) + (1 - t) \cdot \log (1 - \hat {\mathbf {p}} [ c ]). \tag {7}
$$
High-order Matching Cost. In contrast to utilizing the Hungarian algorithm with a matching cost defined as the weighted sum of classification cost,  $\ell_1$  loss, and GIoU loss, we propose employing a high-order matching cost function that captures a more intricate combination of the classification score and the IoU scores:
$$
\mathcal {L} _ {\text {H u n g a r i a n}} ^ {\text {h i g h - o r d e r}} = \hat {\mathbf {p}} [ c ] \cdot \operatorname {I o U} ^ {\alpha}, \tag {8}
$$
where IoU represents the intersection over union scores between the predicted bounding box and the ground truth one. We adopt a larger value of  $\alpha$  (e.g.,  $>2$ ) to prioritize the importance of localization accuracy, thereby promoting more accurate bounding box predictions and downgrading the inaccurate ones. It is worth noting that we use a high-order matching cost from the middle training stage, as most predictions exhibit poor localization quality during the early training stage.
Analysis. The rank-oriented loss function and matching cost are designed to enhance object detection performance at high IoU thresholds. The GIoU-aware classification loss facilitates the ranking