SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Method	Backbone	epochs	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_S \)	\( AP_M \)	\( AP_L \)
Basel#1 (Deformable) [45]	R50	12	43.7	62.0	47.3	26.2	46.8	58.1
Basel#1 (Deformable) [45]	R50	36	46.8	66.0	50.6	29.4	50.1	61.4
H-DETR [12]	R50	12	45.9	-	-	-	-	-
DN-Deformable-DETR [15]	R50	12	46.0	63.8	49.9	27.7	49.1	62.3
DAC-DETR (ours)	R50	12	47.1 (+3.4)	64.8	51.1	29.2	50.6	62.4
DAC-DETR (ours)	R50	24	48.5 (+1.7)*	66.5	52.6	31.0	51.8	63.1
Basel#2 (Deformable++) [12]	R50	12	47.6	65.9	52.3	30.2	51.1	62.6
Basel#2 (Deformable++) [12]	R50	36	49.3	67.5	53.6	32.1	52.5	64.4
H-DETR [12]	R50	12	48.7	66.4	52.9	31.2	51.5	63.5
H-DETR [12]	R50	36	50.0	68.3	54.4	32.9	52.7	65.3
DAC-DETR (ours)	R50	12	49.3 (+1.7)	66.5	53.8	31.4	52.4	64.1
DAC-DETR (ours)	R50	24	50.5 (+1.2)*	67.9	55.2	33.2	53.5	64.8
Basel#3 (DINO) [42]	R50	12	49.0	66.6	53.5	32.0	52.3	63.0
Basel#3 (DINO) [42]	R50	24	50.4	68.3	54.8	33.3	53.7	64.8
Group-DETR [5]	R50	12	49.8	-	-	32.4	53.0	64.2
Align-DETR [1]	R50	12	50.2	67.8	54.4	32.9	53.3	65.0
Align-DETR [1]	R50	24	51.3	68.2	56.1	35.5	55.1	65.6
Stable-DINO-4scale [20]	R50	12	50.4	67.4	55.0	32.9	54.0	65.5
Stable-DINO-4scale [20]	R50	24	51.5	68.5	56.3	35.2	54.7	66.5
DAC-DETR (ours)	R50	12	50.0 (+1.0)	67.6	54.7	32.9	53.1	64.2
DAC-DETR (ours)	R50	24	51.2 (+0.8)	68.9	56.0	34.0	54.6	65.4
DAC-DETR + Align (ours)	R50	12	50.9 (+1.9)	68.3	55.3	33.8	54.1	65.7
DAC-DETR + Align (ours)	R50	24	52.1 (+1.7)	69.7	56.5	34.8	55.2	67.3
Table 1: Evaluation on COCO val2017 with ResNet-50 backbone. For fair comparison, the methods are divided into three groups according to their baseline, i.e., Basel#1: Deformable-DETR [45], Basel#2: Deformable-DETR++ [12], Basel#3: DINO [42].*: We compare performance between 24-epoch DAC-DETR and 36-epoch baseline.
Method	Backbone	epochs	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_S \)	\( AP_M \)	\( AP_L \)
H-DETR [12]	Swin-L	12	56.1	75.2	61.3	39.3	60.4	72.4
H-DETR [12]	Swin-L	36	57.6	76.5	63.2	41.4	61.7	73.9
Basel#3 (DINO) [42]	Swin-L	12	56.8	75.6	62.0	40.0	60.5	73.2
Basel#3 (DINO) [42]	Swin-L	36	58.0	77.1	66.3	41.3	62.1	73.6
Group-DETR [5]	Swin-L	36	58.4	-	-	41.0	62.5	73.9
Stable-DINO-4scale [20]	Swin-L	12	57.7	75.7	63.4	39.8	62.0	74.7
Stable-DINO-4scale [20]	Swin-L	24	58.6	76.7	64.1	41.8	63.0	74.7
DAC-DETR (ours)	Swin-L	12	57.3 (+0.5)	75.7	62.7	40.1	61.5	74.4
DAC-DETR + Align (ours)	Swin-L	12	58.1 (+1.3)	76.5	63.3	40.9	62.4	75.0
DAC-DETR + Align (ours)	Swin-L	24	59.2 (+1.2)*	77.1	64.5	43.1	63.4	76.0
Table 2: Evaluation on COCO val2017 with Swin-Transformer Large backbone. Most competing methods use the same baseline, i.e., DINO [42].*: We compare performance between 24-epoch DAC-DETR and 36-epoch baseline.
by using standard average precision (AP) result under different IoU thresholds. We implement DAC-DETR with two popular backbones, i.e., ResNet50 [10] (pretrained on ImageNet-1k [7]) and Swin-Large [21] (pretrained on ImageNet-22k [7]).
Baseline and implementation details. The baselines for recent state-of-the-art methods vary a lot. For fair comparison with these methods, we adopt three recent popular baselines, e.g., the original Deformable-DETR [45], an improved Deformable DETR (Deformable++) [12] and the DINO [42] baseline. As for training, we use AdamW [22, 14] optimizer with weight decay of  $1 \times 10^{-4}$ . We report experimental results under the  $1 \times$  (12 epochs) and  $2 \times$  (24 epochs) scheme. We find the  $2 \times$  learning scheme already allows DAC-DETR to achieve superior accuracy, compared with most competing methods (some of which uses longer training epochs).
We evaluate DAC-DETR on COCO 2017 detection validation dataset. The results on ResNet50 [10] and Swin-Transformer Large backbones [21] are summarized in Table 1 and Table 2, respectively.
Method	Backbone	epochs	AP	\( AP_{50} \)
Baseline (Deformable-DETR [45])	R50	12	43.7	63.0
Variant1 (freeze cross-atten layers in O-Decoder)	R50	12	45.9	63.3
Variant2 (add self-atten layers in C-Decoder)	R50	12	43.1	58.9
Variant3 (one-to-one matching in C-Decoder)	R50	12	43.5	62.1
DAC-DETR (ours)	R50	12	47.1	64.8
Table 3: Comparison of four variants for DAC-DETR.
Method	Backbone	epochs	AP	\( AP_{50} \)	\( AP_{75} \)
Basel#1 (Deformable) [45]	R50	12	43.7	63.0	47.6
DAC-DETR w/ one-to-many Hungarian	R50	12	45.5	64.0	49.3
DAC-DETR (ours)	R50	12	47.1	64.8	51.1
Basel#2 (Deformable++) [12]	R50	12	47.0	65.2	51.5
DAC-DETR w/ one-to-many Hungarian	R50	12	48.4	65.6	52.8
DAC-DETR (ours)	R50	12	49.3	66.5	53.8
Basel#3 (DINO) [42]	R50	12	49.0	66.6	53.5
DAC-DETR w/ one-to-many Hungarian	R50	12	49.3	66.7	53.8
DAC-DETR (ours)	R50	12	50.0	67.6	54.7
Table 4: Comparison of different one-to-many matching strategies for DAC-DETR. Hungarian-based one-to-many matching is useful but is inferior to our threshold-based strategy.
Based on the ResNet50 backbone, we employ three different baselines in Table 1 ( with ResNet50 backbones), from which we draw two observations: 1) DAC-DETR achieves consistent and remarkable improvements over all three baselines. For example, under  $1\times$  learning scheme, DAC-DETR improves Deformable-DETR [45], Deformable-DETR++ [12] and DINO [42] by  $+3.4$ ,  $+2.3$  and  $+1.0$  AP, respectively. 2), DAC-DETR achieves competitive detection accuracy. We note that recent state-of-the-art methods usually adopt a strong baseline (e.g., DINO) and further integrate a battery of good practices. Among these methods, Stable-DINO is the strongest and combines an IoU-related loss, an improved matching strategy and a novel feature fusion. On the same baseline (DINO), our method (DAC-DETR + Align), while employing fewer tricks (only Align Loss), surpasses Stable-DINO by  $+0.5$  AP (12 epochs) and  $+0.6$  AP (24 epochs).
On the Swin-L backbone (where relatively fewer methods have reported results), we only adopt the strongest baseline (DINO) and compare DAC-DETR against most recent state-of-the-art methods. The results in Table 2 further confirms the effectiveness of DAC-DETR: it improves DINO baseline by  $+0.5$  AP under 12 epochs. After combining an IoU-related loss (Align [1] loss), DAC-DETR surpasses the strongest competitor Stable-DINO by  $+0.4$  AP (12 epochs) and  $+0.6$  AP (24 epochs).
Three variants of DAC-DETR. In Table 3, we are interested in two questions: 1) Since the cross-attention layers already undergo specific training in C-Decoder, can they be frozen in O-Decoder (Variant-1)? 2) Are removing self-attention layers and using one-to-many matching important for C-Decoder? In response, we implement Variant-2&3, which adds the self-attention layers and performs one-to-one matching for C-Decoder, respectively. From Table 3, we draw the two following observations:
First, freezing the cross-attention layers in O-Decoder considerably compromises DAC-DETR (-1.2 AP), but still improves the baseline (+2.2). It shows that training the cross-attention layers only in C-Decoder already suffices a competitive DETR detector and brings major improvement. We thus infer that C-Decoder makes the major contribution to learning the cross-attention layers. On the O-Decoder side, it pays relatively fewer efforts onto cross-attention layers and channels its efforts onto learning the self-attention layers. Second, adding self-attention and using one-to-one matching for C-Decoder both deteriorate DAC-DETR. These two variants are even inferior to the baseline by -0.6 and -0.2 AP respectively. We thus infer that removing the self-attention layers and using
Method	Backbone	epochs	AP	\( AP_{50} \)
Deformable-DETR [45]	R50	12	43.7	63.0
DAC-DETR w/o regression loss in C-decoder	R50	12	45.7	64.0
DAC-DETR w/o sync position	R50	12	46.8	64.3
DAC-DETR (full)	R50	12	47.1	64.8
Table 5: Investigation on some details for learning the query position.
Method	Backbone	queries	AP
Deformable-DETR [45]	R50	300	43.7
H-DETR (our implementation)	R50	300 + 1500	46.4
H-DETR w/o self-atten in one-to-many decoder	R50	300 + 1500	47.0
DAC-DETR	R50	300	47.1
Table 6: Removing the self-attention in the one-to-many decoder improves H-DETR, as well.
one-to-many matching are critical. Being featured for no self-attention layers, C-Decoder has no capability to remove duplicate predictions and thus favors one-to-many matching.
Comparison of different one-to-many matching strategies. In DAC-DETR, C-Decoder adopts a threshold-based one-to-many matching strategy. We investigate another alternative, i.e., Hungarian matching with repeated ground-truth in H-DETR [12] in Table 4 and draw two observations:
First, comparing "DAC-DETR w/ one-to-many Hungarian" against the baseline, we observe consistent improvement. It shows that C-Decoder is compatible to Hungarian-based one-to-many matching, as well. Second, comparing two one-to-many strategies against each other, we observe our threshold-based strategy is better. This is reasonable because Hungarian-based one-to-many matching is based on set-to-set comparison and requires global comparison among all the samples. C-Decoder has no self-attention layers for global comparison and thus favors the threshold-based strategy.
Some details for learning the query position. Table 5 investigates some training details w.r.t. the query position. It is observed that removing the regression loss for C-Decoder achieves relatively small improvement. It is consistent with our motivation in Fig. 1, i.e., the gathering effect concerns the position as well. Moreover, there is a training trick, i.e., given a same query input, we synchronize its position in O-Decoder and C-Decoder through average. This operation brings +0.3 AP improvement on DAC-DETR.
Can removing self-attention benefit other one-to-many decoders? Table 3 (Variant2) shows removing self-attention is critical for C-Decoder. Table 6 further investigates this design on another popular one-to-many decoder, H-DETR [12] and shows non-trivial improvement (+0.6AP) in 12 training epochs, as well. It further confirms that removing the self-attention enhances the effect of gathering multiple queries towards each object on different one-to-many decoders.
This paper reveals in the DETR decoder, the cross-attention and self-attention layers have opposing impacts (i.e., gathering and dispersing) on the object queries. This insight motivates us to divide the cross-attention layers out from this opposition for better conquering. The corresponding method, DAC-DETR, improves both the quantity and quality of the queries that are gathered by each object. Experimental results show that DAC-DETR improves multiple popular DETR baselines (e.g., Deformable-DETR, DINO) and performs favorably against recent state-of-the-art methods.
Acknowledgements. This work is supported by the Fundamental Research Funds for the Central Universities (No. 226-2023-00048).
Broader Impacts. This paper utilizes the open dataset to evaluate the performance. Our method can improve the object detection, which could be applied to automatic driving systems. We will enhance the generalization of method.
[37] Ashish Vavwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[38] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2567-2575, 2022.
[39] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. arXiv preprint arXiv:2310.01403, 2023.
[40] Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318, 2021.
[41] Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, and Shijian Lu. Accelerating detr convergence via semantic-aligned matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 949-958, 2022.
[42] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022.
[43] Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo, and Kai Chen. Dense distinct query for end-to-end object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7329-7338, June 2023.
[44] Chuyang Zhao, Yifan Sun, Wenhao Wang, Qiang Chen, Errui Ding, Yi Yang, and Jingdong Wang. MS-DETR: Efficient DETR training with mixed supervision, 2023.
[45] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.