Title: DAC-DETR: Divide the Attention Layers and Conquer
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep learning

computer vision

object detection

transformer
Abstract: This paper reveals a characteristic of DEtection Transformer (DETR) that negatively impacts its training efficacy, i.e., the cross-attention and self-attention layers in DETR decoder have contrary impacts on the object queries (though both impacts are important). Specifically, we observe the cross-attention tends to gather multiple queries around the same object, while the self-attention disperses these queries far away. To improve the training efficacy, we propose a Divide-And-Conquer DETR (DAC-DETR) that divides the cross-attention out from this contrary for better conquering. During training, DAC-DETR employs an auxiliary decoder that focuses on learning the cross-attention layers. The auxiliary decoder, while sharing all the other parameters, has NO self-attention layers and employs one-to-many label assignment to improve the gathering effect. Experiments show that DAC-DETR brings remarkable improvement over popular DETRs. For example, under the 12 epochs training scheme on MS-COCO, DAC-DETR improves Deformable DETR (ResNet-50) by +3.4 AP and achieves 50.9 (ResNet-50) / 58.1 AP (Swin-Large) based on some popular methods (i.e., DINO and an IoU-related loss). Our code will be made available at https://github.com/huzhengdongcs/DAC-DETR.

DAC-DETR: Divide the Attention Layers and Conquer

Abstract

This paper reveals a characteristic of DEtection Transformer (DETR) that negatively impacts its training efficacy, i.e. , the cross-attention and self-attention layers in DETR decoder have opposing impacts on the object queries (though both impacts are important). Specifically, we observe the cross-attention tends to gather multiple queries around the same object, while the self-attention disperses these queries far away. To improve the training efficacy, we propose a Divide-And-Conquer DETR (DAC-DETR) that separates out the cross-attention to avoid these competing objectives. During training, DAC-DETR employs an auxiliary decoder that focuses on learning the cross-attention layers. The auxiliary decoder, while sharing all the other parameters, has NO self-attention layers and employs one-to-many label assignment to improve the gathering effect. Experiments show that DAC-DETR brings remarkable improvement over popular DETRs. For example, under the 12 epochs training scheme on MS-COCO, DAC-DETR improves Deformable DETR (ResNet50) by +3.4AP and achieves 50.9 (ResNet-50) / 58.1 AP (Swin-Large) based on some popular methods ( i.e. , DINO and an IoU-related loss). Our code will be made available at https://github.com/huzhengdongcs/DAC-DETR .

1 Introduction

Though DEtection Transformers (DETRs) have already achieved great progress, their low training efficacy problem remains a critical challenge. The research community has been paying great efforts to locate the causes of this problem and to improve the training efficacy. For example, DN-DETR [15] and DINO [42] find the one-to-one matching is prone to inconsistent optimization goals. In response, they propose to accelerate DETR training with auxiliary denoising queries. Hybrid DETR [12] and Group DETR [5] attribute the reason to the rare positive training queries and correspondingly increase positive queries through additional one-to-many matching. Some other works [20, 1] notice the misalignment between the predicted probability and position. They suppress such misalignment through IoU-related loss functions and achieve improvement.

This paper, from another perspective, investigates this problem by rethinking the cooperation between cross-attention and self-attention layers in the DETR decoder. We find during their cooperation, these two attention types actually have some opposing impacts on the object queries. These two impacts are both critical for DETR (as explained later), but their opposition impairs the training efficacy. Specifically, we observe the cross-attention layers in DETR decoder tends to gather multiple queries around a single object, while the self-attention disperses these queries far away from each other. The above 'gather ↔ disperse' phenomenon concerns not only the positional relationship (1st row in Fig. 1), but also the feature distance (2nd row in Fig. 1 visualized by t-SNE [36]):

(b)

(c) decoder w/ self-attention

(a)

q1

q2

q3

q4

q1

q2

q3

q4

q1

q4

q2

q3

query distribution (tSNE)

before decoder

decoder w/o self-atttention

query position

q3

q2

q2 (cls:0.42)

q2 (cls:0.14)

q1

q4

q3 (cls:0.40)

q1 (cls:0.44)

q4 (cls:0.36)

q1 (cls:0.60)

q3 (cls:0.13)

q4 (cls:0.09)

Cross-attention layers tend to gather multiple queries around the same object. Given an alreadytrained Deformable DETR [45], we remove all the self-attention layers in its decoder. This removal compromises DETR to duplicate detections. As illustrated in Fig. 1 (b), multiple queries locate a same object ('bear') with relatively large Intersection-over-Union (IoU). Compared with their initial states (before the decoder) in Fig. 1 (a), they become closer towards each other, regarding both their position (1st row) and feature distance (2nd row).

Self-attention layers disperse these queries from each other. In Fig. 1 (c), we restore the original deformable DETR. Correspondingly, the queries are dispersed from each other regarding both position (1st row) and feature distance (2nd row). Due to this dispersion, most queries become farther to their original center point, except that a single query (with the highest classification score) approaches even closer. This phenomenon explains how DETR makes non-duplicate detection, and is consistent with the consensus [23, 3, 41, 25], i.e. , self-attention layers play a critical role in removing duplicates.

In spite of their opposition, these two effects are both critical for DETR. The gathering effect allows the queries to aggregate feature from the nearby object. The dispersing effect allows the self-attention layers to remove duplicate detection. However, learning the stacked self-attention and cross-attention layers within such 'gather ↔ disperse' opposition is difficult. To improve the training efficacy, we propose a Divide-and-Conquer DETR (DAC-DETR) that divides the cross-attention out from this opposition for better conquering . We explain the key points of DAC-DETR in more details as below:

' Divide ': DAC-DETR employs an auxiliary decoder that focuses on learning the cross-attention layers. The auxiliary decoder has NO self-attention layers and shares all the other parameters ( i.e. , the cross-attention layers and the feed-forward networks) with the original decoder (O-Decoder). We name the auxiliary decoder as C-Decoder to highlight its focus on the cross-attention.

' Conquer ': During training, DAC-DETR feeds all the queries into two decoders in parallel. O-Decoder uses the one-to-one matching to learn non-duplicate detection. In contrast, C-Decoder uses the one-to-many matching that assigns each object with multiple positive queries. C-Decoder has no self-attention layers and is prone to duplicate detection results. Therefore, using one-to-many label assignment is a natural choice and benefits learning the gathering effect with more positive queries. During inference, DAC-DETR discards C-Decoder and maintains the baseline efficiency.

Through the above 'divide-and-conquer', DAC-DETR defuses the opposition when learning the 'gathering' and 'dispersing' effects which are both important for DETR. Mechanism analysis in Section 3.3 shows that due to the enhanced feature aggregation ability, DAC-DETR improves both the quantity and quality of intermediate queries that are gathered to each ground-truth object. Based on these improved queries, DETR increases the detection accuracy (after removing duplicate predictions).

We conduct extensive experiments to validate the effectiveness of DAC-DETR and empirically show remarkable improvement over various DETRs. For example, based on a popular baseline, i.e. , ResNet-50 Deformable DETR [45], DAC-DETR brings +3.4 AP improvement and achieves 47.1 AP on MS-COCO within 12 ( 1 × ) training epochs. On some more recent state-of-the-art methods (that usually integrate a battery of good practices), DAC-DETR still gains consistent and complementary benefits. For example, under the 1 × learning scheme on MS-COCO, DAC-DETR improves ResNet50 DINO to 50.0 (+1.0) AP. Combining an IoU-related Loss in Align DETR [1], our method further achieves 50.9 AP (ResNet-50) / 58.1 AP (Swin-Large), surpassing Align DETR itself by +0.7 AP. Moreover, the achieved results (50.9 and 58.1 AP) are also higher than Stable DETR [20] (50.4 and 57.7 AP), which adopts even more good practices ( i.e. , stable matching and memory fusion).

Our main contributions are summarized as follows: First, we reveal the 'gather ↔ disperse' opposition between cross-attention and self-attention layers in DETR decoder as a reason for its low training efficacy. Second, we propose to divide the cross-attention layers out from this opposition for better conquering the DETR training, yielding the so-called Divide-and-Conquer DETR (DAC-DETR). Third, we empirically show that DAC-DETR brings remarkable improvement over popular DETR baselines and performs favorably against most recent state-of-the-art methods.

2 Related works

DETR-like object detectors. The original Detection Transformer (DETR) [37, 3] is featured for its end-to-end detection without any hand-crafted modules ( e.g. , Non-maximum Suppression). Since DETR [3] suffers from low training convergence, many efforts [45, 33, 23, 42, 38, 6, 8, 34, 43, 39, 32, 16] have been paid to improve the training efficacy. Deformable DETR [45] proposes to replace traditional attention modules with deformable attention and improves the training convergence. Conditional DETR [23] performs the conditional spatial query for decoder multi-head cross-attention. DAB-DETR [19] utilizes box coordinates as priors and dynamically updates anchor boxes layerby-layer. DN-DETR [15] introduces denoising part to stabilize the bipartial matching, and DINO [42] further adds contrastive learning to enhance the performance. Some works [20, 1] propose IoU-related loss functions to align the predicted probability score against position. This paper, from another perspective, finds a opposing impact between the cross-attention and self-attention layers and proposes to divide the attention layers during training.

Attention layers in DETR decoder. The cross-attention and self-attention layers are both essential for DETRs. The former allows the object queries to aggregate information from the image features, while the latter facilitates query-to-query interaction to remove duplicate detection [23, 45, 3, 2, 9, 40, 31]. Our observation in this paper is consistent with the above consensus and goes a step forward: the cross-attention layers tend to gather multiple queries towards each object, while the self-attention disperses them far away. By separating the cross-attention layers out from this opposition, the proposed DAC-DETR improves DETR training efficacy.

One-to-many label assignment. One-to-many label assignment is widely utilized in CNN-based detectors [29, 27, 28, 11, 4, 35, 13]. these methods produce duplicate predictions on each object and uses hand-crafted non-maximum suppression (NMS) [24] for post-processing. As the DETRs prevail, some recent literature [5, 12, 26] explore one-to-many label assignment for DETRs and demonstrate benefit. Group DETR [5] employs multiple parallel object queries, which jointly contribute multiple positive queries for each object. Hybrid DETR [12] adds an auxiliary query branch that performs one-to-many label assignment.

DAC-DETR uses the one-to-many label assignment for the auxiliary C-Decoder, because C-Decoder has no self-attention layers that help removing duplicate predictions. Therefore, DAC-DETR also gains the benefit from one-to-many assignment. That being said, the benefit from one-to-many assignment is not our key contribution but a natural gain. While a co-current work Hybrid DETR [12] already shows that adding a one-to-many decoder improves DETR, our method shows that removing

…

…

Cross-Attention

Add & Norm & FFN

heads

heads

C-Decoder

share

Backbone

Transformer Encoder

…

object queries

L

x

L

x

Self-Attention

O-Decoder

Cross-Attention

Add & Norm

Add & Norm & FFN

One-to-one matching

One-to-many matching

Input image

split

NO Self

Attention

share

Encoder outputs

the self-attention in the one-to-many decoder is better (e.g., +0.6 AP on MS-COCO in 12 training epochs, as detailed in Table 6 ).

3 Methods

3.1 Overview

Preliminaries on DETRs. A DETR-style detector typically consists of a backbone ( e.g. , ResNet [10], Swin Transformer [21]), a transformer encoder and a transformer decoder. Given an input image, the DETR-style detector first feeds it into the backbone to extract image features, splits the features into patch tokens and then enhances the patch tokens by the transformer encoder. The enhanced tokens from the encoder are Z = { z 1 , ..., z m } . The transformer decoder takes multiple object queries Q = { q 1 , ..., q n } as its input. The decoder has L stacked decoder blocks. In each decoder block, the object queries Q sequentially undergo a self-attention layer, a cross-attention layer, and a feed-forward network (FFN). Specfically, the cross-attention layer makes Q interact with the patch tokens Z , so as to aggregate features for locating and identifying the objects. Afterwards, the FFN transforms Q into the output embeddings, which make final prediction through a classification and regression head.

DAC-DETR follows the standard DETR pipeline for the backbone and transformer encoder and is featured for the 'Divide-and-Conquer' decoders. As illustrated in Fig. 2, DAC-DETR has two decoders, i.e. , 1) an O-Decoder which is the same as the baseline DETR decoder, and 2) a C-Decoder that divides the cross-attention layers from the 'gather ↔ disperse' for better learning. C-Decoder removes all the self-attention layers and shares all the other decoder parameters (cross-attention, FFN), object queries and the prediction head with the O-Decoder. Section 3.2 elaborates on these two decoders regarding their architecture, training and inference. Section 3.3 looks into the mechanism,

i.e. , how DAC-DETR impacts the learning of the DETR decoder. It shows that DAC-DETR improves both the quantity (number) and the quality (larger IoU) of queries that attend to each object.

3.2 Divide-and-Conquer Decoders

DAC-DETR constructs two parallel decoders, i.e. , a canonical DETR decoder (O-Decoder) and a decoder without self-attention layers (C-Decoder), as shown in Fig. 2. The initial object queries Q are fed into the first block of O-Decoder and C-Decoder in parallel. Since the object queries undergo different transformations in these two decoders, they are denoted as Q l O (queries in O-Decoder) and Q l C (queries in C-Decoder), where the superscript l ( l = 0 , 1 , · · · , 5) indicates the index of the decoder block. We have Q 0 O = Q 0 C = Q .

· O-Decoder: a canonical DETR-decoder revisit. O-Decoder is a canonical DETR decoder with each block consisting of a self-attention layer, a cross-attention layer and a FFN. Specifically, the l -th decoder block first updates all the queries through self-attention, which is formulated as:

where Que , Key , Val are the projections to derive the query , key and value embedding from tokens Q l O , respectively. We omit the description of 'Layer norm" and 'Dropout" in all equations for better clarity.

Given the queries Q l O output from the preceding self-attention layer, the cross-attention layer further updates them through interaction with the feature embedding Z , which is formulated as:

where Que , Key , Val are the projections and are parameterized differently from those in the self-attention layer (Eqn. 1), though sharing the same symbols.

Finally, the output embeddings Q l +1 O are fed into a predictor head to make label and position predictions. The predictions are supervised with the one-to-one matching and L O-decoder , which is the standard Hungarian loss [3] as in the original DETR.

· C-decoder removes all the self-attention layer in each decoder block and shares all the other parameters with O-decoder. Therefore, in C-Decoder, the queries undergo only cross-attention with the feature embeddings Z and a following FFN transformation, which is formulated as:

where all the model parameters are shared with the O-Decoder (Eqn. 2).

In parallel to the prediction from O-Decoder, the output embeddings from C-Decoder Q l +1 C are fed into the shared predictor head. The corresponding label and position prediction from a query q ∈ Q are [ p ( q ) , b ( q ) ] .

Different from the bipartite matching strategy in O-Decoder, C-Decoder employs one-to-many label assignment for supervision [26, 44], i.e. , assigning each ground-truth object with multiple queries. Specifically, given an object with ground-truth annotation y = [ˆ c, ˆ b ] ( ˆ c and ˆ b are the class and bounding box, respectively), we measure its matching scores with the prediction from all the queries by:

where p ( q ) (ˆ c ) denotes the predicted label score on class ˆ c , <,> denotes the IoU operation between predicted box and ground truth ˆ b .

Given the matching scores M for the ground-truth object, we select multiple positive predictions for it by two criteria: 1) the matching score should be larger than a threshold t and 2) the matching score should be among the top-k scores in M . There two criteria are required simultaneously and the second one is to suppress the label imbalance regarding different objects. All the other predictions are assigned with negative labels for this object. With the assigned labels, we supervise the predictions from C-Decoder with L C-decoder , which is a standard detection loss function consisting of classification [18] and regression loss [30].

0

5

10

15

20

1

2

3

4

5

6

7

8

9

10 11 12

t

= 0.6

DAC-DETR

Deformable-DETR

0

2

4

6

8

10

1

2

3

4

5

6

7

8

9

10 11 12

t

= 0.7

DAC-DETR

Deformable-DETR

0

0.4

0.8

1.2

1.6

2

2.4

1

2

3

4

5

6

7

8

9

10 11 12

t

= 0.8

DAC-DETR

Deformable-DETR

epochs

epochs

epochs

nums

nums

nums

During training, L O-decoder and L C-decoder are summed up. During inference, DAC-DETR discards C-Decoder and maintains the same pipeline and efficiency as the baseline.

Discussion. Using the one-to-many label assignment for supervising C-Decoder is a natural choice. It is because C-Decoder has no self-attention layers (which are critical for removing duplicate detections) and is thus prone to multiple detections for each single object. If C-Decoder employs the one-to-one label assignment, DAC-DETR will bring no more improvement but actually compromises the baseline accuracy (Section 4.3).

3.3 Mechanism Analysis

We investigate how the proposed divide-and-conquer method benefits DETR with focus on the gathering effects of the cross-attention layers. Specifically, we are interested in: how many queries can each ground-truth object gather in DAC-DETR and in the baseline? To this end, we count the averaged number of queries that have large affinity (according to the matching score in Eqn. 4) with each object. The queries with high matching scores are important: they aggregate abundant information from a corresponding object and thus serve as the proposals which are potential to predict the object. Based on these highly-matched queries (proposals), the self-attention layer removes redundant queries and gives the final non-duplicate prediction.

The comparison between DAC-DETR and the Deformable-DETR baseline is in Fig. 3.We list the results for the last decoder layer and leave the results for more layers to the supplementary. From Fig. 3, we draw two important observations as below:

Remark-1: DAC-DETR increases the number of queries gathered for each object. Comparing DAC-DETR and its baseline under various matching score threshold ( i.e. , t = 0 . 6 , 0 . 7 , 0 . 8 ), we observe that DAC-DETR consistently gathers more queries for each single object. Please note that the total number of queries is the same for DAC-DETR and the baseline. We thus infer that DAC-DETR makes higher utilization of the object queries.

Remark-2: DAC-DETR improves the quality of the best queries for each object. Focusing on the comparison under large threshold ( e.g. , t = 0 . 8 ), we find the baseline has rare high-quality queries, while DAC-DETR has about 1 high-quality query for each object. We recall that the matching score is composed of IoU and classification score. This suggests that some queries in DAC-DETR either has good IoU or high confidence to the ground-truth object.

Combining the above two remarks, we conclude that DAC-DETR improves both the quantity and quality of the queries for each object. This improvement is the direct reason how DAC-DETR increases the detection accuracy (Section 4).

4 Experiments

4.1 Setup

Dataset and backbone. We evaluate the proposed DAC-DETR on COCO 2017 [17] detection dataset. Following the common practices, we evaluate the performance on validation dataset(5k images)

by using standard average precision (AP) result under different IoU thresholds. We implement DAC-DETR with two popular backbones, i.e. , ResNet50 [10] (pretrained on ImageNet-1k [7]) and Swin-Large [21] (pretrained on ImageNet-22k [7]).

Baseline and implementation details. The baselines for recent state-of-the-art methods vary a lot. For fair comparison with these methods, we adopt three recent popular baselines, e.g. , the original Deformable-DETR [45], an improved Deformable DETR (Deformable++) [12] and the DINO [42] baseline. As for training, we use AdamW [22, 14] optimizer with weight decay of 1 × 10 -4 . We report experimental results under the 1 × (12 epochs) and 2 × (24 epochs) scheme. We find the 2 × learning scheme already allows DAC-DETR to achieve superior accuracy, compared with most competing methods (some of which uses longer training epochs).

4.2 Main Results

We evaluate DAC-DETR on COCO 2017 detection validation dataset. The results on ResNet50 [10] and Swin-Transformer Large backbones [21] are summarized in Table 1 and Table 2, respectively.

Based on the ResNet50 backbone, we employ three different baselines in Table 1 ( with ResNet50 backbones ), from which we draw two observations: 1) DAC-DETR achieves consistent and remarkable improvements over all three baselines. For example, under 1 × learning scheme, DAC-DETR improves Deformable-DETR [45], Deformable-DETR++ [12] and DINO [42] by +3.4, +2.3 and +1.0 AP, respectively. 2) , DAC-DETR achieves competitive detection accuracy. We note that recent state-of-the-art methods usually adopt a strong baseline ( e.g. , DINO) and further integrate a battery of good practices. Among these methods, Stable-DINO is the strongest and combines an IoU-related loss, an improved matching strategy and a novel feature fusion. On the same baseline (DINO), our method (DAC-DETR + Align) , while employing fewer tricks (only Align Loss), surpasses Stable-DINO by + 0.5 AP (12 epochs) and +0.6 AP (24 epochs).

On the Swin-L backbone (where relatively fewer methods have reported results), we only adopt the strongest baseline (DINO) and compare DAC-DETR against most recent state-of-the-art methods. The results in Table 2 further confirms the effectiveness of DAC-DETR: it improves DINO baseline by +0.5 AP under 12 epochs. After combining an IoU-related loss (Align [1] loss), DAC-DETR surpasses the strongest competitor Stable-DINO by + 0.4 AP (12 epochs) and +0.6 AP (24 epochs).

4.3 Ablation Studies

Three variants of DAC-DETR . In Table 3, we are interested in two questions: 1) Since the crossattention layers already undergo specific training in C-Decoder, can they be frozen in O-Decoder (Variant-1)? 2) Are removing self-attention layers and using one-to-many matching important for C-Decoder? In response, we implement Variant-2 & 3, which adds the self-attention layers and performs one-to-one matching for C-Decoder, respectively. From Table 3, we draw the two following observations:

F irst, freezing the cross-attention layers in O-Decoder considerably compromises DAC-DETR (-1.2 AP), but still improves the baseline (+2.2). It shows that training the cross-attention layers only in C-Decoder already suffices a competitive DETR detector and brings major improvement. We thus infer that C-Decoder makes the major contribution to learning the cross-attention layers. On the O-Decoder side, it pays relatively fewer efforts onto cross-attention layers and channels its efforts onto learning the self-attention layers. S econd, adding self-attention and using one-to-one matching for C-Decoder both deteriorate DAC-DETR. These two variants are even inferior to the baseline by -0.6 and -0.2 AP respectively. We thus infer that removing the self-attention layers and using

one-to-many matching are critical. Being featured for no self-attention layers, C-Decoder has no capability to remove duplicate predictions and thus favors one-to-many matching.

Comparison of different one-to-many matching strategies . In DAC-DETR, C-Decoder adopts a threshold-based one-to-many matching strategy. We investigate another alternative, i.e. , Hungarian matching with repeated ground-truth in H-DETR [12] in Table 4 and draw two observations:

First, comparing 'DAC-DETR w/ one-to-many Hungarian' against the baseline, we observe consistent improvement. It shows that C-Decoder is compatible to Hungarian-based one-to-many matching, as well. Second, comparing two one-to-many strategies against each other, we observe our thresholdbased strategy is better. This is reasonable because Hungarian-based one-to-many matching is based on set-to-set comparison and requires global comparison among all the samples. C-Decoder has no self-attention layers for global comparison and thus favors the threshold-based strategy.

Some details for learning the query position . Table 5 investigates some training details w.r.t. the query position. It is observed that removing the regression loss for C-Decoder achieves relatively small improvement. It is consistent with our motivation in Fig. 1, i.e. , the gathering effect concerns the position as well. Moreover, there is a training trick, i.e. , given a same query input, we synchronize its position in O-Decoder and C-Decoder through average. This operation brings +0.3 AP improvement on DAC-DETR.

Can removing self-attention benefit other one-to-many decoders? Table 3 (Variant2) shows removing self-attention is critical for C-Decoder. Table 6 further investigates this design on another popular one-to-many decoder, H-DETR [12] and shows non-trivial improvement (+0.6AP) in 12 training epochs, as well. It further confirms that removing the self-attention enhances the effect of gathering multiple queries towards each object on different one-to-many decoders.

5 Conclusion

This paper reveals in the DETR decoder, the cross-attention and self-attention layers have opposing impacts ( i.e. , gathering and dispersing) on the object queries. This insight motivates us to divide the cross-attention layers out from this opposition for better conquering. The corresponding method, DAC-DETR, improves both the quantity and quality of the queries that are gathered by each object. Experimental results show that DAC-DETR improves multiple popular DETR baselines ( e.g. , Deformable-DETR, DINO) and performs favorably against recent state-of-the-art methods.

Acknowledgements . This work is supported by the Fundamental Research Funds for the Central Universities (No. 226-2023-00048).

Broader Impacts . This paper utilizes the open dataset to evaluate the performance. Our method can improve the object detection, which could be applied to automatic driving systems. We will enhance the generalization of method.

References

Zhi Cai, Songtao Liu, Guodong Wang, Zheng Ge, Xiangyu Zhang, and Di Huang. Align-detr: Improving detr with simple iou-aware bce loss, 2023.

Xipeng Cao, Peng Yuan, Bailan Feng, and Kun Niu. Cf-detr: Coarse-to-fine transformers for end-to-end object detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 185-193, 2022.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.

Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid task cascade for instance segmentation, 2019.

Qiang Chen, Xiaokang Chen, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng, and Jingdong Wang. Group detr: Fast detr training with group-wise one-to-many assignment, 2022.

Xiyang Dai, Yinpeng Chen, Jianwei Yang, Pengchuan Zhang, Lu Yuan, and Lei Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2988-2997, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3621-3630, 2021.

Ziteng Gao, Limin Wang, Bing Han, and Sheng Guo. Adamixer: A fast-converging query-based object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5364-5373, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn, 2018.

Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. arXiv preprint arXiv:2207.13080 , 2022.

Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In Proceedings of the European conference on computer vision (ECCV) , pages 784-799, 2018.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13619-13627, 2022.

Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmiao, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-based visual segmentation: A survey. arXiv pre-print , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In International Conference on Learning Representations , 2022.

Shilong Liu, Tianhe Ren, Jiayu Chen, Zhaoyang Zeng, Hao Zhang, Feng Li, Hongyang Li, Jun Huang, Hang Su, Jun Zhu, and Lei Zhang. Detection transformer with stable matching, 2023.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2017.

Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2021.

Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In 18th international conference on pattern recognition (ICPR'06) , volume 3, pages 850-855. IEEE, 2006.

Duy-Kien Nguyen, Jihong Ju, Olaf Booij, Martin R Oswald, and Cees GM Snoek. Boxer: Box-attention for 2d and 3d transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4773-4782, 2022.

Jeffrey Ouyang-Zhang, Jang Hyun Cho, Xingyi Zhou, and Philipp Krähenbühl. Nms strikes back. arXiv preprint arXiv:2212.06137 , 2022.

Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger, 2016.

Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement, 2018.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks, 2016.

Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 658-666, 2019.

Byungseok Roh, JaeWoong Shin, Wuhyun Shin, and Saehoon Kim. Sparse detr: Efficient end-to-end object detection with learnable sparsity. arXiv preprint arXiv:2111.14330 , 2021.

Wenwei Zhang Size Wu Guangliang Cheng Yunhai Tong Chen Change Loy Shilin Xu, Xiangtai Li. Open-vocabulary object detection via dynamic self-training. arXiv pre-print , 2023.

Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris M Kitani. Rethinking transformer-based set prediction for object detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3611-3620, 2021.

Zongheng Tang, Yifan Sun, Si Liu, and Yi Yang. Detr with additional global aggregation for cross-domain weakly supervised object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11422-11432, 2023.

Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection, 2019.

Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9:2579-2605, 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 2567-2575, 2022.

Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. arXiv preprint arXiv:2310.01403 , 2023.

Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318 , 2021.

Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, and Shijian Lu. Accelerating detr convergence via semantic-aligned matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 949-958, 2022.

Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and HeungYeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022.

Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo, and Kai Chen. Dense distinct query for end-to-end object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7329-7338, June 2023.

Chuyang Zhao, Yifan Sun, Wenhao Wang, Qiang Chen, Errui Ding, Yi Yang, and Jingdong Wang. MS-DETR: Efficient DETR training with mixed supervision, 2023.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 2020.

Peer reviews:
comments: SUMMARY:
This paper introduces DAC-DETR, a method to improve the training efficacy of DEtection Transformers (DETR) by addressing the contrary impacts of cross-attention and self-attention layers in the DETR decoder. DAC-DETR divides the cross-attention layers into an auxiliary decoder, which focuses on learning the cross-attention, while the original decoder handles non-duplicate detection. By employing one-to-many label assignment in the auxiliary decoder, DAC-DETR effectively improves the gathering effect of queries, leading to improved detection accuracy compared to popular DETR baselines.

STRENGTHS:
- The paper is well-organized, presenting its ideas and findings in a clear and coherent manner. 
- The motivation is clear. The proposed methods are straightforward and logical, making them easy to understand and implement.
- The effectiveness of the proposed methods is well-supported by a comprehensive set of experiments. The experimental results demonstrate significant improvements over existing DETR models.

WEAKNESSES:
I have no severe concerns about the paper.

QUESTIONS:
The reviewer is curious about the performance of the one-to-many branch (C-decoder).

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-02

comments: SUMMARY:
This paper observes the problems in cross-attention and self-attention that impacts the queries, and proposes to use divide-and-conquer to improve the training accuracy.

STRENGTHS:
1. The paper is considered novel to me. The insights in analyzing the cross-attention and self-attention and the proposed design are interesting.
2. Strong performance. The final performance is improved compared with SOTA detectors.

WEAKNESSES:
1. The design seems a little complex. Not sure if this design can be plugged into other DETR-like model easily.

QUESTIONS:
No

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors find that the cross-attention and self-attention in the DETR decoder have opposite effects on object queries. This phenomenon reduces the training efficiency of DETR model. To resolve the contradiction, this paper proposes a Divide-And-Conquer DETR that employs an auxiliary decoder that shares parameters with the original decoder. Experimental results show that DAC-DETR achieves significant performance over popular DETRs.\n\nSTRENGTHS:\nThe paper adequately cites relevant work and clearly articulates the differences from existing work. The experiments show promising results. And other researchers may build a network structure based on this paper.\n\nWEAKNESSES:\n* *Originality**: This article is a combination of a series of existing methods and is not innovative enough.\n* *Quality*：The motivation is straightforward, but the authors should give more analysis about contradiction. DAC-DETR brings improvement over baseline, but this does not indicate that the problem has been solved by theoretical analysis or experimental results. \n* *Clarity*：This paper is not well organized and has some grammar errors.\n* *Significance*：The model architecture is not adequately simple, which will affect its usability. From the experimental results, the authors proposed DAC-DETR to solve the conflict problem of cross-attention and self-attention but did not fundamentally innovate the structure to solve the problem.\n\nQUESTIONS:\n1.In Figure 1 (b,c), which module is used to obtain the classification score and bounding box of the object? Why \"cls\" of objects in Figure 1(b) is so different from \"cls\" scores in Figure 1(b). Does the same phenomenon occur at the last layer of the DETR decoder and occur at the DINO?\n2. This paper proposes DAC-DETR to resolve the contradiction between cross-attention and self-attention. Whether this problem in Figure1 is solved by the corresponding method, the authors should give more visualization analysis like Figure1, used DAC or not.\n3. In Table 3. why the AP of varaiant2 and varaiant3 is reduced?\n\nLIMITATIONS:\n1. The description of the one-to-many label for assignment In section 3.2 is a bit too abstract. This is a key component of the proposed method, from Table 4, and the major difference with the existing method.\n2. The authors proposed DAC-DETR to solve the confliction problem of cross-attention and self-attention, but innovative structures are not proposed to fundamentally solve the problem.\n3. Compared with one-to-many Hungarian Loss in Table 4, the increase of AP is due to matching scores introducing an IoU score, which is also compatible with the original DETR.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple modification to the DETR architecture that improves upon several prior implementations. The paper identifies that the single decoder approach causes the model to try and achieve opposing objectives in terms of the query coverage and deduplication, and proposes a solution to the problem : an additional decoder having only cross attention layers is employed which improves performance on a standard benchmark (COCO).\n\nSTRENGTHS:\n## Originality and Significance \nThe findings about the self attention performing deduplication and cross attention gathering information from the image were already knows (as mentioned in this paper). The primary contribution here are\n* Identifying that these two effects actually are in competition with each other, causing the training speed and final performance to be affected. \n* Proposing a solution by adding an additional decoder having only cross attention. \n* Displaying that this approach can be combined with several DETR variants and improves performance while reducing training time in all those cases \n* Ablations showing that freezing weights in the O-decoder, adding self attention layers in C-decoder and removing the one-to-many matching all reduce performance, supporting the findings of the paper. \nAdding an additional cross-attention only decoder is a simple modification that can be implemented on top of most other DETR variants, making the finding useful and significant. Since the added layers share weights with the original decoder, the size of the model does not increase, and the additional decoder is not used during inference, ensuring that there is no latency in inference time due to the proposed modification. \n\n## Clarity and Quality\n* Clear figures with descriptive stand-alone captions\n* The findings are clearly explained with supporting evidence\n\nWEAKNESSES:\n* No analysis on if there is any difference in performance on large/small objects and common/rare objects\n* Some qualitative examples of difference between the proposed approach and the baselines would be helpful\n\nQUESTIONS:\n1. What happens if you try to use the predictions from the C-decoder during inference as well? For instance by combining the predictions from the two decoders? Does it further improve performance? \n2. Line 172 \"and the second one is to suppress the label imbalance regarding different objects.\" Could you explain more clearly how this is achieved?\n3. In Figure 3, is this average number of queries computed on the C-decoder or the O-decoder? \n\n\nSuggestions: \n1. Add an explanation of what t is in Figure 3 caption and change the y axis label to \"avg number of queries / object\"\n2. The scale of the figures in Figure 3 should be kept constant, otherwise at first glance it looks like the gap increases at higher threshold, which is not the case\n3. Writing suggestions: In several places, the phrase \"the contrary\" is used in a gramatically incorrect manner (divide the cross attention from the contrary / their contrary impairs ... etc) and I'd like to suggest that would be better substituted with the following phrasing: \n\n\"To improve the training efficacy, we propose a Divide-And-Conquer DETR (DAC-DETR) that divides the cross-attention out from this contrary for better conquering\" --> \"To improve the training efficacy, we propose a Divide-And-Conquer 6 DETR (DAC-DETR) that separates out the cross attention to avoid these competing objectives.\" \n\n\"have some contrary impacts on the object queries.\" -> \"Have some opposing effects on the object queries\"\n\n\"These two impacts are both critical for DETR (as explained later), and their contrary impairs the training efficacy.\" -> \"These two impacts are both critical for DETR (as explained later), and their contrasting effects impairs...\"\n\n\"In spite of the contrary, these two effects are both important for DETR.\" --> \"In spite of the conflicting /rival objectives, we find that these two effects ....\" \n\nAnd similarly for other occurrences of the word contrary. \n\nL169 grount-truth -> ground-thruth\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper reveals the “gather ↔ disperse” effects between cross-attention and self-attention layers in DETR decoder and proposes to add a decoder as the auxiliary branch without the self-attention blocks. The proposed approach achieves competitive detection performances.

STRENGTHS:
1. this paper leverages the opposite effects of self-attention and cross-attention to build a auxiliary decoder and improves the detection performances

WEAKNESSES:
1. how general does the gather ↔ disperse effect apply? is there more examples beyond Figure 1
2. it's unclear if this can be combined with other practices such as stable matching and memory fusion to further improve the performances

QUESTIONS:
1. Is there any similarity between “gather ↔ disperse” phenomenon vs. the evolution algorithm with “prune out ↔ grow back”?
2. can this be combined with other practices such as stable matching and memory fusion to further improve the performances?
3. is it possible to add the C-decoder with self-attention module only in Table 3 for the completeness?

LIMITATIONS:
adequate
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07