DAC-DETR: Divide the Attention Layers and Conquer
Authors: Zhengdong Hu
Yifan Sun
Jingdong Wang
Yi Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: deep learning
computer vision
object detection
transformer
TL;DR: This paper reveals a characteristic of DEtection Transformer (DETR) that negatively impacts its training efficacy, i.e., the cross-attention and self-attention layers in DETR decoder have contrary impacts on the object queries.

Abstract: This paper reveals a characteristic of DEtection Transformer (DETR) that negatively impacts its training efficacy, i.e., the cross-attention and self-attention layers in DETR decoder have contrary impacts on the object queries (though both impacts are important). Specifically, we observe the cross-attention tends to gather multiple queries around the same object, while the self-attention disperses these queries far away. To improve the training efficacy, we propose a Divide-And-Conquer DETR (DAC-DETR) that divides the cross-attention out from this contrary for better conquering. During training, DAC-DETR employs an auxiliary decoder that focuses on learning the cross-attention layers. The auxiliary decoder, while sharing all the other parameters, has NO self-attention layers and employs one-to-many label assignment to improve the gathering effect. Experiments show that DAC-DETR brings remarkable improvement over popular DETRs. For example, under the 12 epochs training scheme on MS-COCO, DAC-DETR improves Deformable DETR (ResNet-50) by +3.4 AP and achieves 50.9 (ResNet-50) / 58.1 AP (Swin-Large) based on some popular methods (i.e., DINO and an IoU-related loss). Our code will be made available at https://github.com/huzhengdongcs/DAC-DETR.

DAC-DETR: Divide the Attention Layers and Conquer
Zhengdong Hu Yifan Sun', Jingdong Wang', Yi Yang'
ReLER, AAII, University of Technology Sydney
Baidu Inc.
3 CCAL, College of Computer Science and Technology, Zhejiang University huzhengdonggcse gma.sddonnngeemmmmilommmo com, sunyf org.g.uuu.rr...o......... org. cn, ytt..aaawa..aco...o..o..c..eeeee
com yangyicsézju edu..ceddcc.
Abstract
This paper reveals a characteristic of DEtection Transformer (DETR) that nega- tively impacts its training efficacy, i.e., the rross-attention and self-attention layers in DETR decoder have opposing impacts on the object queries (though both impacts important). Specifically,
observe the cross-attention tends gather multiple queries around the same object, while the self-attention disperses these queries far DETR
away. To improve the training efficacy, we propose Divide-And-Conquee (DAC-DETR) that separates out the cross-attention to avoid these competing objec- lives. During training, DAC-DETR employs auxiliary decoder that focuses on learning the cross- aatentint layers. The auxiliary decoder, while sharing all the other parameters, has NO self attentttnnoo layers and employs one-to to-many label assign- ment to improve the gathering effect. Experiments show that DAC-DETR brings remarkable improvement over popular DETRs. For example, under the epochs training scheme on MS-COCO, DAC-DETR improves Deformable DETR (ResNet- 50) by +3.4AP and achieves 50.9 (ResNet-50) 58.1 AP (Swin-Large based on some popular methods (i.e., DINO and an IoU-related loss). Our code will be made available at ht tps: /ttp//ggttt////t cww/cozw/eeeeeeeeccceeeeeco hwtp////henddnngss////ooo DDACs/eeS/AAADDDAA
Introduction
Though DEtection Transformers (DETRs) have already achieved great progress, their low training efficacy problem remains critical challenge. The research community has been paying great efforts to locate the causes of this problem and to improve the training efficacy. For example, DN-DETR [5] and DINO 42 find the one-to-one matching is prone to inconsistent optimization goals. In response, they propose to accelerate DETR training with auxiliary denoising queries. Hybrid DETR [12 and Group DETR attribute the reason to the rare positive training queries and correspondingly increase positive queries through additional one-to-many matching. Some other works [ notice the misalignment between the predicted probability and position. They suppress such misalignment through loU-teltted loss functions and achieve improvement.
This paper, from another perspective, investigates this problem by rethinking the cooperation between cross cross-attention and self-attention layers in the DETR decoder. We find during their cooperation, these two attention types actually have some opposing impacts on the object queries. These two impacts are both critical for DETR (as explained later), but their opposition impairs the training efficacy. Specifically, we observe the cross-attention layers in DETR decoder tends to gather multiple queries around single object, while the self-attention disperses these queries far away from each other. The above "gather ++ disperse" phenomenon concerns not only the positional relationship (Ist row in Fig. but also the feature distance (2nd row in Fig. visualized by t-SNE (6)):
Zhengdong Hu makes part work during intersship Baidu Inc
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

92
Bp Jeke A7
C2
G 1000
query position
Qctck: 361
4: (chv3.001
44 (ct:0.09)
query distribution (1SNE)
43
92
4:
q8
44
q+
(a)
before decoder
(b) decoder w/o self self-atttention
(c) decoder w/ self-attention
Figure 1: Comparison between (a) initial queries before the DETR decoder, (b) queries output from a decoder variant without self-attention layers and (c) queries output from the normal DETR decoder. We visualize their position (Ist row) and feature distribution (2nd row). For clarity, we randomly choose only queries with relatively large IoU. From (a) + (b), multiple queries are pulled towards a same object, showing the gathering effect of cross-attention layers. From (b) + (c), adding the self-attention layers disperses these queries except single one (91), which approaches even closer towards their center. In spite of the conflicting objectives, we find that these two effects are both important for DETR.
Cross-atlention layers tend to gather multiple queries around the same object. Given an already- trained Deformable DETR (45. we remove all the self-attention layers in its decoder. This removal compromises DETR to duplicate detections. As illustrated in Fig. (b), multiple queries locate same object ("bear") with relatively large Intersection-over Union (IoU), Compared with their initial states (before the decoder) in Fig. | (a), they become closer towards each other, regarding both their position (1st row) and feature distance (2nd row).
Self-attention layers disperse these queries from each other. In Fig. (c). we restore the original deformable DETR. Correspondingly, the queries are dispersed from each other regarding both position (1st row) and feature distance (2nd row). Due to this dispersion, most queries become farther to their original center point, except that single query (with the highest classification score) approaches even closer. This phenomenon explains how DETR makes non-duplicate detection, and is consistent with the consensus (23 41 25), i.e., self-attention layers play critical role removing duplicates.
In spite of their opposition, these two effects are both critical for DETR. The gathering effect allows the queries to aggregate feature from the nearby object. The dispersing effect allows the self-attention layers to remove duplicate detection. However, learning the stacked self-attention and cross -attentnton layers within such "gather ++ disperse" opposition is difficult. To improve the training efficacy, propose a Divide-and-CCnqqer DETR (DAC-DETR) that divides the cross-attention out from this opposition for better conquering. We explain the key points of DAC-DETR in details as below:
1) "Divide": DAC-DETR employs an auxiliary decoder that focuses on learning the cross-attention layers. The auxiliary decoder has NO self-attention layers and shares all the other parameters (i.e., the cross-attention layers and the feed-for fordfor networks) with the original decoder (O-Decoder). We name the auxiliary decoder C-Decoder to highlight its focus the cross-attention.
2) "Conquer": During training. DAC-DETR feeds all the queries into two decoders in parallel. O-Decoder uses the one-to-one matching to learn non-duplicate detection. In contrast, C-Decoder uses the one-to-many matching that assigns each object with multiple positive queries. C-Decoder has no self-attention layers and is prone to duplicate detection results. Therefore, using one-to-many label assignment natural choice and benefits learning the gathering effect with more positive queries. During inference, DAC-DETR discards C-Decoder and maintains the baseline efficiency.

---

Through the above "divide-and" "divide-and-conquee ann-conquer", DAC-DETR defuses the opposition when learning the "gathering" and "dispersing" effects which are both important for DETR. Mechanism analysis in Section 3.3 shows that due to the enhanced feature aggregation ability, DAC-DETR improves both the quantity and quality of intermediate queries that are gathered to each ground-truth object. Based on these improved queries, DETR increases the detection accuracy (after removing duplicate predictions).
We conduct extensive experiments to validate the effectiveness of DAC-DETR and empirically show remarkable improvement over various DETRs. For example, based on a popular baseline, i.. ResNet-50 Deformable DETR [45,, DAC-DETR brings +3.4 AP improvement and achieves 47.1 AP on MS-COCO within 12 (1x) training epochs. On some more recent sfae-of-thhe-al methods (that usually integrate a battery of good practices), DAC-DETR still gains consistent and complementary benefits. For example, under the 1x learning scheme on MS- ASSCOOOO, DAC-DETR improves ResNet- 50 DINO to 50.0 (+1.0) AP. Combining an loU-related Loss in Align DETR [1, our method further achieves 50.9 AP (ResNet-50) / 58.1 AP (Swin-Large), surpassing Align DETR itself by +0.7 AP. Moreover, the achieved results (50.9 and 58.1 AP) are also higher than Stable DETR (0 (50.4 and 57.7 AP), which adopts even more good practices (i.e., stable matching and memory fusion).
Our main contributions are summarized follows: First, we reveal the "gather ++ disperse" opposi- tion between cross-attention and self-attention layers in DETR decoder as a reason for its low training efficacy. Second, we propose to divide the cross-attention layers out from this opposition for better conquering the DETR training, yielding the so-called Wwide-and-Conquee DETR (DAC-DETR). Third, we empirically show that DAC-DETR brings remarkable improvement over popular DETR baselines and performs favorably against most recent state-of-the-ar methods.
2 Related works
DETR-like object detectors. The original Detection Transformer (DETR) 37 3 is featured for its end-to-end detection without any hand-crafted modules (e.g., Non-max Suppression). Since
DETR 3 suffers from low training convergence, many efforts 45 33
23
42
38
6
S
34
43
39
proposes to replace traditional attention modules with deformable attention and improves the training convergence. Conditional DETR [23] performs the conditional spatial query for decoder multi-head cross-attention.
32 16 have been paid to improve the training efficacy. Deformable DETR
45
DAB-DETR 199 utilizes box coordinates as priors and dynamically updates anchor boxes layer- by-layer. DN-DETR [5 introduces denoising part to stabilize the bipartial matching, and DINO
further adds contrastive learning enhance the performance. Some works (20 D propose IoU-related loss functions to align the predicted probability score against position. This paper, from another perspective, finds a opposing impact between the cross-attention and self-attention layers and proposes to divide the attention layers during training.
42
Attention layenss layers in DETR decoder. The cross- aeteenttnt and self-attention layers are both essential for DETRs. The former allows the object queries to aggregate information from the image features, while the latter facilitates query-to-query interaction to remove duplicate detection 23 45 3 40 31].
Our observation in this paper is consistent with the above consensus and goes a step forward: the cross-attention layers tend to gather multiple queries towards each object, while the self-attention disperses them far away. By separating the cross-attention layers out from this opposition, the proposed DAC-DETR improves DETR training efficacy.
One-to-many label assignment. One-to-many label assignment is widely utilized in -bseb 335
detectors 22 27
these methods produce duplicate predictions on each object and uses hand-crafted non- mon-maximum suppression (NMS) 24 for post-proced post-processing. As the DETRs prevail, some recent literature 5
28
[1
[
12
explore one-to-many label assignment for DETRs and demonstrate employs multiple parallel object queries, which jointly contribute multiple
26
benefit. Group DETR positive queries for each object. Hybrid DETR [12 adds an auxiliary query branch that performs one-to-many label assignment.
DAC-DETR uses the on-to-mmany label assignment for the auxiliary C-Decoder, because C-Decoder has no self-attention layers that help removing duplicate predictions. Therefore, DAC-DETR also gains the benefit from one-to-many assignment. That being said, the benefit from one-to-many assignment not our key contribution but a natural gain. While a co-current work Hybrid DETR [12 already shows that adding a one-to-many decoder improves DETR, our method shows that removing
3

---

Encoder outputs
One-to-one matching
One-to-mane matching
share
heads
heads
Transformer Encoder
O-Decoder
C-Deodeer
& Norm &
share
Add
split
Cross-Attention
Cross-Attention
Backbone
Add & Norm
NO Self Attention
Self- Leff-Attention
LX
Lx
Input image
object queries
Figure 2: Overview of DAC-DETR. Left: for the backbone and encoder, DAC-DETR follows the standard DETR pipeline: it uses backbone (e.g., ResNet-50) to extract feature, splits the feature into patch tokens and then feeds the tokens into the transformer encoder. The tokens output from the encoder fed into the decoders for object detection. Right: the main difference between DAC-DETR and previous DETRs is that DAC-DETR has two parallel decoders, i.e., an O-Decoder (standard decoder) and an auxiliary C-Decoder (NO self-attention layers). These decoders share most parameters and the same object queries, except that C-Decoder has no self-attention layers. O-Decoder is supervised with one-to-one matching learn non- non-duplicate detection. In contrast, Decoder uses one-to-man matching to assign each ground ground-truth object with multiple positive queries. During inference, DAC-DETR only uses the O-Decoder and maintains the baseline efficiency.
the self-attention in the one-to-many decoder is better (e.g., +0.6 AP MS-COCO in 12 training epochs, as detailed in Table 6.).
3 Methods
3.1 Overview
Preliminaries on DETRs. A DETR-style detector typically consists of backbone (e.g., ResNet [10], Swin Transformer (2I)), a transformer encoder and a transformer decoder. Given an input image, the DETR-style detector first feeds into the backbone to extract image features, splits the features into patch tokens and then enhances the patch tokens by the transformer encoder. The enhanced tokens from the encoder are Z {2,,..., }m The transformer decoder takes multiple object queries Q = ........f 4n as its input. The decoder has L stacked decoder blocks. In each decoder block, the object queries Q sequentially undergo self-attention layer, a cross-attention layer, and feed-forward network (FFN). Specfically, the cross- pross-attention layer makes Q interact with the patch tokens Z, so as to aggregate features for locating and identifying the objects. Afterwards, the FFN transforms Q into the output embeddings, which make final prediction through classification and regression head.
DAC-DETR follows the standard DETR pipeline for the backbone and transformer encoder and featured for the "Divide-and aeieeand--oone and-Conquer" decoders. As illustrated in Fig. [ DAC-DETR has two decoders, i.e., 1) an O-Decoder which is the same as the baseline DETR decoder, and 2) a C-Decoder that divides the cross-attention layers from the "gather <> disperse" for better learning. C-Decoder removes all the self-attention layers and shares all the other decoder parameters (cross-attention, FFN), object queries and the prediction head with the O-Decoder. Section 3.2 elaborates on these two decoders regarding their architecture, training and inference. Section 3.3 looks into the mechanism,

---

i.e. how DAC-DETR impacts the learning of the DETR decoder. It shows that DAC-DETR improves both the quantity (number) and the quality (larger IoU) of queries that attend to each object.
3.2 Divide-and-Conquee Decoders
DAC-DETR constructs two parallel decoders, i.e., a canonical DETR decoder (O-Decoder) and decoder without self-attention layers (C-Decoder), as shown in Fig. 2 The initial object queries Q are fed into the first block of O-Decoder and C-Decoder in parallel. Since the object queries undergo different transformations in these two decoders, they are denoted as Q (queries O-Decoder) and Q (queries in C-Decoder), where the superscript [(1 0, 0... .5) indicates the index of the decoder block. We have Q Q = =Q.
O-Decoder: canonical DETR-decoder revisit. O-Decoder is canonical DETR decoder with each block consisting of self-attention layer, a cross-attention layer and a FFN. Specifically, the I-th decoder block first updates all the queries through self-attention, which is formulated
= Q + stt (Que (Q)), Key(Q)) Val (Q)),,
where Que, Key, Val are the projections to derive the query, key and value embedding from tokens Q respectively. We omit the description of "Layer norm" and "Dropout" in all equations for better clarity.
Given the queries Q% output from the preceding self-attention layer, the cross-attention layer further updates them through interaction with the feature embedding Z, which formulated as:
Q ++ FFN( O + AAttoross (Que (b), Key
Key (Z), Val(Z))),
(2)
where Que, Key, Val are the projections and are parameterized differently from those in the self-attention layer (Eqn. D though sharing the same symbols.
Finally, the output embeddings Q++1 are fed into a predictor head to make label and position predictions. The predictions supervised with the one-to-one matching and Lo-dezoders which is the standard Hungarian loss 3 as in the original DETR.
C-decoder removes all the self-attention layer in each decoder block and shares all the other parameters with O-decoder. Therefore, C-Decoder, the queries undergo only cross-attention with the feature embeddings Z and following FFN transformation, which is formulated as:
+1 FFN At tettara (Que Key Val()))
where all the model parameters are shared with the O-Decoder (Eqn.2)
In parallel to the prediction from O-Decoder, the output embeddings from C-Decoder Q'+1 are fed into the shared predictor head. The corresponding label and position prediction from query y Q are (P(g)= b(g))
Different from the bipartite matching strategy in O-Decoder, C-Decoder employs one-to-many label assignment for supervision (26) 44), i.e., assigning each ground-truth object with multiple queries. Specifically, given an object with ground- annotation y [é. 8] (è and b are the class and bounding box, respectively), we measure its matching scores with the prediction from all the queries by:
M = (P(s)) (c) + IOU < b(o) >
VOg E Q
where P(z) (C) denotes the predicted label score class é, <, denotes the IoU operation between predicted box and ground truth 6.
Given the matching scores M for the ground- truth object, we select multiple positive predictions for it by two criteria: 1) the matching score should be larger than a threshold and 2) the matching score should be among the top-k scores in M. There two criteria are required simultaneously and the second one to suppress the label imbalance regarding different objects. All the other predictions are assigned with negative labels for this object. With the assigned labels, supervise the predictions from C-Decoder with Le-denoder, which is standard detection loss function consisting of classification [18] and regression loss [30].
5

---

0.6
t=8
20
10
DETR
DETR
DAC-DETR
15
Deformabi--DFR
Deformable DETR
2
Deformable DETR
nums
nums
1.0
12
0
3
4
8
10
11
12
2
3
10 11
12
2
3
6 epochs
9
10
12
epochs
epochs
Figure 3: The averaged number of queries that each ground-truth object gathers on the validation set of MS-COCO. Compared with the Deforamble-Der each object, and 2) improves the quality of the closest queries. Results for more layers are provided in the supplementary. y-axis denotes "avg number of queries / object".
baseline, DAC-DETR 1) has more queries for
During training, Lg decodee and Le decede: are summed up. During inference, DAC-DETR discards C-Decoder and maintains the same pipeline and efficiency the baseline.
Discussion. Using the many label assignment for supervising C-Decoder is natural choice. It is because C-Decoder has self-attention layers (which are critical for removing duplicate detections) and is thus prone to multiple detections for each single object. C-Decoder employs the one-to-one label assignment, DAC-DETR will bring improvement but actually compromises the baseline accuracy (Section 4.3)
3.3 Mechanism Analysis
We investigate how the proposed divide-and and-cuur method benefits DETR with focus on the gathering effects of the cross-attention layers. Specifically, we are interested in: how many queries can each ground-truth object gather in DAC-DETR and in the baseline? ? this end, we count the averaged number of queries that have large affinity (according to the matching score in Eqn. 4 with each object. The queries with high matching scores are important: they aggregate abundant information from corresponding object and thus serve as the proposals which are potential to predict the object. Based on these redundant queries and gives the final non-duplicate prediction.
queries (proposals), the self-attention layer removes
The comparison between DAC-DETR and the Deformable-DETT baseline in Fig. 3 We list the results for the last decoder layer and leave the results for more layers to the supplementary. From Fig. we draw two important observations below:
Remark-1: DAC-DETR increases the number of queries gathered for each object. Comparing DAC-DETR and its baseline under various matching score threshold (i.e., 0.6, 0.7, 0.8), observe that DAC-DETR consistently gathers more queries for each single object. Please note that the total number of queries is the same for DAC-DETR and the baseline. We thus infer that DAC-DETR makes higher utilization of the object queries.
Remark-2: DAC-DETR improves the quality of the best queries for each object. Focusing on the comparison under large threshold (e.g., t = 0.8), we find the baseline has rare high-quality queries, while DAC-DETR has about high-quality query for each object. We recall that the matching score composed of loU and classification score. This suggests that some queries in DAC-DETR either has good IoU or high confidence the ground-truth object.
Combining the above two remarks, we conclude that DAC-DETR improves both the quantity and quality of the queries for each object. This improvement is the direct reason how DAC-DETR increases the detection accuracy (Section[4)
4 Experiments
4.1 Setup
Dataset and backbone. We evaluate the proposed DAC-DETR on COCO 2017 [17 detection dataset. Following the common practices, we evaluate the performance validation dataset(Sk images)

---

Method
Backbone
epochs
AP
AP50
AP
APs
APM
APr.
Basel#i (Delormable)
R50
12
43.7
62.0
47.3
26.2
46.8
58.1
Basel#1 (Deformable)
46.8
66.0
50.6
29.4
61.4
H-DETR 12
45.9
DN-Deformable-EETe
46.0
63.8
27.7
62.3
DAC-DETR (ours)
R50
47.1
3.4)
64.8
51.1
29.2
50.6
62.4
DAC-DETR (ours)
R50
48.5
(+ 1.7)"
66.5
52.6
31.0
51.8
63.1
Basel#2 (Deformable++)
12
R50
12
47.6
65.9
52.3
30.2
51.1
62.6
Basel#2 (Deformable++)
12
R50
36
49.3
67.5
53.6
32.1
52.5
64,4
H-DETR 12
R50
12
48.7
66.4
52.9
31.2
51.5
63.5
H-DETR 12
R50
36
50.0
68.3
54.4
32.9
52.7
65.3
DAC-DETR (ours)
R50
12
49.3 (+1.7)
66.5
53.8
31.4
52.4
64.1
DAC-DETR (ours)
R50
24
50.5 (+ (+1
67.9
55.2
33.2
53.5
64.8
Basel#3 (DINO) 442
R50
49.0
66.6
53.5
32.0
52.3
63.0
Basel#3 (DINO)
R50
50.4
68.3
54.8
33.3
64.8
Group-DETR
49.8
32.4
53.0
64.2
Align-DETR 1
50.2
67.8
54.4
32.9
53.3
65.0
Align-DETR 1
51.3
68.2
56.1
35.5
55.1
65.6
Kuble-DINO-4scane
20
R50
12
50.4
67.4
55.0
32.9
54.0
65.5
SubbeeDDNO--Acalee 20
R50
24
51.5
68.5
56.3
35.2
54.7
66.5
DAC-DETR (ours)
R50
12
(+110)
67.6
54.7
32.9
53.1
64.2
DAC-DETR (ours)
R50
24
51.2 (+0.8)
68.9
56.0
34.0
54.6
65.4
DAC-DETR Align (ours)
R50
12
50.9 (+1.9)
68.3
55.3
33.8
54.1
65.7
DAC-DETR Align (ours)
R50
24
52.1 (+1.7)
69.7
56.5
34.8
55.2
67.3
Table Evaluation COCO val2017 with ResNet-50 backbone For fair comparison, the methods are divided into three groups according to their baseline i.e., Basel#1: Eiformable-DETT [45].
Bascl#2: Eeformable-DETR-4. [1], Basel#3: DINO 42 *; We compare performance between
24-epoch DAC-DETR and 36-epoch baseline.
Method
Backbone
epochs
AP
APso
AP75
APg
AP
M
AP
H-DETR
12
Swin-I.
75.2
61.3
39.3
60.4
72.4
H-DETR 12
Swin-L.
36
57.6
76.5
63.2
41.4
61.7
73.9
Bascl#3 (DINO) 42
Swin-L
12
56.8
75.6
62.0
40.0
60.5
73.2
Basel#3 (DINO) 42
Swin-L
36
58.0
77.1
66.3
41.3
62.1
73.6
Group-DETR
Swin-L
58.4
41.0
62.5
73.9
StableDDINO-4scale
20
Swin-L
57.7
75.7
63.4
39.8
62.0
74.7
Stable-DINO-4scale 20
Swin-L
58.6
76.7
64.1
41.8
63.0
74.7
DAC-DETR (ours)
Swin-L
12
57.3 (+0.5)
75.7
62.7
40.1
61.5
74.4
DAC-DETR Align (ours)
Swin-L.
12
58.1 (+1.3)
76.5
63.3
40.9
62.4
75.0
DAC-DETR Align (ours)
Swin-L.
24
59.2 (+1.2)
77.1
64.5
43.1
63.4
76.0
Table 2: Evaluation on COCO va12017 with Swin-Transformer Large backbone, Most competing
methods use the same baseline, i.e., DINO 42
#; We compare performance between 24-epoch
DAC-DETR and 36-epoch baseline.
by using standard average precision (AP) result under different IoU thresholds. We implement DAC-DETR with two popular backbones, i.e., ResNet50 [10] (pretrained on ImageNet-1 1k (7) and
Swin-Large (2 (pretrained on ImageNet-22k 77).
Baseline and implementation details. The baselines for recent state-of-the-art methods vary a lot. For fair comparison with these methods, we adopt three recent popular baselines, e.g., . the original Deformable- DETR [45), an improved Deformable DETR (Deformable+t) [12] and the DINO 42
baseline. As for training, we use AdamW [22] [4] optimizer with weight decay of 10 We report experimental results under the 1x x (12 epochs) and 2x (24 epochs) scheme. We find the 2x learning scheme already allows DAC-DETR to achieve superior accuracy, compared with most competing methods (some of which uses longer training epochs).
4.2 Main Results
We evaluate DAC-DETR on COCO 2017 detection validation dataset. The results on ResNet50 [T) and Tran-Trransfrmmet Large backbones [2 are summarized in Table and Table 2 respectively.
7

---

Method
Backbone
epochs
AP
AP30
Baseline (Deformable-DET1) [45])
R50
12
43.7
63.0
Variant (freeze cross-atten layers in O-Decoder)
R50
12
45.9
63.3
Variant2 (add self-atten layers in C-Decoder)
R50
12
43.1
58.9
Variant3 (one-to-one matching in C-Decoder)
R50
12
43.5
62.1
DAC-DETR (ours)
R50
12
47.1
64.8
Table 3: Comparison of four variants for DAC-DETR.
Method
Backbone
epochs
AP
AP50
AP7s
Basel#I (Deformable) [45]
R50
12
43.7
63.0
47.6
DAC-DETR w/ one-to-many Hungarian
R50
12
45.5
64.0
49.3
DAC-DETR (ours)
R50
12
47.1
64.8
51.1
Basel#2 (Defornable+++) [22
R50
12
47.0
65.2
51.5
DAC-DETR w/ one-to-many Hungarian
R50
12
48.4
65.6
52.8
DAC-DETR (ours)
R50
12
49.3
66.5
53.8
Basellt3 (DINO) 44
R50
12
49.0
66.6
53.5
DAC-DETR w/ one-to-many Hungarian
R50
12
49.3
66.7
53.8
DAC-DETR (ours)
R50
12
50.0
67.6
54.7
Table 4: Comparison of different one-to-many matching strategies for DAC-DETR. Hungarian-based onc-to-many matching is useful but is inferior to our threshold-buut basedddddaadddd strategy.
Based on the ResNet50 backbone, we employ three different baselines in Table [ with ResNet50 backbones) from which we draw two observations: DAC-DETR achieves consistent and remark- able improvements over all three baselines. For example, under 1 x learning scheme. DAC-DETR improves Deformable-DEr -EER 45), Deformable-DETRe
[12] and DINO 442 by +3.4, +2.3 and +1.0 AP, respectively. 2), DAC-DETR achieves competitive detection accuracy. We note that recent state-of-the-art methods usually adopt a strong baseline (e.g, DINO) and further integrate a battery of good practices. Among these methods, Stable-DINO is the strongest and combines an loU-related loss, an improved matching strategy and novel feature fusion. On the same baseline (DINO), method (DAC-DETR Align) while employing fewer tricks (only Align Loss), surpasses Stable-DINO by + 0.5 AP (12 epochs) and +0.6 AP (24 epochs).
On the Swin-L backbone (where relatively fewer methods have reported results), we only adopt the strongest baseline (DINO) and compare DAC-DETR against most recent state-of- sf-te-of-the-art methods. The results in Table 2 further confirms the effectiveness of DAC-DETR: it improves DINO baseline by +0.5 AP under 12 epochs. After combining an IoU-related loss (Align [I] loss), DAC-DETR surpasses the strongest competitor Stable-DINO by + 0.4 AP (12 epochs) and +0.6 AP (24 epochs).
4.3 Ablation Studies
Three variants of DAC-DETR. In Table we are interested in two questions: 1) Since the cross- attention layers already undergo specific training in C-Decoder, can they be frozen in O-Decoder (Variant-1)? 2) Are removing self-attention layers and using one-to-many matching important for C-Decoder? In response, we implement Variant-2&:3, which adds the self-attention layers and performs one-to-one matching for C-Decoder, respectively. From Table 3 observations:
we draw the two following
First, freezing the cross-attention layers in O-Decoder considerably compromises DAC-DETR (-1.2 AP), but still improves the baseline (+2.2). It shows that training the cross-attention layers only in C-Decoder already suffices competitive DETR detector and brings major improvement. We thus infer that C- C-Decoder makes the major contribution to learning the cross-attention layers. On the O-Decoder side, pays relatively fewer efforts onto cross-attention layers and channels efforts onto learning the self-attention layers. Second, adding self-attention and using one-to-one matching for C-Decoder both deteriorate DAC-DETR, These two variants are even inferior the baseline by -0.6 and -0.2 AP respectively. We thus infer that removing the self-attention layers and using

---

Method
Backbone
epochs
AP
AP50
Deformable-DEIR 145
R50
12
43.7
63.0
DAC-DETR w/ regression loss C-decoder
R50
12
45.7
64.0
DAC-DETR w/o syne position
R50
12
46.8
64.3
DAC-DETR (full)
R50
12
47.1
64.8
Table 5: Investigation some details for learning the query position.
Method
Backbone
queries
AP
Deformable-DEn DETR 45
R50
300
43.7
H-DETR (our implementation)
R50
300 + 1500
46.4
H-DETR w/o self-atted in mn-tto-many decoder
R50
300 + 1500
47.0
DAC-DETR
R50
300
47.1
Table 6: Removing the self-attention the one-to-many decoder improves H-DETR, as well.
one-to-many matching are critical. Being featured for no self-attention layers, C-Decoder has capability to remove duplicate predictions and thus favors one-to-many matching.
Comparison of different one-to-many matching strategies. In DAC-DETR, C-Decoder adopts threshold-bedt based one-to- one-to-many matching strategy. We investigate another alternative, i.e., Hungarian in Table 4 and draw two observations:
matching with repeated ground-trun in H-DETR 12
First, comparing "DAC-DETR w/ one-to-many Hungarian" against the baseline, we observe consistent improvement. It shows that C-Decoder is compatible to flungarian-basee one-to-many matching, as well. Second, comparing two mne-oo-many strategies against each other, we observe our threshold- based strategy is better. This is reasonable because Hungarian-basee based one-to-many matching is based on set-to-set comparison and requires global comparison among the samples. C-Decoder has no self-attention layers for global comparison and thus favors the threshold-based strategy.
Some details for learning the query position. Table investigatess some training details w.r.t. the query position. It is observed that removing the regression loss for C-Decoder achieves relatively small improvement, is consistent with our motivation Fig. D i.e. the gathering effect concerns the position as well. Moreover, there is training trick, iee, given a same query input, we synchronize its position O-Decoder and C-Decoder through average. This operation brings +0.3 AP improvement on DAC-DETR.
Can removing self-attention benefit other one-to-many decoders? Table 3 (Variant2) shows removing self-attention is critical for C-Decoder. Table/6 further investigates this design on another popular one-to-many decoder, H-DETR [12] and shows non-trivial improvement (+0.6AP) in 12 training epochs, as well. It further confirms that removing the self-attention enhances the effect of gathering multiple queries towards each object on different one-to-many decoders.
Conclusion
This paper reveals in the DETR decoder, the cross-attention and self-attention layers have opposing impacts (i.e., gathering and dispersing) on the object queries. This insight motivates us to divide the cross-attention layers out from this opposition for better conquering. The corresponding method, DAC-DETR, improves both the quantity and quality the queries that are gathered by each ob- ject. Experimental results show that DAC-DETR improves multiple popular DETR baselines (e.g., Deformable- DETR, DINO) and performs favorably against recent state-of-the. art methods.
Acknowledgements. This work is supported by the Fundamental Research Funds for the Central Universities (No. 226-2023-000483
Broader Impacts. This paper utilizes the open dataset to evaluate the performance. Our method can improve the object detection, which could be applied to automatic driving systems. We will enhance the generalization of method.

---

References
[1] Zhi Cai, Songtao Liu, Guodong Wang, Zheng Ge, Xiangyu Zhang, and Di Huang. Align-detr: Improving detr with simple iou-aware bee loss, 2023.
[2] Xipeng Cao, Peng Yuan, Bailan Feng, and Kun Niu. Cf-detr: Coarse-to-fine transformers for end-to-end object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 185-193, 2022.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.
[4] Kai Chen, Jiangmiao Pang. Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid task cascade for instance segmentation. 2019.
[5] Qiang Chen, Xiaokang Chen, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng, and Jingdong Wang. Group detr: Fast detr training with group-wise one-to to-tto-mmayy assignment, 2022.
[6] Xiyang Dai, Yinpeng Chen, Jianwei Yang. Pengchuan Zhang. Lu Yuan, and Lei Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2988-2997, 2021.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[8] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3621-3630, 2021.
[9] Ziteng Gao, Limin Wang, Bing Han, and Sheng Guo. Adamixer: A fast-converging query-based object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5364-5373, 2022.
[10] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[11] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn, 2018.
[12] Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. arXiv preprint arXiv:2207. 2022.
[13] Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localiza- tion confidence for accurate object detection. In Proceedings of the European conference on computer vision (ECCV), pages 784-799, 2018
[14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412. 2014.
[15] Feng Li. Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni. and Lei Zhang. Du-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13619-13627, 2022.
[16] Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmiao, Kai Chen, Ziwei Liu, and Chen Change Loy. mansformer-basee visual segmentation: A survey, arXiv pre-print, 2023.
[17] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
10

---

[18] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages 2988, 2017.
[19] Shilong Liu, Feng Li, Hao Zhang. Xiao Yang. Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In International Conference on Learning Representations, 2022.
[20] Shilong Liu, Tianhe Ren, Jiayu Chen, Zhaoyang Zeng, Hao Zhang, Feng Li, Hongyang Li, Jun Huang, Hang Su, Jun Zhu, and Lei Zhang. Detection transformer with stable matching, 2023.
[21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[22] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2017.
[23] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Hougiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
[24] Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In 18th in- ternational conference on paltern recognition (ICPR'O0 066) volume 3, pages 850-855. IEEE, 2006.
[25] Duy-Kien Nguyen, Jihong Ju, Olaf Booij, Martin R Oswald, and Cees GM Snoek. Boxer: Box-attention for 2d and 3d transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4773-4782, 2022.
[26] Jeffrey Ouyang Dhanng Jang Hyun Cho, Xingyi Zhou, and Philipp Krähenbühl. Nms strikes back. arXiv preprint wwXiv:2212.061:: 2022.
[27] Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger, 2016.
[28] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement, 2018.
[29] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks, 2016.
[30] Hamid Rezatofighi, Nathan Tsoi, Jun Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658-666, 2019.
[31] Byungseok Roh, JaeWoong Shin, Wuhyun Shin, and Sachoon Kim. Sparse detr: Efficient end-to-end object detection with learnable sparsity. arXiv preprint arXiv:2111. 2021.
[32] Wenwei Zhang Size Wu Guangliang Cheng Yunhai Tong Chen Change Loy Shilin Xu, Xiang- tai Li. Open-vocabulary object detection via dynamic self-training. arXiv pre-print, 2023.
[33] Zhiqing Sun. Shengcao Cao, Yiming Yang, and Kris M Kitani. Rethinking transformer based set prediction for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 361 1-3620, 2021.
[34] Zongheng Tang, Yifan Sun, Si Liu, and Yi Yang. Detr with additional global aggregation for cross-domain weakly supervised object detection. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 11422-11432, 2023.
[35] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection, 2019.
[36] Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008.

---

[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[38] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer- based detector. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2567-2575, 2022.
[39] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. arXiv preprint arXiv:2310.01403, 2023.
[40] Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104001118 2021.
[41] Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, and Shijian Lu. Accelerating detr convergence semantic-aligned matching. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 949-958, 2022.
[42] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022.
[43] Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo, and Kai Chen. Dense distinct query for end-to-end object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7329-7338, June 2023.
[44] Chuyang Zhao, Yifan Sun, Wenhao Wang, Qiang Chen, Errui Ding, Yi Yang, and Jingdong Wang. MS-DETR: Efficient DETR training with mixed supervision, 2023.
[45] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to en- object detection. arXiv preprint arXiv:2010.04159, 2020.
12


Reviews:
[{"review_id": "AoEzSWODzc", "reviewer": "Reviewer_9psD", "summary": "This paper reveals the “gather ↔ disperse” effects between cross-attention and self-attention layers in DETR decoder and proposes to add a decoder as the auxiliary branch without the self-attention blocks. The proposed approach achieves competitive detection performances. ", "questions": "1. Is there any similarity between “gather ↔ disperse” phenomenon vs. the evolution algorithm with “prune out ↔ grow back”?\n2. can this be combined with other practices such as stable matching and memory fusion to further improve the performances?\n3. is it possible to add the C-decoder with self-attention module only in Table 3 for the completeness?", "limitations": "adequate", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. this paper leverages the opposite effects of self-attention and cross-attention to build a auxiliary decoder and improves the detection performances ", "weaknesses": "1. how general does the gather ↔ disperse effect apply? is there more examples beyond Figure 1\n2. it's unclear if this can be combined with other practices such as stable matching and memory fusion to further improve the performances", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**9psD-Q1: How general does the \"gather$\\leftrightarrow$disperse\" effect apply? Is there more examples beyond Fig.1 ?**\n\n **[Ans]:** Thanks for this good question. During rebuttal, we make a statistic across 20\\% randomly-sampled images in COCO-2017 and validate that the \"gather$\\leftrightarrow$disperse\" phenomenon is general. Specifically, we employ **1)** the averaged Euclidean distance between queries around the same object and **2)** the averaged IoU between each query and its nearby object, to measure the feature distance and positional distance, respectively. The results show that for the three cases (before the decoder, without decoder self-attention layers, and with decoder self-attention layers), the averaged feature distances are 22.58 $\\rightarrow$ 12.35 (gather) $\\rightarrow$ 14.13 (disperse) and the averaged IoU scores are 0.41 $\\rightarrow$ 0.72 (gather) $\\rightarrow$ 0.62 (disperse). These statistical results are consistent with our observation in Fig. 1 in the manuscript and show the \"gather$\\leftrightarrow$disperse\" phenomenon is general. Moreover, we provide more visualizations in the supplementary PDF. \n \n* * *\n\n**9psD-Q2: It's unclear if this can be combined with other practices such as stable matching and memory fusion to further improve the performances**\n\n **[Ans]:** Thank you for this suggestion. During rebuttal, we combine our DAC-DETR with the stable matching and memory fusion and observe clear (+0.7 AP) improvement. Specifically, DAC-DETR achieves 50.0 AP on the DINO baseline (COCO, 12 training epochs, ResNet-50 backbone). Adding the stable matching and memory fusion brings another round of +0.7 AP (50.0 $\\rightarrow$ 50.7) improvement, showing good compatibility. We note that the Stable DINO's official code has not been released and the above experiment is based on our own implementation. Using its future official code might bring better results. \n\n* * *\n\n**9psD-Q3: Is there any similarity between \"gather$\\leftrightarrow$ disperse\" phenomenon vs. the evolution algorithm  with \"prune out$\\leftrightarrow$grow back\"?**\n\n **[Ans]:** Thanks. Constructing analogy between the \"gather$\\leftrightarrow$disperse\" phenomenon and the \"prune out$\\leftrightarrow$grow back\" operation in evolution algorithm is insightful and interesting: the \"disperse\" is indeed similar to the \"prune out\". There is difference as well: in evolution algorithm, what have been pruned out are permanently removed and what grow back are actually new$^{[1]}$. In contrast, in DETR, the queries are not removed and will be gathered again, yielding conflict that compromises  the training efficiency.\n\n[1] Torsten Hoefler, et al. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks, 2021.\n\n* * *\n\n**9psD-Q4: Is it possible to add the C-decoder with self-attention module only in Table 3 for the completeness?**\n\n **[Ans]:**  Thanks. Following your suggestion, we build a variant whose C-Decoder has only self-attention layers (no cross-attention layers). As shown in the table below, this variant (Variant-4) is even lower than the baseline by -2.4 AP. It is reasonable because the cross-attention layer is prerequisite for the detector decoder to derive the object information. We will add the result into Table 3 (variants comparison) in the manuscript. \n \n| Method |Backbone  | epochs | AP | AP50 |\n| --- | :---: | :---: | :---: | :---: | \n| Baseline | R50 | 12 | 43.7 | 63.0 |\n| Variant-4  | R50 | 12 | 41.3 | 60.3 |\n| DAC-DETR (ours) | R50 | 12 | 47.1 | 64.8 |\n"}, {"review_id": "BhsMN5yAW6", "reviewer": "Reviewer_LpXx", "summary": "This paper proposes a simple modification to the DETR architecture that improves upon several prior implementations. The paper identifies that the single decoder approach causes the model to try and achieve opposing objectives in terms of the query coverage and deduplication, and proposes a solution to the problem : an additional decoder having only cross attention layers is employed which improves performance on a standard benchmark (COCO). ", "questions": "1. What happens if you try to use the predictions from the C-decoder during inference as well? For instance by combining the predictions from the two decoders? Does it further improve performance? \n2. Line 172 \"and the second one is to suppress the label imbalance regarding different objects.\" Could you explain more clearly how this is achieved?\n3. In Figure 3, is this average number of queries computed on the C-decoder or the O-decoder? \n\n\nSuggestions: \n1. Add an explanation of what t is in Figure 3 caption and change the y axis label to \"avg number of queries / object\"\n2. The scale of the figures in Figure 3 should be kept constant, otherwise at first glance it looks like the gap increases at higher threshold, which is not the case\n3. Writing suggestions: In several places, the phrase \"the contrary\" is used in a gramatically incorrect manner (divide the cross attention from the contrary / their contrary impairs ... etc) and I'd like to suggest that would be better substituted with the following phrasing: \n\n\"To improve the training efficacy, we propose a Divide-And-Conquer DETR (DAC-DETR) that divides the cross-attention out from this contrary for better conquering\" --> \"To improve the training efficacy, we propose a Divide-And-Conquer 6 DETR (DAC-DETR) that separates out the cross attention to avoid these competing objectives.\" \n\n\"have some contrary impacts on the object queries.\" -> \"Have some opposing effects on the object queries\"\n\n\"These two impacts are both critical for DETR (as explained later), and their contrary impairs the training efficacy.\" -> \"These two impacts are both critical for DETR (as explained later), and their contrasting effects impairs...\"\n\n\"In spite of the contrary, these two effects are both important for DETR.\" --> \"In spite of the conflicting /rival objectives, we find that these two effects ....\" \n\nAnd similarly for other occurrences of the word contrary. \n\nL169 grount-truth -> ground-thruth\n\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "## Originality and Significance \nThe findings about the self attention performing deduplication and cross attention gathering information from the image were already knows (as mentioned in this paper). The primary contribution here are\n* Identifying that these two effects actually are in competition with each other, causing the training speed and final performance to be affected. \n* Proposing a solution by adding an additional decoder having only cross attention. \n* Displaying that this approach can be combined with several DETR variants and improves performance while reducing training time in all those cases \n* Ablations showing that freezing weights in the O-decoder, adding self attention layers in C-decoder and removing the one-to-many matching all reduce performance, supporting the findings of the paper. \nAdding an additional cross-attention only decoder is a simple modification that can be implemented on top of most other DETR variants, making the finding useful and significant. Since the added layers share weights with the original decoder, the size of the model does not increase, and the additional decoder is not used during inference, ensuring that there is no latency in inference time due to the proposed modification. \n\n## Clarity and Quality\n* Clear figures with descriptive stand-alone captions\n* The findings are clearly explained with supporting evidence", "weaknesses": "* No analysis on if there is any difference in performance on large/small objects and common/rare objects\n* Some qualitative examples of difference between the proposed approach and the baselines would be helpful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**LpXx-Q1: Analysis on if there is any difference in performance on large/small objects and common/rare objects.**\n\n **[Ans]:**  Thanks for this suggestion. The performance gain on large/small objects is slightly larger/smaller (as shown in Table 1 in the manuscript), and the performance gain on common/rare objects is relatively smaller and larger (as shown in the table below). The details are as below:\n \n1) Large objects vs small objects. Table 1 in the manuscript already provides detailed results on small ( AP$_S$ ), medium ( AP$_M$ ) and large ( AP$_L$ ) objects. It is observed that AP$_L$ has slightly larger improvement than ( AP$_S$ ). For example, on the Deformable-DETR baseline, the improvement on AP$_L$ and AP$_S$ is +4.3 AP and +3.0 AP, respectively. This is because small objects are inherently hard to detect and improving AP$_S$ is more difficult.\n\n2) Common objects vs rare objects. We compare two most-frequent classes ('person' and 'car') against two rarest classes ('toaster' and 'hair direr') in COCO in the table below. It is observed that the improvement on the rare classes (+6.1 AP) is larger than on the common classes (+3.8 AP). We infer it is because the benefit of DAC-DETR (*i.e.*, improving the gathering effect) is more significant on rare classes. \n\n\n\n| Method | Common (person  \\&  car)  |  Rare (toaster \\&   hair direr)  | \n| :---: | :---: | :---: | \n| Baseline | 49.6 | 22.2 | \n| Variant-4  | 53.4 (+3.8) | 28.3 (+6.1) |  \n\n* * *\n\n**LpXx-Q2: Some qualitative examples of difference between the proposed approach and the baselines would be helpful.**\n\n**[Ans]:** Thanks. Section A.6 (appendix) provides some qualitative examples of our DAC-DETR and the baseline. In the visualized examples, DAC-DETR shows better IoU and higher confidence on some hard examples (*e.g.*, occluded zebra, a cat wearing the hat). We think this is because DAC-DETR can improve both the quantity and quality of the queries for each object. \n\n* * *\n\n**LpXx-Q3: What happens if you try to use the predictions from the C-decoder during inference as well? For instance by combining the predictions from the two decoders? Does it further improve performance?**\n\n**[Ans]:** Thanks for this good suggestion. During rebuttal, we combine the predictions from the O-Decoder and C-Decoder and find it brings slight improvement (e.g., +0.3 AP). Specifically, we average the predicted logits of two decoders, use softmax to transform the averaged logits into classification scores and then use NMS to suppress the duplicate detections. On the Deformable-DETR baseline, while our DAC-DETR already achieves 47.1 AP (12 training epochs on CoCo), combining C-Decoder and O-Decoder brings another round of +0.3 AP improvement. We note that this improvement might become smaller on higher baselines and it considerably increases the inference cost. \n\n\n* * *\n\n**LpXx-Q4: Line 172 \"and the second one is to suppress the label imbalance regarding different objects.\" Could you explain more clearly how this is achieved?**\n\n**[Ans]:** Thanks for your kind reminder. It is because an inherently easy-to-recognize object tends to attract many queries, therefore having many high-scored queries (which is also observed by DETA). Only using the threshold will make the easy-to-recognize objects have much more positive queries than the hard-to-recognize objects have, therefore causing label imbalance. Adding the top-$k$ selection suppresses the imbalance problem to some extent. \n\n* * *\n\n**LpXx-Q5: In Fig.3, is this average number of queries computed on the C-decoder or the O-decoder?**\n\n**[Ans]:** Thanks for the question. For our DAC-DETR, the average number of queries are computed on the C-decoder (no self-attention). As for the baseline, we remove its self-attention layers in the decoder on the already-trained model. It allows us to make a fair comparison on the gathering effect of two models. We will add these details to the manuscript.\n\n* * *\n\n**LpXx-Suggestions on figures and writings:**\n\n**[Ans]:** We sincerely thank your valuable suggestions and kind reminders. These suggestions help a lot in refining our manuscript. We will go through these details and make the revisions as below:\n\n1) In Fig.3 , $t$ is the threshold on the matching score. We will add the explanation into the caption and change the y axis label to \"avg number of queries / object\".\n\n2) We will make the scale of the figures constant in Fig. 3.\n\n3) We appreciate that you pointed out the use of word \"contrary\" is actually inaccurate and grammatically incorrect. We will replace it with the suggested words (*e.g.*, conflicting objectives, opposing). We will carefully go through these typos and make our expression accurate. \n\n* * *"}, {"review_id": "difN8WtPj1", "reviewer": "Reviewer_CkgD", "summary": "The authors find that the cross-attention and self-attention in the DETR decoder have opposite effects on object queries. This phenomenon reduces the training efficiency of DETR model. To resolve the contradiction, this paper proposes a Divide-And-Conquer DETR that employs an auxiliary decoder that shares parameters with the original decoder. Experimental results show that DAC-DETR achieves significant performance over popular DETRs.", "questions": "1.In Figure 1 (b,c), which module is used to obtain the classification score and bounding box of the object? Why \"cls\" of objects in Figure 1(b) is so different from \"cls\" scores in Figure 1(b). Does the same phenomenon occur at the last layer of the DETR decoder and occur at the DINO?\n2. This paper proposes DAC-DETR to resolve the contradiction between cross-attention and self-attention. Whether this problem in Figure1 is solved by the corresponding method, the authors should give more visualization analysis like Figure1, used DAC or not.\n3. In Table 3. why the AP of varaiant2 and varaiant3 is reduced?", "limitations": "1. The description of the one-to-many label for assignment In section 3.2 is a bit too abstract. This is a key component of the proposed method, from Table 4, and the major difference with the existing method.\n2. The authors proposed DAC-DETR to solve the confliction problem of cross-attention and self-attention, but innovative structures are not proposed to fundamentally solve the problem.\n3. Compared with one-to-many Hungarian Loss in Table 4, the increase of AP is due to matching scores introducing an IoU score, which is also compatible with the original DETR.\n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper adequately cites relevant work and clearly articulates the differences from existing work. The experiments show promising results. And other researchers may build a network structure based on this paper. ", "weaknesses": "* *Originality**: This article is a combination of a series of existing methods and is not innovative enough.\n* *Quality*：The motivation is straightforward, but the authors should give more analysis about contradiction. DAC-DETR brings improvement over baseline, but this does not indicate that the problem has been solved by theoretical analysis or experimental results. \n* *Clarity*：This paper is not well organized and has some grammar errors.\n* *Significance*：The model architecture is not adequately simple, which will affect its usability. From the experimental results, the authors proposed DAC-DETR to solve the conflict problem of cross-attention and self-attention but did not fundamentally innovate the structure to solve the problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**CkgD-Q1: This article is a combination of a series of existing methods and is not innovative enough.**\n\n**[Ans]:**  We respectfully disagree with this point. We would like to highlight our major contributions as below:\n\n1) We reveal a characteristic of DETR, *i.e.*, the cross-attention and self-attention layers in DETR decoder have opposing effects of \"gather $\\leftrightarrow$ disperse\". This characteristic has not been noticed before and provides a new insight for understanding the training difficulty of DETR. \n\n\n2) Based on this insight, we propose to separate the cross-attention out from the contradiction. This objective is implemented with a simple design, *i.e.*, adding a C-Decoder that replicates the original decoder but removes all the self-attention layers. \n\n3) We show that DAC-DETR brings consistent improvements to popular DETR-like methods. For example, under 12 epochs learning scheme, DAC-DETR improves Deformable-DETR, Deformable-DETR++ and DINO by +3.4, +2.3 and +1.0 AP, respectively.\n\n* * *\n\n**CkgD-Q2: The motivation is straightforward, but there should be analysis (in addition to the accuracy improvement) to show the problem has been solved.**\n\n**[Ans]:** Thanks. Section 3.3 (mechanism analysis) and Fig. 3 in the manuscript have already shown that DAC-DETR suppresses the contradiction: **1)** the number of queries gathered to each object is enlarged, and **2)** the best query for each object becomes even closer to the corresponding object.\n\n* * *\n\n**CkgD-Q3: The model architecture is not adequately simple, which affects its usability.**\n\n**[Ans]:** We respectfully recall that our method only adds small complexity (C-Decoder) to the baseline structure. C-Decoder is simple: it replicates the structure of the original decoder (O-Decoder) and removes the self-attention layers. Our architecture complexity is comparable (usually lower), compared with recent state-of-the-art methods, *e.g.*, Hybrid-DETR adds a parallel decoder branch that have more quires, Group-DETR replicates the original decoder for multiple times. Correspondingly, our DAC-DETR has higher training efficiency compared with recent state-of-the-art methods (Section A.3 in the appendix).\n\n* * *\n\n**CkgD-Q4: In Fig.1 (b,c), which module is used to obtain the classification score and bounding box of the object? Why \"cls\" of objects in Fig.1(b) is different from \"cls\" scores in Fig.1 (c). Does the same phenomenon occur at the last layer of the DETR decoder and occur at the DINO?**\n\n**[Ans]:** Thanks for the questions. **1)** The classification score and the bounding box prediction are obtained at the last (6-th) decoder layer. **2)** The classification score difference is because in Fig.1 (b) removes all the self-attention layers in the decoder of the already-trained model in Fig.1 (c). Correspondingly, the detector loses the capability to suppress duplicates (L36 in the manuscript) and thus increases the predicted score for multiple queries. **3)** Fig. 1 already uses the last layer. The phenomenon is similar in earlier layers and also occurs in DINO (as visualized in the supplementary PDF for rebuttal).\n\n* * *\n\n**CkgD-Q5: Analysis on whether the contradiction problem in Figure1 is solved by the corresponding method.**\n\n**[Ans]:** Thanks. Section 3.3 (mechanism analysis) and Fig. 3 in the manuscript have already shown that DAC-DETR suppresses the contradiction. The evidences are **1)** the number of queries gathered to each object is enlarged and **2)** the best query for each object becomes even closer to the corresponding object. We note that Fig. 3 is a statistic across the whole validation set and is more general than single sample visualization.\n\n* * *\n\n**CkgD-Q6: In Table 3. why the AP of variant-2\\&3 is reduced?**\n\n**[Ans]:** Thanks for this question. We guess your question is why the AP of variant-2 and variant-3 is even slightly lower than the baseline. This is reasonable. **1)** Variant-2 re-adds the self-attention layers into the C-Decoder, and thus can be viewed as having two O-Decoders. Applying additional one-to-many matching for the O-Decoder makes the model prone to duplicate predictions and thus reduces the accuracy. **2)** Variant-3 uses one-to-one matching for C-Decoder, while C-Decoder has no self-attention and is naturally prone to make duplicate (one-to-many) predictions. In other words, in Variant-3, the supervision has some conflict with the characteristic of C-Decoder (L255 in the manuscript), therefore reducing the accuracy.\n\n* * *\n\n**CkgD-Q7: The description of the one-to-many matching in section 3.2 is a bit too abstract. This is a key component of the proposed method, from Table 4, and the major difference with the existing method.** \n\n**[Ans]:** We respectfully disagree. The one-to-many label assignment is only a technical detail for training our C-Decoder and is indeed simple: we calculate the matching score (Eqn. 4) and assign the positive labels to highly-scored queries. Our core contributions are the discovery of the opposing \"gather $\\leftrightarrow$ disperse\" phenomenon and the corresponding C-Decoder solution. The C-Decoder is not restricted to any particular one-to-many method and is potential to utilize better one-to-many method for further improvement.\n\n* * *\n\n**CkgD-Q8: In Table 4, compared with one-to-many Hungarian Loss, the increase of AP is due to your matching score introducing an IoU score.** \n\n**[Ans]:** We apologize for causing a confusion. In Table 4, two editions of DAC-DETR (\"w/ one-to-many Hungarian\" and ours) actually use the same matching score definition (Eqn. 4). Their only difference is how to assign positive labels: the former uses set-to-set Hungarian while the latter (ours) uses the \"threshold + ranking\" criterion. Therefore, our superiority is not because better matching score, but is because C-Decoder favors the threshold strategy (L265 in the manuscript). We will clarify this point in the manuscript.\n\n"}, {"review_id": "lzxarO19vo", "reviewer": "Reviewer_QUSJ", "summary": "This paper observes the problems in cross-attention and self-attention that impacts the queries, and proposes to use divide-and-conquer to improve the training accuracy.", "questions": "No", "limitations": "No", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is considered novel to me. The insights in analyzing the cross-attention and self-attention and the proposed design are interesting.\n2. Strong performance. The final performance is improved compared with SOTA detectors.", "weaknesses": "1. The design seems a little complex. Not sure if this design can be plugged into other DETR-like model easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**QUSJ-Q1: The design seems a little complex. Not sure if this design can be plugged into other DETR-like model easily.**\n\n**[Ans]:** Thanks for this question. Plugging our method into the DETR-like models is easy. Given a DETR-like model, we only need to append a C-Decoder with two steps: **1)** replicating its original decoder and **2)** removing the self-attention layers. The C-Decoder is supervised with the one-to-many matching, which may seem a bit complex but is reliable (DAC-DETR has considerable robustness on the one-to-many hyper-parameters, as shown in Fig. 4 in the manuscript). In our experiments, we already plug our DAC-DETR into multiple popular baselines, *i.e.*, Deformable-DETR, Deformable-DETR++, DINO, and achieve consistent improvement. We show our method is compatible to several most recent good practices, *e.g.*, Aligned loss and stable matching (during rebuttal), as well. In all these experiments, we use the same hyper-parameters for DAC-DETR."}, {"review_id": "D2XkqG0h3v", "reviewer": "Reviewer_VXuS", "summary": "This paper introduces DAC-DETR, a method to improve the training efficacy of DEtection Transformers (DETR) by addressing the contrary impacts of cross-attention and self-attention layers in the DETR decoder. DAC-DETR divides the cross-attention layers into an auxiliary decoder, which focuses on learning the cross-attention, while the original decoder handles non-duplicate detection. By employing one-to-many label assignment in the auxiliary decoder, DAC-DETR effectively improves the gathering effect of queries, leading to improved detection accuracy compared to popular DETR baselines.", "questions": "The reviewer is curious about the performance of the one-to-many branch (C-decoder). ", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-organized, presenting its ideas and findings in a clear and coherent manner. \n- The motivation is clear. The proposed methods are straightforward and logical, making them easy to understand and implement.\n- The effectiveness of the proposed methods is well-supported by a comprehensive set of experiments. The experimental results demonstrate significant improvements over existing DETR models.", "weaknesses": "I have no severe concerns about the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**VXuS-Q1: The reviewer is curious about the performance of the one-to-many branch (C-decoder).**\n\n**[Ans]:** Thanks for this question. C-Decoder (with NMS) achieves slightly lower accuracy than the O-Decoder. Since C-Decoder has no self-attention layers and makes duplicate detection, NMS is prerequisite for filtering out the duplicate predictions. Based on the Deformable-DETR baseline (43.7 AP), using C-Decoder (+NMS) achieves 46.9 AP, which is -0.2 lower than using O-Decoder (47.1 AP). \n\nMoreover, we notice a related question (raised by review **LpXx-Q3**) that might draw your interest as well: combining the predictions from C-Decoder and O-Decoder may further bring incremental improvement. For example, on the above setting, combining the C-Decoder (46.9 AP) and O-Decoder (47.1 AP) achieves 47.4 AP. Please kindly refer to **LpXx-Q3** for more details."}]

Author rebuttal:
**General response**

We thank all the reviewers for their valuable comments. We provide point-to-point responses to each reviewer, as well as a supplementary  PDF for some visualization results.

Meta-review: The reviewers agree that the paper presents a valuable contribution. The main strengths of the paper are 
- the idea that the effect of deduplication and gathering information are in conflict with each other, leading the authors to propose an additional decoder without self-attention.
- strong empirical performance.
While some reviewers noted that the approach adds some complexity to the model and the novelty of the architecture is limited, the value of the experiment outweigh these limitations and I recommend acceptance.
Decision: Accept (poster)