DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR
Authors: Shilong Liu
Feng Li
Hao Zhang
Xiao Yang
Xianbiao Qi
Hang Su
Jun Zhu
Lei Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: Object detection
Transformer

Abstract: We present in this paper a novel query formulation using dynamic anchor boxes for DETR (DEtection TRansformer) and offer a deeper understanding of the role of queries in DETR. This new formulation directly uses box coordinates as queries in Transformer decoders and dynamically updates them layer by layer. Using box coordinates not only helps using explicit positional priors to improve the query-to-feature similarity and eliminate the slow training convergence issue in DETR, but also allows us to modulate the positional attention map using the box width and height information. Such a design makes it clear that queries in DETR can be implemented as performing soft ROI pooling layer by layer in a cascade manner. As a result, it leads to the best performance on the MS-COCO benchmark among the DETR-like detection models under the same setting, e.g., AP 45.7\% using ResNet50-DC5 as backbone trained in 50 epochs. We also conducted extensive experiments to confirm our analysis and verify the effectiveness of our methods. Code is available at \url{https://github.com/IDEA-opensource/DAB-DETR}.

Published as a conference paper at ICLR 2022
DAB-DETR: DYNAMIC ANCHOR BOXES ARE BETTER QUERIES FOR DETR
Shilong Liu Feng Li Hao Zhang?23, Xiao Yang', Xianbiao Qi?, Hang Su Su Jun Zhu Lei Zhang" 21
Deppt. of Comp. Sci. and Tech., BNRist Center, State Key Lab for Intell. Tech. & Sys., Institute for AI, Tsinghua-Bosch Joint Center for ML, Tsinghua University.
IInternational Digital Economy Academy (IDEA).
3 Hong Kong University of Science and Technology.
*Peng Cheng Laboratory, Shenzhen, Guangdong, China.
yat..0yyyyngxxill999mm.... 1.. i inguaa edu.cn
{fliay, hww.hhhgg///ccooeetttukk ust.
fgixianbia,, Leizaang) edu. (suhangss, dos2j | tssss:// edu. on
ABSTRACT
We present in this paper novel query formulation using dynamic anchor boxes for DETR (DEtection TRansformer) and offer deeper understanding of the role queries DETR. This new formulation directly uses box coordinates as queries in Transformer decoders and dynamically updates them layer by layer. Using box coordinates not only helps using explicit positional priors to improve the query- to-feature similarity and eliminate the slow training convergence issue in DETR. but also allows us to modulate the positional attention map using the box width and height information. Such design makes it clear that queries in DETR be implemented as performing soft ROI pooling layer by layer in a cascade man- ner. As result, it leads to the best performance on MS-COCO benchmark among the DETR- detection models under the same setting, e.g., AP 45.7% using ResNet50-DCC5 as backbone trained in 50 epochs. We also conducted extensive experiments to confirm our analysis and verify the effectiveness of our meth- ods. Code available at https: /tap::///itiie wtt.thttt.uuu.goo com IDEA- Bet///DEEAO-Deeeee opensource/ DAB-DETR.
INTRODUCTION
Object detection is fundamental task in computer vision of wide applications. Most classical detectors are based on convolutional architectures which have made remarkable progress in the last decade (Ren al., 2017; Girshick, 2015; Redmon al., 2016; Bochkovskiy al., 2020; Ge et al., 2021). Recently, Carion et al. (2020) proposed a Transformer-based end-to-end detector named DETR (DEtection TRansformer). which eliminates the need for hand-designed components, e.g., anchors, and shows promising performance compared with modern anchor-based detectors such as Faster RCNN (Ren et al., 2017).
In contrast to anchor-based detectors, DETR models object detection as a set prediction problem and uses 100 learnable queries to probe and pool features from images, which makes predictions without the need of using non mox-mmm suppression. However, due to its ineffective design and use of queries, DETR suffers from significantly slow training convergence, usually requiring 500 epochs to achieve a good performance. To address this issue, many follow-up works attempted to improve the design of DETR queries for both faster training convergence and better performance (Zhu al., 2021; Gao et al., 2021; Meng et al., 2021; Wang al., 2021).
'This work was done when Shilong Liu was intern at IDEA
"oorrsponniig author.

---

Published as a conference paper at ICLR 2022
Layor1
Layer
Emassiaation
Emnosittetion
Rcforence PORYS
Encedings
Encosinngs
Encosings
Decuuder
Liatralee
Decadder
Leammable
Decodder
Entoediinngs
Queras
Ersteddings
Quaikes
eelaate
Erstedditgss
Anchtors
(a) DETR
Conditional DETR
(c) DET
Figure I: Comparison of DETR, Conditional DETR, and our proposed DAB-DETR. For clarity, we only show the cross-attention part in the Transformer decoder. (a) DETR uses the learnable queries for the layers without any adaptation, which accounts for its slow training convergence. (b) Con- ditional DETR adapts the learnable queries for each layer mainly to provide better reference query point to pool features from the image feature map. In contrast, (c) DAB-DETE directly uses dynam- ically updated anchor boxes to provide both reference query point (ar, y) and a reference anchor size (w, h) to improve the cross-attention computation. We marked the modules with difference in purple.
Despite all the progress, the role of the learned queries in DETR is still fully understood uti- lized. While most previous attempts make each query in DETR more explicitly associated with one specific spatial position rather than multiple positions, the technical solutions are largely different. For example, Conditional DETR learns conditional spatial query by adapting query based on its content feature for better matching with image features (Meng al., 2021). Efficient DETR intro- duces dense prediction module to select top-K K object queries (Yao et al., 2021) and Anchor DETR formulates queries as 2D anchor points (Wang et al., 2021), both associating each query with spe- cific spatial position. Similarly, Deformable DETR directly treats 2D reference points queries and performs deformable cross-attention operation at each reference points (Zhu et al., 2021). But all the above works only leverage 2D positions as anchor points without considering the object scales.
Motivated by these studics, we take a closer look at the cross-attention module in Transformer decoder and propose
anchor boxes, i.e., 4D box coordinates (Ir, y, w, h), queries in DETR and update them layer by layer. This new query formulation introduce better spatial priors for the cross-attention module by considering both the position and size of each anchor box, which also leads to a much simpler implementation and a deeper understanding of the role of queries in DETR.
The key insight behind this formulation is that each query in DETR is formed by two parts: content part (decoder self-attention output) and positional part (e.g.. learnable queries in DETR) The cross-attention weights are computed by comparing query with a set of keys which consists of two parts as a content part (encoded image feature) and a positional part (positional embedding). Thus, queries in Transformer decoder can be interpreted as pooling features from a feature map based on the query-to-feature similarity measure, which considers both the content and positional information. While the content similarity is for pooling semantically related features, the positional similarity is to provide positional constraint for pooling features around the query position. This attention computing mechanism motivates us to formulate queries as anchor boxes as illustrated in Fig. (c), allowing to use the center position (z, y) of an anchor box to pool features around the center and use the anchor box size (w. h) to modulate the cross-attention map, adapting it to anchor of coordinates queries, anchor boxes can be updated
box size. In addition, because of the layer by layer dynamically. In this way, queries in DETR can be implemented as performing soft ROI pooling layer by layer in a cascade way.
We provide a better positional prior for pooling features by using anchor box size to modulate the cross-attention. Because the cross-attention can pool features from the whole feature map, it is
'See the DETR implementation at https: / //g hub. ponents of queries and keys are also shown in each subplot of Fig. 1. Note that the learnable queries in DETR only for the positional part. Related discussion can also be found in Conditional DETR (Meng et al., 2021).
The com-
2

---

Published as a conference paper at ICLR 2022
erucial provide proper positional prior for each query to let the cross-attention module focus on a local region corresponding to a target object. It can also facilitate to speed up the training convergence of DETR. Most prior works improve DETR by associating each query with a specific location, but they assume an isotropic Gaussian positional prior of a fixed size 4 (b)), which is inappropriate for objects of different scales. With the size information (w, h) available in each query anchor box, we can modulate the Gaussian positional prior as an oval shape. More specifically, we divide the width and height from the cross-attention weight (before softmax) for its x part and y part separately, which helps the Gaussian prior to better match with objects of different scales(Fig. 4 (c)). To further improve the positional prior, we also introduce a temperature parameter to tune the flatness of positional attention, which has been overlooked in all prior works.
In summary, our proposed DAB-DETR (Dynamic Anchor Box DETR) presents a novel query for- mulation by directly learning anchors as queries. This formulation offers a deeper understanding of the role of queries, allowing us to use anchor size to modulate the positional cross-attention map in Transformer decoders and perform dynamic anchor update layer by layer. Our results demon- strate that DAB-DETR attains the best performance among DETR-like architectures under the same setting on the COCO object detection benchmark. The proposed method can achieve 45.7% AP when using a single ResNet-50 (He et al., 2016) model as backbone for training 50 epochs. We also conducted extensive experiments confirm our analysis and verify the effectiveness of our methods.
2 RELATED WORK
Most classical detectors are anchor-based, using either anchor boxes (Ren et al., 2017; Girshick, 2015: Sun et al., 2021) or anchor points (Tian et al., 2019; Zhou et al., 2019). contrast, DETR (Carion et al., 2020) is fully anchor-free detector using set of lcarnable vectors queries. Many follow-up works attempted to solve the slow convergence of DETR from different perspec- tives. Sun et al. (2020) pointed out that the cause of slow training of DETR is due to the cross- attention in decoders and hence proposed an encoder-only model. Gao et al. (2021) instead intro- duced Gaussian prior to regulate the cross-altention. Despite their improved performance, they did not give proper explanation of the slow training and the roles of queries in DETR.
Another direction to improve DETR, which is more relevant to our work, is towards a deeper un- derstanding of the role of queries in DETR. As the learnable queries in DETR are used to provide positional constraints for feature pooling, most related works attempted to make each query in DETR more explicitly related to a specific spatial position rather than multiple position modes in the vanilla DETR. For example, Deformable DETR (Zhu et al., 2021) directly treats 2D reference points as queries and predicts deformable sampling points for each reference point to perform the deformable cross-attention operation. Conditional DETR (Meng et al., 2021) decouples the attention formula- tion and generates positional queries based reference coordinates. Efficient DETR (Yao et al., 2021) 1) introduces a dense prediction module to select top-K positions as object queries. Although these works connect queries with positional information, they do not have explicit formulation to use anchors.
Different from the hypothesis in prior works that the learnable query vectors contain box coordinate information, our approach is based on a new perspective that all information contained in queries box coordinates. That is, anchor boxes are belier queries for DETR. A concurrent work Anchor DETR (Wang et al., 2021) also suggests learning anchor points directly, while it ignores the anchor width and height information in other prior works. Besides DETR. Sun et al. (2021) proposed a sparse detector by learning boxes directly. which shares a similar anchor formulation with us. but discards the Transformer structure and leverages hard ROI align for feature extraction. Table summarizes the key differences between related works and our proposed DAB-DETR. We compare our model with related works on five dimensions: the model directly learns anchors, if the model predicts reference coordinates (in its intermediate stage), if the model updates the reference anchors layer by layer, if the model uses the standard dense cross- attentienn if the attention is modulated to better match with objects of different scales, and if the model updates the learned queries layer by layer. A more detailed comparison of DETR-like models is available in Sec. of Appendix. We recommend this section for readers who have confusions about the table.
3

---

Published as a conference paper at ICLR 2022
Miski
Leam Anchon??
Refercect Anchots
Dyeamic Anciic
Sundaad Aleerii
Size Midallzed Alieniinn
Update Leancd Spaiia Queries?
DETR
Deformable BETR
No
40
SMCA
No
Conditinn
No
Andho DETS
20
20
Spane KCNN
DAS-DETR
40
40
Table 1: Comparison of representative related models and our DAB-DETR. The term "Learn An- chors?" asks if the model learns 2D points or 4D anchors as learnable parameters directly. The term "Reference Anchors" means if the model predicts relative coordinates with respect to a ref- erence points/anchors. The term "Dynamie Anchors" indicates if the model updates its anchors layer. The term "Standard Attention" shows whether the model leverages the standard dense attention in cross-attention modules. The term "Object Scale-Modulated Attention" means the attention is modulated to better match with object scales, The term "Size-Modulated Attention" means if the attention is modulated to better match with object scales. The term "Update Spatial Learned Queries?" means if the learned queries updated layer by layer. Note that Sparse RCNN not a DETR-like architecture. we list here for their similar anchor formulation with us. See Sec. of Appendix for detailed comparison of these models.
3 WHY A POSITIONAL PRIOR COULD SPEEDUP TRAINING?
decoder
Image
Cross-Attenion
Image Festunes
Sel-Attenion
Posiional Embedeings
Postiional Embeddinng
Decoder F Embeidings
Learmalle Querles
(8) Self-attention encoder
(b) Crass-attention decoder DETR
Figure 2: Comparison of self-attention in encoders and cross-atiention decoders of DETR. As they have the same key and value components, the only difference comes from the queries. Each query an encoder composed of an image feature (content information) and positional embedding (positional information), whereas each query in decoder is composed of a decoder embedding (content information) and learnable query (postional information). The differences between two modules are marked in purple.
Much work has been done to accelerate the training convergence speed of DETR, while lacking unified understanding of why their methods work. Sun et al. (2020) showed that the cross-attention module is mainly responsible for the slow convergence, but they simply removed the decoders for faster training. We follow their analysis to find which sub-module in the cross-attention affects the performance. Comparing the self-attention module in encoders with the cross-attention module in decoders, we find the key difference between their inputs comes from the queries, as shown in Fig. 2. As the decoder embeddings are initialized O, they are projected to the same space as the image features after the first cross-attention module. After that, they will through a similar process in decoder layers as the image features in encoder layers. Hence the root cause likely due to the learaalee queries.
Two possible reasons in cross-attention account for the model's slow training convergence: 1) is hard to learn the queries due to the optimization challenge, and 2) the positional information in the learned queries is not encoded in the same way as the sinusoidal positional encoding used for image features. To see if it is the first reason, reuse the well-learned queries from DETR (keep them fixed) and only train the other modules. The training curves in Fig. 3(a) show that the fixed queries only slightly improve the convergence in very early epochs, e.g., the first 25 epochs. Hence the query learning (or optimization) is likely not the key concern.
Then turn to the second possibility and try to find out if the learned queries have some undesir- able properties. As the learned queries are used to filter objects in certain regions, we visualize a few positional attention maps between the learned queries and the positional embeddings of image
4

---

Published as a conference paper at ICLR 2022
1919
CUOCOM
4por7
6922)
65301
<pEc3
(a)
(b)
Figure 3: a): Training curves of the original DETR and DETR with fixed queries. b): Training curves of the original DETR and DETR We run each experiment times and plot the mean value and the 95% confidence interval of each item.
DETR
(3) Conditona DETR
(d DAB-DETR
Figure 4: We visualize the positional attention between positional queries and positional keys for DETR, Conditional DETR, and our proposed DAB-DETR. Four attention maps in (a) are randomly sampled, and select figures with similar query positions in (a) for (b) and (c). The darker the color, the greater the attention weight, and vice versa. calculated by performing dot product between a learned query and positional embeddings from feature map, and can have multiple modes and unconcentrated attentions. (b) The positional queries in Conditional DETR encoded in the same way as the image positional embeddings, resulting in Gaussian-lic like attention maps. However, it cannot adapt to objects of different scales. (c) DAB- DETR explicitly modulates the attention map using the width and height information of an anchor, making it more adaptive to object size and shape. The modulated attentions can be regarded as helping perform soft ROI pooling.
(a) Each attention map in DETR is
features in Fig. 4(a). Each query can regarded as a positional prior to let decoders focus on region of interest. Although they serve as a positional constraint, they also carry undesirable prop- erties: multiple modes and nearly uniform attention weights. For example, the two attention maps at the top of Fig, 4(a) have two or more concentration centers, making it hard to locate objects when multiple objects exist in an image. The bottom maps of Fig. 4(a) focus on areas that are either too large or too small, and hence cannot inject useful positional information into the procedure of fea- ture extraction. We conjecture that the multiple mode property of queries in DETR likely the root cause for its slow training and we believe introducing explicit positional priors constrain queries on a local region is desirable for training. To verify this assumption, we replace the query formula- tion in DETR with dynamic anchor boxes, which can enforce each query focus on specific area, and name this model DETR+DAB. The training curves in Fig. 3(b) show that DETR+DAB leads to much better performance compared with DETR, in terms of both detection AP and training/testing loss. Note that the only difference between DETR and DETR+DAB is the formulation of queries and no other techniques like 300 queries or focal loss are introduced. shows that after addressing the multi-mode issue of DETR queries, we can achieve both faster training convergence and higher detection accuracy.
Some previous works also have similar analyses and confirmed this. For example, SMCA (Gao al., 2021) speeds up the training by applying pre-defined Gaussian maps around reference points. Conditional DETR (Meng et al., 2021) uses explicit positional embedding training, yielding attention maps similar to Gaussian kernels as shown in Fig. 4(b). Although explicit positional priors lead to good performance in training, they ignore the scale information of object. In contrast, our proposed DAB- B-B-DETRR explicitly takes into account the object scale information to adaptively adjust attention weights, as shown in Fig. 4(c).
positional queries for
5

---

Published as a conference paper at ICLR 2022
4 DAB-DETR
NLF
ofof w/h)
(v.4
Adti
Add & Norm
Image Features
Wishh & lieech Rodiatee Mub-iteed
Maihhd
NX
MLP
MLP
Muth Head Seet
Ancher Site
Spatel
Redaa
Figure 5: Framework of our proposed DAB-DETR.
4.1 OVERVIEW
Following DETR (Carion et al., 2020), our model an end-to-end object detector which includes CNN backbone, Transformer (Vaswani et al., 2017) encoders and decoders, and prediction heads for boxes and labels. We mainly improve the decoder part, as shown Fig. 5.
Given an image, we extract image spatial features using CNN backbone followed with Transformer encoders to refine the CNN features. Then dual queries, including positional queries (anchor boxes) and content queries (decoder embeddings), are fed into the decoder to probe the objects which correspond to the anchors and have similar patterns with the content queries. The dual queries are updated layer by layer to get close to the target ground-truth objects gradually. The outputs of the final decoder layer are used to predict the objects with labels and boxes by prediction heads, and then bipartite graph matching is conducted to calculate loss in DETR.
To illustrate the generality of our dynamic anchor boxes, we also design a stronger DAB- Choomable-DETR. which is available in Appendix.
4.2 LEARNING ANCHOR BOXES DIRECTLY
As discussed Sec. regarding the role of queries in DETR, we propose directly learn query boxes say anchor boxes and derive positional queries from these anchors. There two atten- tion modules in each decoder layer, including self-attention module and cross-attention module, which used for query updating and feature probing, respectively. Each module needs queries, keys, and values to perform attention-basee value aggregation, yet the inputs of these triplets differ.
We denote Ag (Ig. Vu Wey hg) the q-th anchor, 2qg Vq. Wy. hy E R. and C E R and Pa R its corresponding content query and positional query, where D is the dimension of decoder embeddings and positional queries.
Given anchor Ag. its positional query Pg is generated by:
Pg MLP(PE(A,)),
(I)
where PE means positional encoding to generate sinusoidal embeddings from float numbers and the parameters of MLP are shared across all layers. As A, a quaternion, we overload the PE operator here:

---

Published as a conference paper at ICLR 2022
PE(A,) g Mqs Wy: wg:hg) hy)
(E(u)) PE(h)))
The notion Cat means concatenation function. In our implementations, the positional encoding function PE maps a float to a vector with D/2 dimensions as: PE: R - IR 1/2, Hence the function MLP projects 2D dimensional vector into D dimensions: MLP: I2 - R The MLP module has two submodules, each of which is composed of a linear layer and a ReLU activation, and the feature reduction is conducted the first linear layer.
In the self-attention module, all three of queries, keys, and values have the same content items, while the queries and keys contain extra position items:
Self-Attn: Qq - - Pa: Kg C V = Cg.
(3)
Inspired by Conditional DETR (Meng al., 2021), concatenate the position and content in- formation together as queries and keys in the cross- rross-attention module, so that we can decouple the content and position contributions to the query-to-feature similarity computed as the dot product be- tween query and key. To rescale the positional embeddings, we leverage the conditional spatial
query (Meng et al., 2021) as well. More specifically, we learn MLP(ssg) IR + IR to obtain scale vector conditional on the content information and use it perform element-wise multiplication with the positional embeddings:
Cross-Attm:
Cat PE( Ua) MLP(sH)( K K Cat(Pa.,y PE(x,
-
(4)
Va.y
Fz.91
where Fa, E RRD is the image feature at position (z, y) and is an element-wise multiplication. Both the positional embeddings in queries and keys are generated based on 2D coordinates, making it more consistent to compare the positional similarity, in previous works (Meng et al., 2021; Wang et al., 2021).
4.3 ANCHOR UPDATE
Using coordinates queries for learning makes it possible to update them layer by layer. In contrast, for queries high dimensional embeddings, such in DETR (Carion et al., 2020) and Conditional DETR (Meng et al., 2021), is hard to perform layer-by-layee query refinement, because it is unclear how to convert an updated anchor back to nigh-dimensionn
query embedding.
Following the previous practice (Zhu al., 2021; Wang et al., 2021), update anchors in each layer after predicting relative positions (Ar, Ay, Aw, Ah) )) by prediction head, as shown in Fig. 5. Note that all prediction heads in different layers share the same parameters.
4.4 WIDTH & HEIGIT-MODULATED GAUSSIAN KERNEL
Tenperstrest
Tempenstane400
-1000
Figure 6: Positional attention maps modu- lated by width and height.
Figure 7: Positional attention maps with dif- ferent temperatures.
Traditional positional attention maps are used as a Gaussian- prior, as shown Fig. 6 left. But the prior is simply assumed isotropic and fixed size for all objects, leaving their scale information (width and height) ignored. To improve the positional prior, we propose to inject the scale informa- tion into the attention maps.

---

Published as a conference paper at ICLR 2022
The query-to-key similarity in the original positional attention map is computed as the sum of dot products of two coordinate encodings:
(5) where 1/vD is used to rescale the value suggested in Vaswani et al. (2017). We modulate the positional attention maps (before softmax) by dividing the relative anchor width and height from its part and y part separately to smooth the Gaussian prior better match with objects of different scales:
Attn((z, y), (Iret, Bter)) (PB(2) PE(()))) + PE PE(yict))/VD
ModulateAAtn((x:]) v) (Zref, Mrer)) = (PB(x) PE(()ee) Wggref + PE(y)
PE(()re) Mg.sel )/VD, (6)
Wy
hn
where W/ and hg are the width and height of the anchor Ag. and Wayte and hguee are the reference width and height that are calculated by:
Wg,eff ngg = (MLP((C,).
This modulated positional attention helps us extract features objects with different widths and heights, and the visualizations of modulated attentions shown in Fig. 6.
4.5 TEMPERATURE TUNING
For position encoding, we use the sinusoidal function (Vaswani et al., 2017), which is defined as:
I PE(x)2: = sin( (2y/D) PE(x)2--1 cos( T24/D
x
(8)
where T' is hand-design temperature, and the superseript 2i and 2i denote the indices in the encoded vectors. The temperature T in Eq. (8) influences the size of positional priors, as shown in Fig. 7. larger T results in more flattened attention map, and vice versa. Note that the temperature T' is hard-coded in (Vaswani et al., 2017) as 10000 for natural language processing, in which the values of x are integers representing each word's position in sentence. However, in DETR, the values of x are floats between ( and representing bounding box coordinates. Hence different temperature highly desired for vision tasks. In this work, we empirically choose 20 in all our models.
EXPERIMENTS
We provide the training details in Appendix A.
5.1
MAIN RESULTS
Table 2 shows our main results the COCO 2017 validation set. We compare proposed DAB- DETR with DETR (Carion et al., 2020), Faster RCNN (Ren et al., 2017), Anchor DETR (Wang et al., 2021), SMCA (Gao et al., 2021). Deformable DETR (Zhu al., 2021). TSP (Sun et al., 2020), and Conditional DETR (Meng et al., 2021). We showed two variations of our model: standard models and models marked with superscript that have 3 pattern embeddings (Wang al., 2021). Our standard models outperform Conditional DETR with large margin. We notice that our model introduces slight increase of GFLOPs. GFLOPs may differ depending on the calculation scripts and we use the results reported by the authors in Table 2. Actually, we find in our tests that the GFLOPs of our standard models are nearly the same the corresponding Conditional DETR models based on our GFLOPs calculation scripts, hence our model still has advantages over previous work under the same settings. When using pattern embeddings, our DAB-DETR with outperforms previous DETR-like methods on all four backbones with large margin, even better than multi- scale architectures. It verifies the correctness of our analysis and the effectiveness of our design.
5.2 ABLATIONS
Table shows the effectiveness of each component in model. We find that all modules proposed contribute remarkably to our final results. The anchor box formulation improves the per- formance from 44.0% AP to 45.0% AP compared with the anchor point formulation (compare Row
8

---

Published as a conference paper at ICLR 2022
Model
MultiScale
#opochs
AP
APso
APrs
APs
APM
APL
GFLOPs
Params
DETR-R5O
500
12.0
62.1
11.2
20.5
15.5
61.1
86
11M
Faster RCNN-FPN-R50
108
42.0
02.1
45.5
26.6
45.5
53.4
180
42M
Anchor DETR-R50"
42.1
63.1
44.9
22.3
46.2
60.0
39M
Conditional DETR-R50
40.9
61.8
44.6
44M
DAB-DETR-R50
46.7
DAD-DETR-RSS
46.2
DETR- BETRR-DCS.RR DC5-R50
500
43.3
63.1
45.9
22.5
47.3
61.1
187
41M
Deformable DETR-R50
43.8
62.6
47.7
26.4
47.1
58.0
173
40M
SMCA-R50
50
13.7
63.66
17.
24.2
17.0
60.1
152
20M
TSP-RCNN-R50
45.0
64.5
40.6
29.7
47.7
58.0
188
Anchor DETR-DCS-R55
44.2
64.7
47.5
24.7
48.2
60.6
151
39M
Conditional DETR-DC5-R50
43.8
64.4
46.7
24.0
47.6
60.7
195
44M
DAB-DETR-DCS-R50
DAD-DETR-DC5.RS
49.4
216
DETR-R101
500
13.5
63.8
16.1
21.9
18.0
61.8
152
G0M
Faster RONN-FPN-RI0I
108
44.0
63.9
47.8
27.2
48.1
56.0
246
60M
Anchor DETR-RIO1
50
43.5
61.3
16.6
23.2
17.7
61.1
5KM
Conditional DETR-R101
50
42.8
03.7
46.0
21.7
46.6
60.9
156
63M
DAB-DETR-RI01
50
43.5
03.9
46.6
23.6
47.3
61.6
174
63M
DAB-DETR-RIID
44.1
64.7
24.1
48.2
62.9
179
63M
DETR. DITTR-DSS-R011
49.5
253
TSP-RCNN-RIOG
29.9
49.7
59.2
SMCA-RI01
11.1
63.2
18.0
24.3
48.5
61.0
218
50M
Anchor DETR-R101"
45.1
65.7
48.8
25.8
49.4
61.6
58M
Conditional DETR-DCS-RI01
50
45.0
65.5
48.4
26.1
48.9
62.8
262
63M
DAB CAB-DDET--DE DETR-DCS-RIDD DAB-DETR-DCS-RUE
50
45.8
65.9
49.3
27.0
49.8
63.8
282
63M
50
46.6
67.0
50.2
28.1
50.5
64.1
296
63M
Table 2: Results for our DAB-DETR and other detection models. All DETR-like models except
DETR use 300 queries, while DETR uses 100. The models with superscript use pattern embed- dings as in Anchor DETR (Wang et al., 2021). We also provide stronger results of our DAB-DETR in Appendix G and Appendix C.
BRow
Anchoe Rox (4D) vs. Point (2D) Anchor Updaate
orl-Modilatot Atlention
Tespentuue Tuning
AP
45.7
3
45.0
4
2D
46.01
5
4D
44.4
Table 3: Ablation results for DAB-DETR. All models are tested over ResNet-50-DCC. backbone and the other parameters are the same as our default settings.
3 and Row 4) the anchor update introduces 1.7% AP improvement (compare Row 1 and Row 2), which demonstrates the effectiveness of dynamic anchor box design.
After removing modulated attention and temperature tuning, the model performance drops to 45.0% (compare Row and Row 3) and 44.4% (compare Row I and Row 5), respectively. Hence fine- grained tuning of positional attentions is great importance for improving the detection perfor- mance well.
6 CONCLUSION
We have presented in this paper novel query formulation using dynamic anchor boxes for DETR and offered a deeper understanding the role of queries DETR. Using anchor boxes as queries leads to several advantages, including better positional prior with temperature tuning, size- modulated attention to account for objects different scales, and iterative anchor update for im- proving anchor estimate gradually. Such design makes it clear that queries in DETR can be imple- mented as performing soft ROI pooling layer by layer in a cascade manner. Extensive experiments were conducted and effectively confirmed analysis and verified algorithm design.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This work was supported by the National Key Research and Development Program of China (2020AAA0104304,, 2020AAA0106000, 2020AAA0106302), NSFC Projects (Nos. 61620106010, 62061136001 61621136008 136008, 62076147, U19B2034, U181 1461, U19A2081), 1), Beijing NSF Project (No. JQ19016), Beijing Academy of Artificial Intelligence (BAAI), Tsinghua- Alibaba Joint Re- search Program, Tsinghua Institute for Guo Qiang, Tsinghua-OPPO Joint Research Center for Fu- ture Terminal Technology.
We thank all anonymous reviewers for their valuable comments and suggestions, especially the instructive questions from Reviewer 3.
ETHICS STATEMENT
Object detection is fundamental task in computer vision with wide applications. Hence any im- provement of this field will yield lots of impacts. To visually perceive and interact with the envi- romment, autonomous vehicles highly depend on this technique and will benefit from any of its im- provement. It has also led advances in medical imaging, word recognition, instance segmentation on natural images, and so on. Therefore a failure in this model could affect many tasks. Our study provides a deeper understanding of the roles of queries in DETR and improves the interpretability of this important submodule in the end-to-end Transformer-base
detection framework.
As our model relies on deep neural networks, can be attacked by adversarial examples. Similarly, as it relies on training data, it may produce biased results induced from training samples. These are common problems in deep learning and our community working together to improve them. Finally, it is worth noting that detection models, especially face human detection models, might pose a threat to people's privacy and security if used by someone up to no good.
REPRODUCIBILITY STATEMENT
We confirm the reproducibility of the results. We have released the source code on Github at https: /ttps://giaai thub. con/ IDEA-opensoureeeeooo to reproduce our results.
with all materials that are needed
REFERENCES
Alexey Bochkovskiy, Chien- Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 2000200004.094,,,. 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Xiyang Dai, Yinpeng Chen, Jianwei Yang, Pengchuan Zhang. Lu Yuan, and Lei Zhang. Dynamic detr: End-to-end object detection with dynamic attention. Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pp. 2988-2997, October 2021.
Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. arXiv preprint arXiv:2101.07444 2021.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430 2021.
Ross Girshick. Fast r-cnn. In 2015 IEEE International Conference Computer Vision (ICCV), pp- 1440-1448, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):318- 327, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2018.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li. Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. arXiv preprint arXiv:2108.061 2021.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 3966):1177-1149 2017.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. In Pro- ceedings the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 658-666, 2019.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, and Ping Luo. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14454-14463, 2021.
Zhiqing Sun, Shengeao Cao, Yiming Yang, and Kris Kitani. Rethinking transformer- set pre- diction for object detection. arXiv preprint arXiv:201 10181, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9627- 9636, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yingming Wang, Xiangyu Zhang, Tong Yang. and Jian Sun. Anchor detr: Query design for transformer-based detector. arXiv preprint arXiv:2109.07100 2021.
Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: Improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318, 2021.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint arXiv:1904.078: 078500 2019.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR 2021: The Ninth International Conference on Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2022
Appendix for DAB-DETR
A TRAINING DETAILS
Architecture. Our model is almost the same as DETR which includes a CNN backbone, multiple Transformer (Vaswani et al., 2017) encoders and decoders, and two prediction heads for boxes and labels. We use ImageNet-pretrained ResNet (He et al., 2016) as our backbones, and 6 Transformer encoders and 6 Transformer decoders in our implementations. We follow previous works report results over four backbones: ResNet-50, ResNet- and their 16x-resolution extensions ResNet- 50-DC5 and ResNet-101-DCC MLP networks for box and label predictions share the same parameters across different decoder layers. As inspired by Anchor DETR, we also leverage multiple patter embeddings to perform multiple predictions at one position and the number of patterns is set as 3 which is the same as Anchor DETR. We also leverage PReL.U (He et al., 2015) as our activations.
As we need to predict boxes and labels in each decoder layer, the
Following Deformable DETR and Conditional DETR, we use 300 anchors queries. We select 300 predicted boxes and labels with the largest classification logits for evaluation as well. We also use focal loss (Lin et al., 2020) with bipartite matching and final loss calculating. but with different coefficients. Classification loss with coefficient 2.0 is used in bipartite matching but 1.0 in the final loss. L.1 loss with coefficient 5.0 and GIOU loss (Rezatofighi et al., 2019) with coefficient 2.0 are consistent in both the matching and the final loss calculation procedures. All models are trained on 16 GPUs with I image per GPU and AdamW (Loshchilov & Hutter, 2018) is used for training with weight decay 10-4. The learning rates for backbone and other modules are set to 10-5 and 10-4, respectively. We train our models for 50 epochs and drop the learning rate by 0.1 after 40 epochs. All models are trained on Nvidia A100 GPU. We search hyperparameters with batch size 64 and all results in our paper are reported with batch size 16. For better reproducing our results, we provide the memory needed and batch size/GPU in Table 4.
0.25, 7 = 2 for classification. The same loss terms are used in
Dataset. We conduct the experiments on the COCO (Lin et al., 2014) object detection dataset. All models are trained on the train2017 split and evaluated on the va12017 split.
Model
Batch Size/GPU
GPU Memory (MB)
DAB-DETR-R50
6527
DAB- MAB.DETR.RPE R50*
3573
DAB-DETR-RSO-DCCS
13745
MAR-DETR-RS0-DES
DAB-DETR-RE DETR-RIO!
6913
RAB-DETR-RIDE
4369
MAN-DETR-RIOI-
13148
DAB DETR--ROI--DS
16744
Table 4: GPU memory usage of each model.
B COMPARISON OF DETR-LIKE MODELS
In this section, we provide more detailed comparison of DETR-like models, including DETR (Car- ion et al., 2020), Conditional DETR (Meng et al., 2021), Anchor DETR (Wang et al., 2021), De- formable DETR (Zhu al., 2021),, our proposed DAB-DETR, and DAB-DeformableDEETR Their model designs are illustrated in Fig. 8. We will discuss the difference between previous models and our models.
Anchor DETR (Wang et al., 2021) improves DETR by introducing 2D anchor points, which are updated layer by layer. shares a similar motivation with our work. But it leaves the object scale information unconsidered and thus cannot modulate the cross-attention to make it adapt to objects of different scales. Moreover, the positional queries in its framework are of high dimension and passed to the self-attention modules in all layers without any adaptation. See the brown-colored part in Fig. 8 (d) for details. This design might be sub-optimal as the attentioo modules cannot leverage the refined anchor points in different layers.
12

---

Published as a conference paper at ICLR 2022
Deformable DETR (Zhu et al., 2021) introduces 4D anchor boxes and updates them layer by layer, which is called iterative bounding box refinement in its paper. Its algorithm is mainly developed based on deformable attention, which requires reference points to sample attention points and mean- while utilizes box width and height to modulate attention areas. However, as iterative bounding box refinement is closely coupled with the special design of deformable attention, it is nontrivial to apply it to general Transformer decoder-based DETR models. This is probably the reason why few works after Deformable DETR adopt this idea. Moreover, the position queries in Deformable DETR are passed to both the self-attention modules and the cross-attention modules in all layers without any adaptation. See the coon-cclooeed part in Fig. 8 (e) for details. As a result, both its self attentonn modules and cross-attention modules cannot fully leverage the refined anchor boxes in different layers.
To verify our analysis, we develop variant of Deformable-DETT by formulating its queries as which is illustrated in Fig. 8 (f). Under exactly the same setting using R50 as the backbone, DAB- Deformable-DETR improves Deformable-Der
dynamic anchor boxes as in DAB-DETR. We call this variant as DAB--eformable-DEF/T
by 0.5 AP (46.3 to 46.8) on COCO. See Table 5 for the performance comparison and Sec. C for more implementation details.
Dynamic DETR (Dai et al., 2021) is another interesting improvement of DETR. It also leverages anchor boxes to pool features, but it uses ROI pooling for feature extraction, which makes it less general to DETR-like models compared with our dynamic anchor boxes. Moreover, compared with cross-attention in Transformer decoders, which performs global feature pooling in soft manner (based on attention maps), the ROI pooling operation only performs local feature pooling within a ROI window. In our opinion, the ROI pooling operation can help faster convergence enforces each query to associate with a specific spatial position. But it may lead to a sub-optimal result due its ignorance of the global context outside ROI window.
C DAB-DEFORMABLE-DETT
To further demonstrate the effectiveness of our dynamic anchor boxes, we develop DAB- Deformable-DETR by adding our dynamic anchor boxes design to Deformable DETR (Zhu et al., 2021) 2, The difference between Deformable DETR and BAA--Deorrmabe-DDe DETR is shown in Fig. 8 (c) and (f). The results of Deformable DETR and DAB- Dewooeable--eeeeeeeeee are shown in Table 5. With no more than 10 lines of code modified, our DAB-Deformabbe-DEr
(row 4) results in a significant performance improvement (+0.5 AP) compared with the original Deformable DETR (row3) 3). All other settings except the query formulation are exactly the same in this experiment.
We also compare the speed of convergence in Fig. 9. It shows that our proposed dynamic anchor boxes speed up the training as well (left Fig. 9). We believe one of the reasons for better per- formance is the update of learned queries. We plot the change of total loss, which is the sum-up of losses of all decoder layers, during training in the middle figure Fig. 9. Interestingly. shows is larger than Deformable DETR. However, the loss is lower than that in Deformable DETR (right in Fig.
that the total loss of BAB--eformable-- of the final layer of 9), which is good indicator of the better performance of DAB-Deformable- DETR the inference result only takes from the last layer.
D ANCHORS VISUALIZATION
We visualize the learned anchor boxes in Fig. 10. When learning anchor points as queries, the learned points are distributed evenly around the image, while the centers seem distribute randomly when learning anchor boxes directly. This might be because the centers are coupled with anchor sizes. The right-most figure shows the visualization of the learned anchor boxes. We only show partial set for visualization clarity. Most boxes are of medium size and no particular pattern is found the distribution of boxes.
con/ We used the open-source implementation from htttp:://githiii http:///...oooo/oooneeooaaea fundamenta vi sion/
Deformable e-mmaall---eeeeeee DETR
13

---

Published as a conference paper at ICLR 2022
Layer
Creap Alarae
Attentnntton
Sef-Attestin
SaV-Amention
Layer
Svar
Layer
Image Festures
Cross Astention
Artesinttttioon
Crean Klaykr
Positional Emibectting
SelfAttertion
SeFAttenia
Sef-Amenion
Learnable
anchor
(a) DETR
Lesmable quaries
(b) Conditional DETR
Leamable
(c) DAB-DETR (Ours)
boves
Layer?
ColeColam
Deformable Arte 1on
Setomstedterte
SoltAitetionn
Self Atteetion
Self Aize-tttiionn
Layer
Layer
Deloraaide AlBrSSnn
Detormaa Alartor
Sel-Anertiio
Self-Attatton
Sel-Attantin
queries for
Leamed
Leamable
Learnalle anchor
layers
quaries
boxes
(d) DER
(e) Deformable DETR
If) DeB---Doaabbe-DEEE (Ours)
Figure 8: Comparison of DETR-like models. For elarity, only show two layers of Transformer decoder and omit the FFN blocks. We mark the modules with difference in purple and marked the learned -indddmiiiiinlll queries in brown. DAB-DETR (c) is proposed in our paper, and DAB- Deformable-DEr -DERR (f) is a variant of Deformable DETR modified by introducing our dynamic an- chors boxes. All previous models (a,b,d,e) leverage high-dimensional queries (shaded in brown) to pass positional information to each layers, which are semantic ambiguous and are not updated layer by layer. In contrast, DAB-DETR (c) directly uses dynamically updated anchor boxes to provide both reference query point (xc y) and reference anchor size (w, h)) to improve the cross-altention computation. DAB-Deformable-DETT (f) uses dynamically updated anchor boxes to formulate its queries as well.
14

---

Published as a conference paper at ICLR 2022
# row
Model
AP
AP50
AP75
APs
APM
APr.
Params
Deformable DETR
43.8
62.6
47.7
26.4
47.1
58.0
40M
2
Deformable DETR+
45.4
64.7
49.0
26.8
48.3
61.7
40M
3
Deformable + (open source)
46.3
65.3
50.2
28.6
49.3
62.1
47M
4
DAB-Deformable DEETR(OUs)))
46.8
66.0
50.4
29.1
49.8
62.3
47M
Table Comparison of the results of Deformable DETR and DAB- HAB--Deforabbl-DETT The models in row and row 2 are copied from the original paper, and the models in row 3 and row are tested under the same standard R50 multi-scale setting. Deformable DETR+ means the Deformable DETR model with iterative bounding box refinement and the result of Deformable DETR+ (open source) reported by us using the open-source code. The only difference between row and row formulation of queries.
CETRITER
Deleerooe DETRITOR
0.45
DAs Dei-Ceforrree OS
040-
DVA www.oaaaaeeeeeeeeo kild
0.8-
Teaus of the lank
020
8-
0.5
Dnfernndie BFTA
20
30
20
30
20
30
opozh
opach
opach
Figure 9: Comparison of the training of Deformable DETR and DABB-Defomable-DETT models. We plot the change of AP (left), the loss of all layers (middle), and the loss of the last layer (right) during training. respectively. With more than 10 lines of code modified, results in a better performance compared with the original Deformable DETR model (see the left figure). While the loss of all layers MAB--eformable-DDe DETR (see the middle figure), our models have a lower loss of the last layer (see the right figure), which the most important as inference result only takes from the last layer. The two models are tested under the same standard R50 multi-scale setting.
larger than that in Deformable
Figure 10: Learned anchor points when learning 2D coordinates only (left), and anchor center points (middle) and partial anchor boxes (right) when learning anchor boxes directly.
E RESULTS WITH DIFFERENT TEMPERATURES
Table 6 shows the results of models using different temperatures in the positional encoding function. As larger temperature generates more flattened attention maps, leads to better performances for larger objects. For example, the model with T = 2 and the model with T' 10000 have similar AP results, but the former has better performances on APs and APMs while the latter works better on AP, L. which also validates the role of positional priors in DETR.
15

---

Published as a conference paper at ICLR 2022
Temperature
AP
APso
AP
APs
APM
APt.
2
39.6
60.7
41.9
19.3
43.3
58.0
40.0
42.1
19.5
43.4
58.9
10
40.0
61.1
42.3
19.7
43.5
59.3
20
40.1
61.1
42.8
19.8
43.7
58.6
50
39.8
61.0
42.2
19.7
43.2
58.8
100
39.8
60.8
42.1
19.3
43.3
58.4
10000
39.5
60.7
41.7
18.9
42.6
58.9
Table 6: Comparison of models with different temperatures. All models trained with the ResNet- 50 backbone, batch size 64, no multiple pattern embeddings, and no modulated attentions. Default Settings are used for the rest of the parameters.
F RESULTS WITH LESS DECODER LAYERS
Table 7 shows the results of models with different decoder layers. All models are trained under our standard ResNet-50-DD setting except the number of decoder layers.
decoder layers
GFLOPs
Parmas
AP
AP50
AP75
APs
AP
APr
2
202
36M
40.2
59.0
42.9
22.2
43.5
55.4
3
206
38M
43.9
63.4
47.4
24.6
47.8
60.5
4
210
40M
44.9
64.5
48.2
25.9
48.5
61.0
5
213
42M
45.2
65.5
48.6
26.6
48.9
62.3
6
216
44M
45.7
66.2
49.0
26.1
49.4
63.1
Table 7: Comparison of models with different number of decoder layers. All models are trained under our standard ResNet-50-DC setting except the number of decoder layers.
G FIXED x, FOR BETTER PERFORMANCE
We provide in this section an interesting experiment. As we all know, box coordinates x. y, h, w are learned from data. When we fix x, y of the anchor boxes with the random initialization, the model's performance increases consistently. The comparison
standard DAB-DETR and DAB- DETR with fixed æ, y coordinates is shown in Table 8. Note that we only fix ø. y at the first layer to prevent them from learning information from data. But x, y will be updated other layers. We conjecture that the randomly initialized and fixed v, y coordinates help avoid overfitting, which may account for this phenomenon.
H COMPARISON OF Box UPDATE
To further demonstrate the effectiveness of our dynamic anchor box design, we plot the layer-by- layer update result of boxes of DAB-DETR and Conditional DETR in Fig. 11. All DETR-IBE like models have a stacked layers structure. Hence the outputs of each layer can be viewed as a refining procedure. However, due to the high-dimensional queries that are shared across all layers, the update of queries between layers is not stable. As shaded in yellow Fig. (b), some boxes predicted in the latter layers are worse than their previous layers.
I ANALYSIS OF FAILURE CASES
Fig. 12 presents some samples where our model does not predict well. We find our model may have some troubles when facing dense objects, very small objects, very large objects in image. To
16

---

Published as a conference paper at ICLR 2022
Model
AP
APs0
AP75
APs
APM
APz
DAB-DETR-REl
42.6
63.2
45.6
21.8
46.2
61.1
DAB-DETR R50" -ixeddddy
42.9(++0.3)
63.7
45.3
22.0
46.8
60.9
DAB-DETR BEE-DETR-DCS---RE
44.5
65.1
47.7
25.3
48.2
62.3
44.7(+-0.2)
65.3
47.9
24.9
48.2
62.0
DAB-DETR-DCS-R550
45.7
66.2
49.0
26.1
49.4
63.1
DC5--500 fixedx&y -ixeeRyo
45.8(++0.1)
66.5
48.9
26.4
49.6
62.7
DAB-DETR-RI01*
44.1
64.7
47.2
24.1
48.2
62.9
DAB-DETR-RII -IIRRR R -fieedxxx
44.8(+-0.7)
65.4
48.2
25.1
48.9
63.1
PAB-DETR-DC5-RIIE
46.6
67.0
50.2
28.1
50.5
64.1
DEC-R--------I "1001"-fiixee -fieedr&
46.7(
67.3
50.7
27.3
50.9
64.1
Table 8: Comparison of DAB-DETR and DAB-DETR with fixed anchor centers x; y- When fixing of queries with random values, the performance of the models is improved consistently. The models with superscript 3 pattern embeddings as Anchor DETR.
layer3
layer4
layer3
layer4
layer2
layer3
layer2
layer3
layer4
layer5
layer4
layer5
a) DAB-DETR
Conditional DETR
Figure 11: We compare the layer-by-layer update of boxes of DAB- DAB-DETR (a) and Conditional DETR (b). The green boxes are ground truth annotations while the red boxes are model predictions. The boxes of Conditional DETR have larger variances and we mark some boundaries of boxes with large change in yellow.
improve the performance of our model, we will introduce multi-scale technique into our model to improve the detection performance on small and large objects.
COMPARISON OF RUNTIME
We compare the runtime of DETR, Conditional DETR, and our proposed DAB-DETR in Table
9. Their runtime speeds reported single Nvidia A100 GPU. Our DAB-DETR has simi- lar inference speed but better performance compared with Conditional DETR, which is our direct competitor.
17

---

Published as a conference paper at ICLR 2022
(a)
(b)
(c)
(d)
(e)
Figure 12: We visualize some images where our model does not predict well, including dense ob- jects (a.b,c), very small objects (d), and very large objects (c.f). The green boxes ground truth annotations while red boxes are predictions models.
Model
timo(s/img)
epecbes
AP
APso
APrs
APs
AP
APt.
Parmas
DETR-R50
0,018
500
12.0
62.4
41.2
20.5
45.8
61.1
41M
Conditional DETR-R50
0.057
40.9
61.8
43.3
20.8
44.6
59.2
44M
DAB-DETR-R50
DETR-R101
0.074
500
43.5
63.8
46.4
21.9
48.0
61.8
60M
Conditional DETR-RIO1
0.082
50
42.8
63.7
46.0
21.7
46.6
60.9
63M
DAB-DETR-RIO1
0.085
50
43.5
63.9
46.6
23.6
47.3
61.5
63M
Table 9: Comparison of the runtime DETR, Conditional DETR, and our proposed DAB-DETR. All speeds reported single Nvidia A100 GPU.
K COMPARISON OF MODEL. CONVERGENCE
We present convergence curves of DETR, Conditional DETR, and our DAB-DETR in Fig. 13. All models are trained under the standard R50 (DC5) setting. The results demonstrate the effectiveness of model. Our DAB-DETR trained with our ʃ fix r&y variants. Appendix G for more details about the fix r&y results. Both Conditional DETR and DAB-DETR use 300 queries, while DETR leverages 100 queries.
Our DAB-DETR converges faster than Conditional DETR, especially in early epochs, as shown in Fig. 13.
L VISUALIZATION RESULTS OF ITERATIVE Box UPDATE
We present more visualization results of iterative box update in Fig. 14 and Fig. 15, The initial anchors, anchors updated after the first decoder layer, and the anchors predicted from the last decoder layer are plotted in the first, the second, and the third columns, respectively.
18

---

Published as a conference paper at ICLR 2022
44.7(Ours)
0.440
43.8
3
0.400
0.375
MAP
0.350
0.300
DETR(DC5)
Conditional DETR(DCS) DAB-DETR(DCS)Ours)
0.250
4050
90
390
500
Epoch
Convergence curves of DETR, Conditional DETR, and our DAB-DETR, All models are trained under the R50 (DC5) setting.
Figure 13:
Figure 14: Visualizations for (left), anchor boxes after the first decoder layer (middle), and the output of the last decoder layer (right), respectively. The green boxes are ground truth annotations, while the red boxes are predic- tions of our model. The results obtained using the ResNet-50 backbone. More visualizations available in Fig. 15.
anchor box update. We plot the initial anchor boxes
19

---

Published as a conference paper at ICLR 2022
Figure 15: More visualizations for layer- anchor box update. We plot the initial anchor boxes (left), anchor boxes after the first decoder layer (middle), and the output of the last decoder layer (right), respectively. The green boxes ground truth annotations, while the red boxes predictions of our model. The results are obtained using the ResNet-50 backbone.
20


Reviews:
[{"review_id": "b6-3KvrTa14", "reviewer": "Reviewer_iJ5R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Main strengths:\n\n1) Providing detailed diagrams highlighting the differences between the proposed method and the baselines\n2) Performing an experiment explaining to rebut some potential doubts about sources of improvements\n3) Visualizing the attention between positional queries and positional keys, justifying the limitations of the previous works and their proposed method's advantage\n4) Improving benchmark results over SOTA\n\n## Main Weaknesses:\n\n5) Based on my understanding, the experiment visualized in figure 3, aims to justify that something is wrong with the way positional information is represented and it is not merely an issue with the optimization. The authors should generate training curves of a similar experiment, after making the proposed changes too. It would much better reveal the impact of their changes in the discussed matter and clear the doubt about the reasoning of the experiment, knowing how using \"well-trained\" queries of their proposed method compare to training them from scratch. furthermore, comparing the described training curves to those in figure 3 would much better illustrate the improvement in rate of convergence which is the main issue targeted by the proposed method. Drawing these curves as a result of multiple runs with an average curve and some measure of confidence could further support the claims based on these types of experiments.\n6) Hyperparameter study is limited to temperature and number of decoder layers. Effects of other hyperparameter choices are not clear to the reader. Even if their values are copied from the baseline methods, the proposed changes in the method could alter their behavior. Also, it seems like the provided ablations are compared using the the hyperparameter choices that work best with the proposed configuration. This could mean that those choices are biased towards the proposed method in the ablations. The hyperparameters for each individual configuration should be chosen independent of the others.\n7) While Deformable DETR (Zhu et al., 2021) is discussed in the introduction as the last paper in this line of research, its results are not compared in Table 2 and I am not sure the proposed method is necessarily better in terms of AP, and comparative in terms of GFLOPs and number of Parameters.\n8) Some of the compared methods had improved results with increased number of training epochs and other configuration of architectures such as backbones. It is not clear how would the proposed method compare to the SOTA in such settings.\n9) Qualitative results are quite limited in the paper. From the only provided figure (9), the main visible difference in the last two columns is how the books in the second row of images are detected. A comparison between the proposed method and the baselines highlighting a few examples that contribute to the improvements of the quantitative results could improve this aspect of the paper. Knowing the SOTA on this dataset is still under 50% in AP, visualizing some failure cases and trying to explain why the method failed could further improve the work.\n\n#### Minor points:\n10) There is also a paper called \" Dynamic DETR: End-to-End Object Detection With Dynamic Attention\" from ICCV'21 which better be included in the literature review and compared experiments of this work.\n11) Knowing the amount of GPU memory used could be important for reproducing the results.\n12) Only releasing the code would not suffice to support the statement on reproducibility. The mentioned points on hyperparameter studies and providing statistical analysis on the results rather than results from a single run, and providing details on the computational infrastructure used could further support this statement.", "summary_of_the_paper": "The paper addresses the problem of slow convergence in training of object detection transformer. The main contribution is to use trainable anchor boxes (4D) in a layer-by-layer manner. They also motivate their contribution by visualizing positional attentions and explaining the advantage of their proposed method in adapting to scale ratio of objects. They further propose to change the size of the positional attention by an additional temperature parameter, which they choose empirically for the dataset. They perform experiments to justify aspects of their claims, and provide ablations to prove their results being improved over the SOTA on the COCO '17 validation set.", "summary_of_the_review": "The paper seems to be a step in the right direction to address the issues with detection transformers. The provided figures of architectures are well illustrative and helps readers to understand the differences. The performed experiments to some extent justify the decisions made in the design choices. However, there are missing complementary experiments that I feel are required to fully explain the results. The empirical approaches are lacking as also described in the main review which damages the credibility of some of the claims.\nTherefore, I am leaning towards a reject for this version of the work. But I do feel like the paper has the potential to be a great submission in a future revision, assuming the points of concerns are covered with additional experiments. I have provided more details on how to improve the paper. I look forward to see how the authors respond and address my concerns before I finalize my decision.\n\nEdit: I thank the authors for responding to my comments in details. \nI am glad to see that the authors found some of my comments useful and took steps to further improve their work.\nSome of my main concerns are resolved (5 and 9), while some other concerns are only partially addressed and they still remain a concern in my opinion (6, 7, and 8).\n\n- The authors have not provided even a hint of performing a somewhat systematic/objective hyperparameter search to convince the readers that the hyperparameters are not biased towards the proposed method in the ablation studies such as in table 3.\n- Some improved variants of the Deformable DETR paper including the two-stage Deformable DETR and results from the table 3 in that paper, seem to be still missing with appropriate comparisons in the updated draft.\n- The authors have provided new results using a couple of alternative architectures using their method which shows potential for further improvements. However, these are not provided side-by-side in comparison to other methods, So that the proposed method can be judged on its capacity for improvement compared to the alternatives discussed in the paper. I do understand that such experiments are time consuming, however, the paper having taken such a comparative approach to motivate its contributions, needs such thorough empirical comparisons to prove its claims on improvements.\n- It seems that the concerns of reviewer zkuH regarding contributions being marginally significant or novel, are not sufficiently resolved.\n\nTypos I noticed in the updated draft:\n- \"Some previous work also has similar analysis and confirmed this.\": \"work\" --> \"works\"\n- \"The modulated attentions can be regarded as helping perform soft ROI pooling.\": \"perform\" --> \"to perform\"\n\nOverall, the updated draft is improved and so I increase the recommendation score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZxmGMV1G_N", "reviewer": "Reviewer_H1Mp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ This paper is well organized, with clear comparison to prior works in tables and graphs, good problem analysis on the role of queries in DETR and conditional DETR, and informative figures.\n+ Experiments demonstrate the advantage of dynamic anchor queries over previous implicit and explicit query formulations including original DETR, conditional DETR and anchor DETR. \n+ The proposed query formulation introduces marginal computational overhead compared to other formulations in terms of GFLOPs.\n\nWeaknesses:\n- It seems the proposed query formulation works well with the Deformable Convolution Network (DCN) backbone but less competitive without the DCN backbone. For example, DAB-DETR-DC5-R50 outperforms Anchor DETR-DC5-R50 by 1.5 AP but DAB-DETR- R50 outperforms Anchor DETR-R50 by only 0.5 AP. In the ablation study, all experiments are conducted with the DCN. It would be more comprehensive to provide analysis or insights on why dynamic anchor queries cannot offer significant improvements without DCN.\n- GFLOPs is a partial indicator of computational efficiency. The authors might want to report the actual runtime speed (ms per image).\n- A minor reminder on citation correctness, e.g. Section 4.3 2nd line DETR should not cite Meng et al., 2021.\n", "summary_of_the_paper": "This paper introduces dynamic anchor boxes as a query formulation for detection transformer (DETR). The basic idea is to use explicit positional priors and scale priors from anchor boxes as the queries for training the decoder of DETR. Anchor boxes are also adjusted layer-by-layer to learn the optimal anchor setting. The proposed query formulation provides a better understanding of the query for vision transformer and is proven to outperform previous implicit and explicit query formulations.", "summary_of_the_review": "This paper has the merits of good performance, clear presentation, and a neat idea. Although more analysis and experiments could further improve the manuscript, overall this is a good paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qj-onU0a3BK", "reviewer": "Reviewer_zkuH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\n\n- This paper provides a systematic comparison between several detection transformer variants, from the perspective of positional encoding and attention mechanism, as summarized in Table 1. \n- The experiment results are promising, and DAB-DETR achieves state-of-the-art performance on COCO 2017 dataset.\n\n### Cons\n\n- The novelty of this paper is very limited, the idea of using explicit positional priors to accelerate training and improve explainability has already been adopted in Deformable DETR, Conditional DETR, and Anchor DETR.\n- This is incremental work. Almost every part of the method can be found in existing work. For example, the iterative update of 4-D anchor boxes resembles the update of reference points in Deformable DETR, the decoupling of the content and positional queries resembles Conditional DETR, and the explicit anchor design resembles Anchor DETR. This paper only makes a combination of those methods, while offering little novel insights.\n- The observations in Section 3 are almost obvious, and the reviewer did not feel that new information was brought to the table after reading.\n- The conclusion in Section 3 is somewhat weak, as the authors described \"the multiple mode property of queries in DETR **is likely** the root cause for its slow training\". And the method is proposed based on this somewhat invalid conclusion, which makes the motivation less clear, and the method less convincing. Moreover, the observations in Figure 4 are not supported by quantitative experiments, and it is difficult to give conclusions from the qualitative visualization of 4 out of hundreds of anchors.", "summary_of_the_paper": "This paper proposes DAB-DETR, a DETR variant that incorporates dynamic anchor boxes for object queries. Based on the analysis of the learned queries in DETR, the authors conclude the slow convergence of DETR is likely caused by the multiple mode property of the queries. To solve the problem, explicit positional priors (box coordinates) are used to improve query-to-feature similarity and accelerate model convergence. Experiments and analysis show that DAB-DETR achieves promising results on the object detection task.", "summary_of_the_review": "This paper proposes a DETR variant that incorporates dynamic anchor boxes to accelerate model convergence. Though promising result has been achieved, the method lacks novelty and the motivation is not very clear. Thus, the paper is not ready for publication. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Somewhat borderline paper given the scores, but leaning on the side of accepting mostly because the positive (and weak positive) reviews are a little more persuasive. The negative review is a bit of an outlier; the main issues raised in the negative review are that the novelty is on the lower side or otherwise that the work is incremental. These complaints are largely not shared by the other reviewers, and furthermore seem not like deal-breakers. Still a borderline paper, but fairly safe to accept.
Decision: Accept (Poster)