DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR

1 INTRODUCTION . Object detection is a fundamental task in computer vision of wide applications . Most classical detectors are based on convolutional architectures which have made remarkable progress in the last decade ( Ren et al. , 2017 ; Girshick , 2015 ; Redmon et al. , 2016 ; Bochkovskiy et al. , 2020 ; Ge et al. , 2021 ) . Recently , Carion et al . ( 2020 ) proposed a Transformer-based end-to-end detector named DETR ( DEtection TRansformer ) , which eliminates the need for hand-designed components , e.g. , anchors , and shows promising performance compared with modern anchor-based detectors such as Faster RCNN ( Ren et al. , 2017 ) . In contrast to anchor-based detectors , DETR models object detection as a set prediction problem and uses 100 learnable queries to probe and pool features from images , which makes predictions without the need of using non-maximum suppression . However , due to its ineffective design and use of queries , DETR suffers from significantly slow training convergence , usually requiring 500 epochs to achieve a good performance . To address this issue , many follow-up works attempted to improve the design of DETR queries for both faster training convergence and better performance ( Zhu et al. , 2021 ; Gao et al. , 2021 ; Meng et al. , 2021 ; Wang et al. , 2021 ) . Despite all the progress , the role of the learned queries in DETR is still not fully understood or utilized . While most previous attempts make each query in DETR more explicitly associated with one specific spatial position rather than multiple positions , the technical solutions are largely different . For example , Conditional DETR learns a conditional spatial query by adapting a query based on its content feature for better matching with image features ( Meng et al. , 2021 ) . Efficient DETR introduces a dense prediction module to select top-K object queries ( Yao et al. , 2021 ) and Anchor DETR formulates queries as 2D anchor points ( Wang et al. , 2021 ) , both associating each query with a specific spatial position . Similarly , Deformable DETR directly treats 2D reference points as queries and performs deformable cross-attention operation at each reference points ( Zhu et al. , 2021 ) . But all the above works only leverage 2D positions as anchor points without considering of the object scales . Motivated by these studies , we take a closer look at the cross-attention module in Transformer decoder and propose to use anchor boxes , i.e . 4D box coordinates ( x , y , w , h ) , as queries in DETR and update them layer-by-layer . This new query formulation introduce better spatial priors for the cross-attention module by considering both the position and size of each anchor box , which also leads to a much simpler implementation and a deeper understanding of the role of queries in DETR . The key insight behind this formulation is that each query in DETR is formed by two parts : a content part ( decoder self-attention output ) and a positional part ( e.g . learnable queries in DETR ) 1 . The cross-attention weights are computed by comparing a query with a set of keys which consists of two parts as a content part ( encoded image feature ) and a positional part ( positional embedding ) . Thus , queries in Transformer decoder can be interpreted as pooling features from a feature map based on the query-to-feature similarity measure , which considers both the content and positional information . While the content similarity is for pooling semantically related features , the positional similarity is to provide a positional constraint for pooling features around the query position . This attention computing mechanism motivates us to formulate queries as anchor boxes as illustrated in Fig . 1 ( c ) , allowing us to use the center position ( x , y ) of an anchor box to pool features around the center and use the anchor box size ( w , h ) to modulate the cross-attention map , adapting it to anchor box size . In addition , because of the use of coordinates as queries , anchor boxes can be updated dynamically layer-by-layer . In this way , queries in DETR can be implemented as performing soft ROI pooling layer-by-layer in a cascade way . We provide a better positional prior for pooling features by using anchor box size to modulate the cross-attention . Because the cross-attention can pool features from the whole feature map , it is crucial to provide a proper positional prior for each query to let the cross-attention module focus on a local region corresponding to a target object . It can also facilitate to speed up the training convergence of DETR . Most prior works improve DETR by associating each query with a specific location , but they assume an isotropic Gaussian positional prior of a fixed size , which is inappropriate for objects of different scales . With the size information ( w , h ) available in each query anchor box , we can modulate the Gaussian positional prior as an oval shape . More specifically , we divide the width and height from the cross-attention weight ( before softmax ) for its x part and y part separately , which helps the Gaussian prior to better match with objects of different scales . To further improve the positional prior , we also introduce a temperature parameter to tune the flatness of positional attention , which has been overlooked in all prior works . 1See the DETR implementation at https : //github.com/facebookresearch/detr . The components of queries and keys are also shown in each subplot of Fig . 1 . Note that the learnable queries in DETR are only for the positional part . Related discussion can also be found in Conditional DETR ( Meng et al. , 2021 ) . In summary , our proposed DAB-DETR ( Dynamic Anchor Box DETR ) presents a novel query formulation by directly learning anchors as queries . This formulation offers a deeper understanding of the role of queries , allowing us to use anchor size to modulate the positional cross-attention map in Transformer decoders and perform dynamic anchor update layer-by-layer . Our results demonstrate that DAB-DETR attains the best performance among DETR-like architectures under the same setting on the COCO object detection benchmark . The proposed method can achieve 45.7 % AP when using a single ResNet-50 ( He et al. , 2016 ) model as backbone for training 50 epochs . We also conducted extensive experiments to confirm our analysis and verify the effectiveness of our methods . 2 RELATED WORK . Most classical detectors are anchor-based , using either anchor boxes ( Ren et al. , 2017 ; Girshick , 2015 ; Sun et al. , 2021 ) or anchor points ( Tian et al. , 2019 ; Zhou et al. , 2019 ) . In contrast , DETR ( Carion et al. , 2020 ) is a fully anchor-free detector using a set of learnable vectors as queries . Many follow-up works attempted to solve the slow convergence of DETR from different perspectives . Sun et al . ( 2020 ) pointed out that the cause of slow training of DETR is due to the crossattention in decoders and hence proposed an encoder-only model . Gao et al . ( 2021 ) instead introduced a Gaussian prior to regulate the cross-attention . Despite their improved performance , they did not give a proper explanation of the slow training and the roles of queries in DETR . Another direction to improve DETR , which is more relevant to our work , is towards a deeper understanding of the role of queries in DETR . As the learnable queries in DETR are used to provide positional constrains for feature pooling , most related works attempted to make each query in DETR more explicitly related to a specific spatial position rather than multiple position modes in the vanilla DETR . For example , Deformable DETR ( Zhu et al. , 2021 ) directly treats 2D reference points as queries and predicts deformable sampling points for each reference point to perform deformable cross-attention operation . Conditional DETR ( Meng et al. , 2021 ) decouples the attention formulation and generates positional queries based on reference coordinates . Efficient DETR ( Yao et al. , 2021 ) introduces a dense prediction module to select top-K positions as object queries . Although these works connect queries with positional information , they do not have an explicit formulation to use anchors . Different from the hypothesis in prior works that the learnable query vectors contain box coordinate information , our approach is based on a new perspective that all information contained in queries are box coordinates . That is , anchor boxes are better queries for DETR . A concurrent work Anchor DETR ( Wang et al. , 2021 ) also suggests learning anchor points directly , while it ignores the anchor width and height information as in other prior works . Besides DETR , Sun et al . ( 2021 ) proposed a sparse detector by learning boxes directly , which shares a similar anchor formulation with us , but it discards the Transformer structure and leverages hard ROI align for feature extraction . Table 1 summarizes the key differences between related works and our proposed DAB-DETR . We compare our model with related works on five dimensions : if the model directly learns anchors , if the model predicts reference coordinates ( in its intermediate stage ) , if the model updates the reference anchors layer-by-layer , if the model uses the standard dense cross-attention , if the attention is modulated to better match with objects of different scales . and if the model updates the learned queries layer-bylayer . A more detailed comparison of DETR-like models is available in Sec . B of Appendix . We recommend this section for readers who have confusions about the table . 3 WHY A POSITIONAL PRIOR COULD SPEEDUP TRAINING ? . Much work has been done to accelerate the training convergence speed of DETR , while lacking a unified understanding of why their methods work . Sun et al . ( 2020 ) showed that the cross-attention module is mainly responsible for the slow convergence , but they simply removed the decoders for faster training . We follow their analysis to find which sub-module in the cross-attention affects the performance . Comparing the self-attention module in encoders with the cross-attention module in decoders , we find the key difference between their inputs comes from the queries , as shown in Fig . 2 . As the decoder embeddings are updated by linearly combining the image features , they are in the same space as the image features . Hence the root cause is likely due to the learnable queries . Two possible reasons in cross-attention account for the model ’ s slow training convergence : 1 ) it is hard to learn the queries due to the optimization challenge , and 2 ) the positional information in the learned queries is not encoded in the same way as the sinusoidal positional encoding used for image features . To see if it is the first reason , we reuse the well-learned queries from DETR ( keep them fixed ) and only train the other modules . The training curves in Fig . 3 ( a ) show that the fixed queries only slightly improve the convergence in very early epochs , e.g . the first 25 epochs . Hence the query learning ( or optimization ) is likely not the key concern . Then we turn to the second possibility and try to find out if the learned queries have some undesirable properties . As the learned queries are used to filter objects in certain regions , we visualize a few positional attention maps between the learned queries and the positional embeddings of image features in Fig . 4 ( a ) . Each query can be regarded as a positional prior to let decoders focus on a region of interest . Although they serve as a positional constraint , they also carry undesirable properties : multiple modes and nearly uniform attention weights . For example , the two attention maps at the top of Fig . 4 ( a ) have two or more concentration centers , making it hard to locate objects when multiple objects exist in an image . The bottom maps of Fig . 4 ( a ) focus on areas that are either too large or too small , and hence can not inject useful positional information into the procedure of feature extraction . We conjecture that the multiple mode property of queries in DETR is likely the root cause for its slow training and we believe introducing explicit positional priors to constrain queries on a local region is desirable for training . To verify this assumption , we replace the query formulation in DETR with dynamic anchor boxes , which can enforce each query to focus on a specific area , and name this model DETR+DAB . The training curves in Fig . 3 ( b ) show that DETR+DAB leads to a much better performance compared with DETR , in terms of both detection AP and training/testing loss . Note that the only difference between DETR and DETR+DAB is the formulation of queries and no other techniques like 300 queries or focal loss are introduced . It shows that after addressing the multi-mode issue of DETR queries , we can achieve both a faster training convergence and a higher detection accuracy . Some previous work also has similar analysis and confirmed this . For example , SMCA ( Gao et al. , 2021 ) speeds up the training by applying pre-defined Gaussian maps around reference points . Conditional DETR ( Meng et al. , 2021 ) uses explicit positional embedding as positional queries for training , yielding attention maps similar to Gaussian kernels as shown in Fig . 4 ( b ) . Although explicit positional priors lead to good performance in training , they ignore the scale information of an object . In contrast , our proposed DAB-DETR explicitly takes into account the object scale information to adaptively adjust attention weights , as shown in Fig . 4 ( c ) .

Summary: This paper introduces dynamic anchor boxes as a query formulation for detection transformer (DETR). The basic idea is to use explicit positional priors and scale priors from anchor boxes as the queries for training the decoder of DETR. Anchor boxes are also adjusted layer-by-layer to learn the optimal anchor setting. The proposed query formulation provides a better understanding of the query for vision transformer and is proven to outperform previous implicit and explicit query formulations.