Deformable DETR: Deformable Transformers for End-to-End Object Detection
Authors: Xizhou Zhu
Weijie Su
Lewei Lu
Bin Li
Xiaogang Wang
Jifeng Dai
Venue: ICLR 2021 Oral (2021)
Keywords: Efficient Attention Mechanism
Deformation Modeling
Multi-scale Representation
End-to-End Object Detection

Abstract: DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10$\times$ less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach. Code is released at https://github.com/fundamentalvision/Deformable-DETR.

Published as a conference paper at ICLR 2021
DEFORMABLE DETR: DEFORMABLE TRANSFORMERS FOR END TO- TO-END OBJECT DETECTION
Xizhou Zhu Weijie Su²*, Lewei Lu', Bin Li,, Xiaogang Jife'g Dai
Time Research
University Science and Technology of China
'The Chinese University of Hong Kong
(zhuwalter, luotto, dai jackroos@mail cateroosmmmml..s..eeeeeee www..uuu..uuu...... en, xgwang@ee. cuhk. edu. .ddu.h.
com
ABSTRACT
DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10x less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach. Code is released at https::/ github. com/ www.dmmenwaavviiionoonnooooomoae wwtp::awwwwiiiieeeeeoeaaaleeeee.
INTRODUCTION
Modern object detectors employ many hand-crafted components (Liu et al. 2020), c.g., anchor gen- eration, rule-based training target assignment, non- ----imuim suppression (NMS) post-processing They not fully end-to-end. Recently, Carion et al. (2020) proposed DETR eliminate the need for such hand-crafted components, and built the first fully end-to-end object detector, achieving very competitive performance. DETR utilizes a simple architecture, by combining convolutional neural networks (CNNs) and Transformer (Vaswani et al., 2017) encoder-decoders. They exploit the ver- satile and powerful relation modeling capability of Transformers to replace the hand-crafted rules, under properly designed training signals.
Despite its interesting design and good performance, DETR has its own issues: (1) It requires much longer training epochs to converge than the existing object detectors. For example, on the COCO (Lin al.) 2014) benchmark, DETR needs 500 epochs to converge, which around 10 to 20 times slower than Faster R-CNN (Ren et al. 2015). (2) DETR delivers relatively low performance detecting small objects. Modern object detectors usually exploit multi-scale features, where small objects are detected from high-resolution feature maps. Meanwhile, high-resolution feature maps lead to unacceptable complexities for DETR. The above-mentioned issues can be mainly attributed to the deficit of Transformer components in processing image feature maps. At initialization, the attention modules cast nearly uniform attention weights to all the pixels in the feature maps. Long training epoches is necessary for the attention weights to be learned focus sparse meaning- ful locations. On the other hand, the attention weights computation in Transformer encoder of quadratic computation w.r.t. pixel numbers. Thus, it is of very high computational and memory complexities to process high-resolution feature maps.
In the image domain, deformable convolution (Dai al. 2017) is of a powerful and efficient mech- anism to attend to sparse spatial locations. It naturally avoids the above-mentioned issues. While it lacks the element relation modeling mechanism, which is the key for the success of DETR.
Equull contribution. 'Corresponding author. Work is done during an internship SenseTime Research.

---

Published as a conference paper at ICLR 2021
Multi-scale Deformable Self Attlettttnnon Encoder
Multi-scale Feature Maps
Bounding Box Predictions
Multi-scale Deformable Cross Attention in Decoder
Transformer Self-Attention Decoder
Image Feature Maps
x
X
Decoder
Encoder
Image
Object Queries
Figure I: Illustration of the proposed Deformable DETR object detector.
In this paper, we propose Deformable DETR, which mitigates the slow convergence and high com- plexity issues DETR. It combines the best of the sparse spatial sampling of deformable convo- lution, and the relation modeling capability of Transformers. We propose the deformable attention module, which attends to a small set of sampling locations as a pre-filter for prominent key elements of all the feature map pixels. The module can be naturally extended to aggregating multi-scale features, without the help of FPN (Lin et 2017a). In Deformable DETR, we utilize (multi-scale) deformable attention modules replace the Transformer attention modules processing feature maps, as shown in Fig. 1
Deformable DETR opens up possibilities for to exploit variants of end-to-end object detectors, thanks to its fast convergence. and computational and memory efficiency. We explore simple and effective iterative bounding box refinement mechanism to improve the detection performance. We also try two-stage Desormable DETR, where the region proposals are also generated by vaiant of Deformable DETR, which are further fed into the decoder for iterative bounding box refinement.
Extensive experiments on the COCO (Lin et al. 2014) benchmark demonstrate the effectiveness of our approach. Compared with DETR, Deformable DETR can achieve better performance (es- pecially small objects) with 10x less training epochs. The proposed variant of two-stage De- formable DETR can further improve the performance. Code is released at hhttps://qithe com/ htap://naaaaaeettttllsssooooooon
2 RELATED WORK
Efficient Attention Mechanism. Transformers (Vaswani al. 2017) involve both self-attention and cross-attention mechanisms. One of the most well-known concern of Transformers is the high time and memory complexity at vast key element numbers. which hinders model scalability in many cases. Recently, many efforts have been made to address this problem (Tay et al. (020b). which can be roughly divided into three categories practice.
The first category is to use pre-defined sparse attention patterns on keys. The most straightforward paradigm is restricting the attention pattern to be fixed local windows. Most works (Liu et al,
201 Sal Parmar et al. 2018; Child al. 2019 Huang et al., 2019 Ho et al. 2019] Wang et al.
2020a Hu et al. [2019] Ramachandran et al. 2019 QQu et al. 2019 Beltagy et al. 2020 Ainslie al. 2020: Zaheer al. (020) follow this paradigm. Although restricting the attention pattern to a local neighborhood can decrease the complexity, it loses global information. To compensate,
Child et al. (2019): (Huang et al. (2019) Ho al. (2019): Wang al. (2020a) attend key elements
2

---

Published as a conference paper at ICLR 2021
at fixed intervals to significantly increase the receptive field on keys. Beltagy al. (2020); Ainslie jet al. (2020); Zaheer et al. (2020) allow a small number of special tokens having access to all key elements. Zaheer al. (2020); Qiu al. (2019) also add some pre-fixed sparse attention patterns to attend distant key elements directly.
The second category is learn data-dependent sparse attention. KKitev et al. 2020 proposes locality sensitive hashing (LSH) based attention, which hashes both the query and key elements to different bins. A similar idea is proposed by Roy al. (2020), where k-means finds out the most related keys. Tay et al. (020aa learns block permutation for block-wise sparse attention.
The third category is to explore the low-rank property in self-attention, [Wann et al. (2020b) reduces the number of key elements through a linear projection on the size dimension instead of the channel dimension. KKatharopoulos et al. (2020); Choromanski et al. (2020) rewrite the calculation of self- attention through kernelization approximation.
In the image domain, the designs of efficient attention mechanism (e.g., Parmar et (2018 Child
et al.] (2019): Huang et al. (2019); Ho et al. (2019): Wang et al. (2020a) : Hu et al. (2019): Ramachan dran et al. (2019)) are still limited to first category. Despite the theoretically reduced complexity, Ramachandran et al. (219);; Hu et al. (2019) admit such approaches are much slower in implemen- tation than traditional convo lution with the same FL.OPs (at least 3x slower), due to the intrinsic limitation in memory access patterns.
On the other hand, as discussed in Zhu et al. (2019a), there are variants of convolution, such as deformable convolution (Dai et al., 2017: Zhu et al. 2019b) and dynamic convolution (Wu et al.) 2019), that also can be viewed as self-attention mechanisms. Especially, deformable convolution operates much more effectively and efficiently image recognition than Transformer self-attention. Meanwhile, lacks the element relation modeling mechanism.
Our proposed deformable attention module is inspired by deformable convolution, and belongs to the second category. It only focuses on a small fixed set of sampling points predicted from the feature of query elements. Different from Ramachandran et al. (2019); Hu al. (2019), deformable attention is just slightly slower than the traditional convolution under the same FLOPs.
Multi-scale Feature Representation for Object Detection. One of the main difficulties in object detection is to effectively represent objects at vastly different scales. Modern object detectors usually exploit multi-scale features to accommodate this. As one of the pioneering works, FPN (Lin al., 2017a) proposes a top-down path to combine multi-scale features. PANet (Liu et al. 20188) further adds an bottom-up path on the top of FPN. Kong et al. (2018) combines features from all scales by global attention operation. Zhao et al. (2099) proposes U-shape module to fuse multi-scale features. Recently, NAS-FPN (Ghiasi et al. 2019) and Auto-FPN (Xu et al. 2019) are proposed to automatically design cross-scale connections via neural architecture search. (Tan et al. (2020) proposes the BiFPN, which is a repeated simplified version PANet. Our proposed multi-scale deformable attention module can naturally aggregate multi-scale feature maps via attention mecha- nism, without the help of these feature pyramid networks.
REVISITING TRANSFORMERS AND DETR
Multi-Head Attention in Transformers. Transformers (Vaswani et al., 2017) are of a network architecture based on attention mechanisms for machine translation. Given query element (e.g., a target word in the output sentence) and set of key elements (e.g., source words in the input sentence), the multi-head attention module adaptively aggregates the key contents according to the attention weights that measure the compatibility of query-key pairs. To allow the model focusing on contents from different representation subspaces and different positions, the outputs of different attention heads are linearly aggregated with learnable weights. Let q E Ry indexes query element with representation feature 2y E C and l E Ωg indexes a key element with representation feature E CRC, where C is the feature dimension, Py and Ωg specify the set of query and key elements, respectively. Then the multi-head head attention feature is calculated by
MultiHeadAttn(z,
Σ
Wm Σ k
Amgk: W W',
3

---

Published as a conference paper at ICLR 2021
where indexes the attention head, W E RCxC and Wm E RCx are of learnable weights (C, = C/M by default). The attention weights Amgk x exp(
x - Vn } are normalized as VC.
Exeo, Amgt 1, in which Um, Vn E RC are also learnable weights. To disambiguate different spatial positions, the representation features 2n and Dt are usually of the concatena- tion/summation of element contents and positional embeddings.
There are two known issues with Transformers. One is Transformers need long training schedules before convergence. Suppose the number of query and key elements are of N₂ and Ni. respectively. Typically, with proper parameter initialization, 2n and Vn follow distribution with mean of O and variance of 1, which makes attention weights Amgk = N. when N₂ is large. It will lead to ambiguous gradients for input features. Thus, long training schedules are required so that the attention weights can focus specific keys. In the image domain, where the key elements usually of image pixels, N& can be very large and the convergence is tedious.
On the other hand, the computational and memory complexity for multi-head attention can be very high with numerous query and key elements. The computational complexity of Eq. I of O(N,C + 2₂ + N.)).. In the image domain, where the query and key elements both of pixels, N₂ = N& >> C, the complexity is dominated by the third term, as ONN,N,,. Thus, the multi-head attention module suffers from a quadratic complexity growth with the feature map size.
DETR. DETR (Carion et al. 2020) is built upon the Transformer encoder-decoder architecture, combined with set-based Hungarian loss that forces unique predictions for each ground-truth bounding box via bipartite matching. We briefly review the network architecture as follows.
Given the input feature maps æ RCx extracted by a CNN backbone (e.g.. ResNet (He et al., 2016). DETR exploits a standard Transformer mncoder-decoder architecture to transform the input feature maps to be features of set of object queries. A 3-layer feed-forward neural network (FFN) and a linear projection are added on top of the object query features (produced by the decoder) as the detection head. The FFN acts as the regression branch to predict the bounding box coordinates b E [0. 1]3, where b = {b.,, by, bw. bn encodes the normalized box center coordinates, box height and width (relative to the image size). The linear projection acts as the classification branch to produce the classification results.
For the Transformer encoder in DETR, both query and key elements are of pixels in the feature maps. The inputs are of ResNet feature maps (with encoded positional embeddings). Let H and W denote the feature map height and width, respectively. The computational complexity of self-attention is of O(HW C), which grows quadratically with the spatial size.
For the Transformer decoder in DETR, the input includes both feature maps from the encoder, and N object queries represented by learnable positional embeddings (e.g., N 100). There are two types of attention modules in the decoder, namely, cross-attention and self-attention modules. In the cross-attention modules, object queries extract features from the feature maps. The query elements are of the object queries, and key elements are of the output feature maps from the encoder. In it, N2 = N, N₂+ = H W and the complexity of the cross-attention is of O(H + N HHWC WWC).. C"). The complexity grows linearly with the spatial size of feature maps. In the self-attention modules, object queries interact with each other, so as to capture their relations. The query and key elements are both of the object queries. In it, N₂ N₂ = N, and the complexity of the self-attention module is of O(2NC + N2C.). The complexity is acceptable with moderate number of object queries.
DETR is an attractive design for object detection, which removes the need for many hand-designed components. However, it also has its own issues. These issues can be mainly attributed to the deficits of Transformer attention in handling image feature maps as key elements: (1) DETR has relatively low performance in detecting small objects. Modern object detectors use high-resolution feature maps to better detect small objects. However, high-resolution feature maps would lead to an unacceptable complexity for the self-attention module in the Transformer encoder of DETR, which has a quadratic complexity with the spatial size of input feature maps. object detectors, DETR requires many more training epochs to converge. This is mainly because the attention modules processing image features are difficult to train. For example, at initialization, the cross-attention modules almost of average attention the whole feature maps. While, at the end of the training, the attention maps are learned to be very sparse, focusing only on the object
(2) Compared with modern
4

---

Published as a conference paper at ICLR 2021
Fastun
Linser
Reformss Point (erPoo) Po?
LiDASE
Softras
Sumpliy (APea)
Aasiion Wagghs
Heed
Hood
Feod
Heed
Heed
Head
Aggragga
Aggumggate
Input Festur Map
Linovr
porrggtt
1ood 2
Liteer
Ostput
Headh
Read
Aarrggttd Sumpled
Valuom
Figure Illustration of the proposed deformable attention module.
extremities, seems that DETR requires a long training schedule learn such significant changes in the attention maps.
METHOD
4.1 DEFORMABLE TRANSFORMERS FOR END-TO-END OBJECT DETECTION
Deformable Attention Module. The issue of applying Transformer attention image feature maps is that it would look over all possible spatial locations. To address this, we present deformable atlention module. Inspired by deformable convolution (Dai et al. 2017; Zhu et al. 2019b). the deformable attention module only attends to a small set of key sampling points around a reference point, regardless of the spatial size of the feature maps, as shown in Fig. 2 By assigning only small fixed number of keys for each query, the issues of convergence and feature spatial resolution can be mitigated.
Given input feature map E RCxH let index query element with content feature Zy and a 2-d reference point pq. the deformable attention feature is calculated by
DeformAttn(, Pa, x)
W m
Σ Amgr. Rm/
W Wh + Apmqi))
where m indexes the attention head, i indexes the sampled keys, and is the total sampled key number (K & HW). Apmgk and Amgk denote the sampling offset and attention weight of the ttth sampling point in the mh attention head, respectively. The scalar attention weight Amgk lies in the range [0, 1], normalized by E Amgk 1. Apmnk R of 2-d real numbers with unconstrained range. As + Apmak is fractional, bilinear interpolation is applied as in Dai et al. 2017)) in computing (p,, AApak))))) Both Apmgk and Amgk are obtained via linear projection over the query feature zg. In implementation, the query feature 2g is fed to linear projection operator of 3MK channels, where the first 2MK channels encode the sampling offsets Apmak. and the remaining M K channels are fed to softmax operator to obtain the attention weights Amgk-
The deformable attention module designed for processing convolutional feature maps as key ele- ments. Let N, be the number of query elements, when M K is relatively small, the complexity of the deformable attention module is of O(2N, C2 min HWW C N, N,KC)) (See Appendix A. for de- tails). When it is applied DETR encoder, where N = HW, the complexity becomes O (THWW) which is of linear complexity with the spatial size. When it is applied as the cross-attention modules
5

---

Published as a conference paper at ICLR 2021
in DETR decoder, where Ng = N (N is the number of object queries), the complexity becomes O(NKC2), which is irrelevant to the spatial size HW.
Multi-scale Deformable Attention Module. Most modern object detection frameworks benefit from multi-scale feature maps (Liu et al. (020). Our proposed deformable attention module can be naturally extended for multi-scale feature maps.
Let {a' {a be the input multi-scale feature maps, where æ' E C Let P. E [0. 1J2 be the normalized coordinates of the reference point for each query element q, then the multi-scale deformable attention module is applied as
M Σ
MSDeformAtt(zg: Pg. (a')
W
Σ [=1
Amigk W a' (or(p.)) + Apmigh)] (3)
where m indexes the attention head, indexes the input feature level, and &o indexes the sampling point. Apmnig and Amiyk denote the sampling offset and attention weight of the 1thh sampling point in the 1th feature level and the m"h attention head, respectively. The scalar attention weight Amigk
is normalized by E/= Σx=1 Amigh = L. Here, we use normalized coordinates p. E [0. 1]2 for the clarity of scale formulation, in which the normalized coordinates (0, and (1, indicate the top-left and the nottom-right image corners, respectively. Function di (p,)) in Equation 3 re-scales the normalized coordinates Pg to the input feature map of the 1-th level. The multi-scale deformable attention is very similar to the previous single-scale version, except that samples LK points from multi-scale feature maps instead of K points from single-scale feature maps.
The proposed attention module will degenerate to deformable convolution (Dai et al. 2017). when I. 1, K 1, and W., RCC. is fixed designed for single-scale inputs, focusing only on one sampling point for each attention head. How- ever, our multi-scale deformable attention looks over multiple sampling points from multi-scale in- puts. The proposed (multi-scabe deformable attention module can also be perceived as an efficient variant of Transformer attention, where pre-filterng mechanism is introduced by the deformable sampling locations. When the sampling points traverse all possible locations, the proposed attention module is equivalent to Transformer attention.
identity matrix. Deformable convolution is
Deformable Transformer Encoder. We replace the Transformer attention modules processing feature maps in DETR with the proposed multi-scale deformable attention module. Both the input and output of the encoder are of multi-scale feature maps with the same resolutions. In encoder, we extract multi-scale feature maps {a* | (L 4) from the output feature maps of stages C3 through C3 in ResNet (He et al. 2016) (transformed by a x 1 convolution), where C is of resolution 2 lower than the input image. The lowest resolution feature map x² is obtained via a 3 x 3 stride convolution on the final C stage, denoted Cg. All the multi-scale feature maps are of C = 256 channels. Note that the top-down structure in FPN (Lin et al.] 2017a) is not used, because our proposed multi-scale deformable attention in itself can exchange information among multi-scale feature maps. The constructing of multi-scale feature maps are also illustrated in Appendix A.2 Experiments in Section 5.2 show that adding FPN will not improve the performance.
In application of the multi-scale deformable attention module in encoder, the output are of multi- scale feature maps with the same resolutions as the input. Both the key and query elements are of pixels from the multi-scale feature maps. For each query pixel, the reference point is itself. To identify which feature level each query pixel lies in, we add scale-level embedding, denoted as e,, to the feature representation, in addition to the positional embedding. Different from the positional embedding with fixed encodings, the scale-level embedding {ei are randomly initialized and jointly trained with the network.
Deformable Transformer Decoder. There are cross-attention and self-attention modules in the decoder. The query elements for both types of attention modules are of object queries. In the cross- attention modules, object queries extract features from the feature maps, where the key elements are of the output feature maps from the encoder. In the self-attention modules, object queries interact with each other, where the key elements are of the object queries. Since our proposed deformable attention module is designed for processing convolutional feature maps key elements, we only replace each cross aoos-attention module to be the multi-scale deformable attention module, while leaving the self-attention modules unchanged. For each object query, the 2-d normalized coordinate of the

---

Published as a conference paper at ICLR 2021
reference point Py is predicted from its object query embedding via a learnable linear projection followed by a sigmoid function.
Because the multi-scale deformable attention module extracts image features around the reference point, we let the detection head predict the bounding box as relative offsets w.r.t the reference point to further reduce the optimization difficulty. The reference point is used the initial guess of the box center. The detection head predicts the relative offsets w.r.t. the reference point. Check Appendix [A.3] for the details. In this way, the learned decoder attention will have strong correlation with the predicted bounding boxes, which also accelerates the training convergence.
By replacing Transformer attention modules with deformable attention modules in DETR, we tablish an efficient and fast converging detection system, dubbed as Deformable DETR (see Fig.
4.2 ADDITIONAL IMPROVEMENTS AND VARIANTS FOR DEFORMABLE DETR
Deformable DETR opens up possibilities for us to exploit various variants of end-to-end object de- tectors, thanks to its fast convergence, and computational and memory efficiency. Due to limited space, we only introduce the core ideas of these improvements and variants here. The implementa- tion details are given in Appendix A.4
Iterative Bounding Box Refinement. This is inspired by the iterative refinement developed in optical flow estimation (Ted & Deng (020). We establish simple and effective iterative bounding box refinement mechanism to improve detection performance. Here, each decoder layer refines the bounding boxes based the predictions from the previous layer.
Two-Stage Deformable DETR, the original DETR, object queries in the decoder are irrelevant to the current image. Inspired by two-stage object detectors, explore variant of Deformable DETR for generating region proposals as the first stage. The generated region proposals will be fed into the decoder object queries for further refinement, forming two-stage Deformable DETR.
In the first stage, to achieve high-recall proposals, each pixel in the multi-scale feature maps would serve an object query. However, directly setting object queries as pixels will bring unacceptable computational and memory cost for the self-attention modules in the decoder, whose complexity grows quadratically with the number of queries. To avoid this problem, we remove the decoder and form an encoder-only Deformable DETR for region proposal generation. In it, each pixel is assigned as an object query, which directly predicts bounding box. Top scoring bounding boxes are picked as region proposals. No NMS is applied before feeding the region proposals to the second stage.
5 EXPERIMENT
Dataset. We conduct experiments on COCO 2017 dataset (Lin et al. 2014). Our models are trained on the train set, and evaluated on the val set and test-dev set.
Implementation Details. ImageNet Deng et al., 2009) pre-trained ResNet-50 (He et al. 2016)
is utilized the backbone for ablations. Multi-scale feature maps are extracted without FPN (Lin et al. 2017a). M = 8 and K = 4 are set for deformable attentions by default. Parameters of the deformable Transformer encoder are shared among different feature levels. Other hyper-parameter setting and training strategy mainly follow DETR (Carion et al., 2020). except that Focal Loss (Lin) al. 2017b) with loss weight of is used for bounding box classification, and the number of object queries is increased from 100 to 300. We also report the performance of DETR-DO these modifications for fair comparison, denoted as DETR-DC5- By default, models are trained for 50 epochs and the learning rate is decayed at the 40-th epoch by a factor of 0.1. Following DETR(Carion et al. 2020). we train our models using Adam optimizer (Kingma & Baj 2015) with base learning rate of 2 x 10-1, 31 = 0.9, B2 = 0.999, and weight decay of 10-1. Learning rates of the linear projections, used for predicting object query reference points and sampling offsets, are multiplied by factor of O. Run time is evaluated on NVIDIA Tesla V100 GPU.
with
5.1 COMPARISON WITH DETR
As shown in Table [ compared with Faster R-CNN FPN, DETR requires many more training epochs to converge, and delivers lower performance detecting small objects. Compared with
7

---

Published as a conference paper at ICLR 2021
DETR, Deformable DETR achieves better performance (especially small objects) with 10x less training epochs. Detailed convergence curves are shown in Fig. 3 With the aid of iterative bounding box refinement and two-stage paradigm, method can further improve the detection accuracy.
Our proposed Deformable DETR has on par FLOPs with Faster R-CNN FPN and DETR-DGG But the runtime speed is much faster (1.6x) than DETR-DC5, and is just 25% slower than Faster R-CNN + FPN. The speed issue DETR-DCS is mainly due to the large amount of memory access in Transformer attention. Our proposed deformable attention can mitigate this issue, the cost of unordered memory access. Thus, it is still slightly slower than traditional convolution.
Table 1: Comparision of Deformable DETR with DETR COCO 2017 val DETR-DCS* denotes DETR-DC5 with Focal Loss and 300 object queries.
Training
Inference
Method
Epochs
AP
AP50
AP3s
APs
APM
APL
params
FLOPs
GPU hours
FPS
Faster R-CNN FPN
42.0
62.1
45.5
53.4
DETR
500
42.0
62.4
44.2
20.5
45.8
61.1
41M
86G
2000
28
DETR-DC5
500
43.3
63.1
45.9
22.5
47.3
61.1
41M
187G
7000
12
DETR-DC5
35.3
55.7
36.8
15.2
37.5
53.6
41M
187G
700
DETR-DOS*
50
36.2
57.0
37.4
16.3
39.2
53.9
41M
187G
700
12
Deformable DETR
43.8
62.6
47.7
26.4
47.1
58.0
40M
173G
325
iterative hounding box refinement
50
45.4
64.7
49.0
26.8
48.3
61.7
40M
173G
325
19
two-stage Deformable DETR
50
46.2
65.2
50.0
28.8
49.2
61.7
40M
173G
340
19
44.9
45.3
45.5
45
43.6
40
Aup
35
30
Deformable DETR
25
DETR-DCS
50
100
150
200
250
300
350
400
450
500
Epochs
Figure 3: Convergence of Deformable DETR and DETR-DC5 COCO 2017 val For Deformable DETR, we explore different training schedules by varying the epochs at which the leariing rate reduced (where the AP score leaps).
5.2 ABLATION STUDY ON DEFORMABLE ATTENTION
Table 2 presents ablations for various design choices of the proposed deformable attention module. Using multi-scale inputs instead of single-scale inputs effectively improve detection accuracy with 1.7% AP, especially on small objects with 2.9% APs. Increasing the number of sampling points K further improve 0.9% AP. Using multi-scale deformable attention, which allows information exchange among different scale levels, can bring additional 1.5% improvement in AP. Because the cross-level feature exchange is already adopted, adding FPNs will not improve the performance. When multi- scale attention is not applied, and K = 1, our imulti-scale deformable attention module degenerates deformable convolution, delivering noticeable lower accuracy.
5.3 COMPARISON WITH STATE-OFF OF-TTE--TTTTT METHODS
Table 3 compares the proposed method with other state-of the-art methods. Iterative bounding box refinement and two-stage mechanism both utilized by models in Table With ResNet-101 and ResNeXt-101 (Xie et al.) (017). our method achieves 48.7 AP and 49.0 AP without bells and whistles, respectively. By using ResNeXt-100 with DCN (Zuu et al.) 2019b), the accuracy rises to 50.1 AP. With additional test-time augmentations, the proposed method achieves 52.3 AP.
8

---

Published as a conference paper at ICLR 2021
Table 2: Ablations for deformable attention COCO 2017 val "MS inputs" indicates ing multi-scale inputs. "MS attention" indicates using multi-scale deformable attention. K is the number sampling points for each attention head on each feature level.
MS inpuls MS attention
K
FPNs
AP
AP
AP
APs
APM
APt.
4
FPN Lin et al.
2017a
43.8
62.6
47.8
26.5
47.3
58.1
4
BiFPN
Tan et al. 2020
43.9
62.5
47.7
25.6
47.4
57.7
1
39.7
60.1
42.4
21.2
44.3
56.0
41.4
60.9
44.9
24.1
44.6
56.1
w/o
4
42.3
61.4
46.0
24.8
45.1
56.3
4
43.8
62.6
47.7
26.4
47.1
58.0
Table 3: Comparison of Deformable DETR with state-of-the-art methods COCO 2017 test-dev
set. "TTA" indicates test-time augmentations including horizomtal flip and multi-scale testing.
Method
Backbone
TTA
AP
APso
AP1s
APs
APM
APL
FCOS
Tian al.
2019
ResNeXt-101
44.7
64.1
48.4
27.6
47.5
55.6
ATSS
Zhang et al.
2020
ResNeXt-101 + DCN
50.7
68.9
56.3
33.2
52.9
62.4
TSD
Song
2020
SENet154 DCN
51.2
71.9
56.0
33.8
54.8
64.2
EfficienDDet-D7 Tan al.
2020
Efficien[Net-B6
52.2
71.4
56.3
Deformable DETR
ResNet-50
46.9
66.4
50.8
27.7
49.7
59.9
Deformable DETR
ResNet-101
48.7
52.9
29.1
51.5
62.0
Deformable DETR
ResNeXt-101
49.0
68.5
53.2
29.7
51.7
62.8
Deformable DETR
ResNeXt-101 + DCN
50.1
69.7
54.6
30.6
52.8
64.7
Deformable DETR
ResNeXt-101
52.3
71.9
58.1
34.4
54.4
65.6
CONCLUSION
Deformable DETR an end-to-end object detector, which is efficient and fast-converging. It enables us to explore more interesting and practical variants of end-to-end object detectors. At the core of Deformable DETR are the (multi-scale) deformable attention modules, which an efficient attention mechanism in processing image feature maps. We hope our work opens up new possibilities in exploring end-to-end object detection.
ACKNOWLEDGMENTS
The work is supported by the National Key R&D Program of China (2020AAA0105200), Beijing Academy of Artificial Intelligence, and the National Natural Science Foundation of China under grand No. U1980444 and No.61836011.
REFERENCES
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. Etc: Encoding long and structured data in transformers. arXiv preprint arXiv:2004.08483 2020.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- documentt transformer. arXiv preprint arXhv:2004.05150, 2020.
Nicolas Carion. Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Rewon Child, Scott Gray, Alee Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint
2019.
Krzysztof Choromanski, Valeri Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamas Sarlos, David Belanger, Lucy Colwell, and Adrian Weller. Masked language modeling for pro- teins via linearly scalable long-context transformers. arXiv preprint arXiv:2006.03555, 2020.

---

Published as a conference paper at ICLR 2021
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Golnaz Ghiasi, Tsung-Yi Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid archi- tecture for object detection. In CVPR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenbom, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprini arXiv:/9/2.12/86 2019.
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV, 2019.
Zilong Huang, Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In ICCV, 2019.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Flcuret. Transformers are mns: Fast autoregressive transformers with linear attention. arXiv preprint arXiv:2006.16236, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
Tao Kong, Fuchun Sun, Chuanqi Tan, Huaping Liu, and Wenbing Huang. Deep feature pyramid reconfiguration for object detection. In ECCV, 2018.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. CVPR, 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017b.
Li Li,, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey. IJCV, 2020.
Peter Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In ICLR, 2018a.
Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In CVPR, 2018b.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In JCML, 2018.
Jiezhong Qiu, Hao Ma. Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self-attention for long document understanding. arXiv preprint arXiv:1911.02972. 2019.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. In NeurIPS, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NewrIPS, 2015.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. arXiv preprint arXiv:2003.05999, 2020.
10

---

Published as a conference paper at ICLR 2021
Guanglu Song, Yu Liu, and Xiaogang Wang. Revisiting the sibling head in object detector. In CVPR, 2020.
Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. In CVPR, 2020.
Yi Tay. Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In ICML, 2020a.
Yi Tay, Mostafa Dehghami. Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint arXiv:2009.0672 2020b.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In ICCV, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. arXiv preprint arXiv:2003.07853, 2020a.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.00 04768, 2020b.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Hang Xu, Lewei Yao, Wei Zhang, Xiaodan Liang, and Zhenguo Li. Auto-fpn: Automatic network architecture adaptation for object detection beyond classification. In ICCV, 2019.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007. 14062, 2020.
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan 7. Li. Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In CVPR, 2020.
Qijic Zhao, Tao Sheng, Yongtao Wang, Zhi Tang, Ying Chen, Ling Cai, and Haibin Ling. M2det: A single-shot object detector based on multi-level feature pyramid network. In AAAI, 2019.
Xizhou Zhu, Dazhi Cheng, Zheng Zhang, Stephen Lin, and Jifeng Dai. An empirical study of spatial attention mechanisms deep networks. ICCV, 2019a.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In CVPR, 2019b.
11

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 COMPLEXITY FOR DEFORMABLE ATTENTION
Supposes the number of query elements is Nq. in the deformable attention module (see Equation the complexity for calculating the sampling coordinate offsets Apmyk and attention weights Amgk is of (33,, ((3N,CMM)) Given the sampling coordinate offsets and attention weights, the complexity of computing Equation 2 is ONN,C² + N22 + 5N,KK) where the factor of 5 in 5 5 because of bilinear interpolation and the weighted sum attention. On the other hand, we can also calculate W before sampling, as it independent to query, and the complexity of computing +5N
2 will become as ONN,C² attention is O22 min(HW C C,, 5N,KC 3N, CMK.. In our experiments, M = 8, K I and C = 256 by default, thus 5K + 3MK < C and the complexity is of O(2N,C* C2 + min(HWC2 )2
Equation
So the overall complexity of deformable
A.2 CONSTRUCTING MULT-SCALE FEATURE MAPS FOR DEFORMABLE DETR
As discussed in Section and illustrated in Figure the input multi-scale feature maps of the
encoder {xx 1==1 (L = 4) are extracted from the output feature maps of stages C3 through C3 in ResNet (He et al. 2016 (transformed by a 1 x convolution). The lowest resolution feature map x' obtained via 3 x stride 2 convolution on the final Cş stage. Note that FPN (Lin et al. (2077)) is not used, because our proposed multi-scale deformable attention in itself can exchange information among multi-scale feature maps.
Cony x 3trride
H/64 W
Conv x stsride
H/32x /32 2048
H/32 H332XW/32X255
1,stride
H/16xW/1C
H/16xW/161
Conv sttride
H/8x W 512
H/Bx 77x226
ResNet Feature Maps
Input Multi-scale Feature Maps(x (
Figure 4: Constructing mult-scale feature maps for Deformable DETR.
A.3 BOUNDING Box PREDICTION IN DEFORMABLE DETR
Since the multi-scale deformable attention module extracts image features around the reference point, we design the detection head to predict the bounding box as relative offsets w.r.t. the reference point to further reduce the optimization difficulty. The reference point is used as the initial guess of the box center. The detection head predicts the relative offsets w.r.t. the reference point P, +o
(bay - (Pqy)) (bbg)) (bgh))) (bb))) where by(=..nooh)) are predicted by the detection head. a and o denote the sigmoid and the inverse sigmoid function, respectively. The usage of a and o- is to ensure b is of normalized coordinates, as b, E [0, 1 this way, the learned decoder attention will have strong correlation with the predicted bounding boxes, which also accelerates the training convergence.
(oge:Pup,,i.e.. i.e. 6,
A.4 MORE IMPLEMENTATION DETAILS
Iterative Bounding Box Refinement. Here, each decoder layer refines the bounding boxes based on the predictions from the previous layer. Suppose there are D number of decoder layers (e.g., D 6). given normalized bounding box b predicted by the (d 1)-th decoder layer, the d-th
12

---

Published as a conference paper at ICLR 2021
decoder layer refines the box
63 (o(Ab to to- ( (6)g))))) or (Abe 39 -
to (6 o Ab quo to ( (Abg to (Bx))) })}
IR are predicted at the d-th decoder layer. Prediction heads for different decoder layers do not share parameters. The initial box set as 6gm Pow- 60
where E {1, 2, D}. Add p(a.y,,h)
"y Pay. 6Gg 0.1, and 80 0.1. The system is robust to the choice of 6g. and pp.- We tried setting them 0.05. 0.1, 0.2, 0.5, and achieved similar performance. To stabilize training, similar
to Teed & Deng 2020), the gradients only back propagate through Abd "(w.y,w,')' and are blocked at
(ba-1
((2.9.wAA)
In iterative bounding box refinement, for the d-th decoder layer, we sample key elements respective to the box bạ-1 predicted from the (d - 1) 3-th decoder layer. For Equation in the cross-attention module of the d-th decoder layer, (b==1, b serves as the new reference point. The sampling offset Apmigk is also modulated by the box size, as (Apmiako b/ Apmigky bg Such modifi- cations make the sampling locations related to the center and size of previously predicted boxes.
Two-Stage Deformable DETR. In the first stage, given the output feature maps of the encoder, a detection head is applied to each pixel. The detection head is of 3-layer FFN for bounding box regression, and a linear projection for bounding box binary classification (i.c., foreground and back- ground), respectively. Let index pixel from feature level 4 E {1,2, ..., L} with 2-d normalized coordinates p: (Pir, E [0, 1)2, its corresponding bounding box is predicted by
b, = to to- o (Abig to + of (Abiu to (2h--)))) ( to (2hh--))))) s))},
where the base object scale s is set as 0.05, Ab((a.m.m.)) R are predicted by the bounding box regression branch. The Hungarian loss in DETR is used for training the detection head.
Given the predicted bounding boxes in the first stage, top scoring bounding boxes are picked as region proposals. In the second stage, these region proposals are fed into the decoder as initial boxes for the iterative bounding box refinement, where the positional embeddings of object queries are set as positional embeddings of region proposal coordinates.
Initialization for Multi-scale Deformable Attention. In our experiments, the number of tention heads is set as M = 8. In multi-scale deformable attention modules, W E RCxxC and Wm E CCC. are randomly initialized. Weight parameters of the linear projection for initialized to zero. Bias parameters of the linear projection
predicting Amigk and Apmigk are initialized to make Amigk LK and (Apuih (-k, -k), Ap2izk (-k, 0), Apsigk (-k, k), Aptigh = (0, -k), Apoiak = k0, Apsigk = (k, -k), Aprigk = (k, 0), Apsigk = (k, (k E (1,2.... at initialization.
For iterative bounding box refinement, the initialized bias parameters for Apmigk prediction in the decoder are further multiplied with 2K so that all the sampling points at initialization are within the corresponding bounding boxes predicted from the previous decoder layer.
1
A.5 WHAT DEFORMABLE DETR LOOKS AT?
For studying what Deformable DETR looks at to give final detection result, we draw the gradient norm of each item in final prediction (i.e., x/y coordinate object center, width/height of object bounding box, category score of this object) with respect to each pixel in the image, as shown in
Fig. 5 According to Taylor's theorem, the gradient norm can reflect how much the output would be changed relative to the perturbation of the pixel, thus it could show which pixels the model mainly relys on for predicting each item.
The visualization indicates that Deformable DETR looks at extreme points of the object to deter- mine its bounding box, which is similar to the observation in DETR (Carion et al. 2020). More con- cretely, Deformable DETR attends to left/right boundary of the object for coordinate and width, and top/bottom boundary for y coordinate and height. Meanwhile, different to DETR (Carion et al. 2020), our Deformable DETR also looks at pixels inside the object for predicting its category.
13

---

Published as a conference paper at ICLR 2021
Oz ar
Eho
OA || af
dc
Figure 5: The gradient norm of each item (coordinate of object center (x, y). width/height of object bounding box w/ h, category score of this object) in final detection result with respect each pixel in input image I.
A.6 VISUALIZATION OF MULTI-SCALE DEFORMABLE ATTENTION
For better understanding learned multi-scale deformable attention modules, visualize sampling points and attention weights of the last layer in encoder and decoder, as shown in Fig. [ For readibility, we combine the sampling points and attention weights from feature maps of different resolutions into one picture.
Similar DETR (Carion al. 2020), the instances are already separated in the encoder of De- formable DETR. While in the decoder, our model is focused on the whole foreground instance instead of only extreme points observed in DETR (Carion al. 2020). Combined with the visu- de alization of 87 in Fig. 5 we can guess the reason that our Deformable DETR needs not only extreme points but also interior points to detemine object category. The visualization also demon- strates that the proposed multi-scale deformable attention module can adapt its sampling points and attention weights according to different scales and shapes the foreground object.
14

---

Published as a conference paper at ICLR 2021
high
l0w
(a) multi-scale deformable self-attention in encoder
high
(b) multi-scale deformable cross-attention in decoder
Figure 6: Visualization of multi-scale deformable attention. For readibility, we draw the sampling points and attention weights from feature maps of different resolutions one picture. Each sampling point is marked as a filled circle whose color indicates its correspoinding attention weight. The reference point shown as green cross marker, which is also equivalent query point in encoder. In decoder, the predicted bounding box is shown as green rectangle and the category and confidence score are texted just above
15

---

Published as a conference paper at ICLR 2021
A.7 NOTATIONS
Table 4: Lookup table for notations in the paper.
Notation Description
m
index for attention head
index for feature level of key element
index for query element
index for key element
number of query elements
N
number of key elements
M
number of attention heads
L
number of input feature levels
K
number of sampled keys in each feature level for each attention head input feature dimension
C
C
feature dimension at each attention head
height of input feature map width of input feature map height of input feature map of (th feature level width of imput feature map of 7th feature level
W
H'
W
Amak
attention weight of 9th query to 1tth key at m head attention weight of gdh query to htth key in 1th feature level at mhth head input feature of 9th query 2-d coordinate of reference point for th query normalized 2-d coordinate of reference point for 4th query
Amigk
Rq
Pa
Pa
a
input feature map (input feature of key elements)
input feature of 4thh key input feature map of 7th feature level
Ammal
sampling offset of 4t query hth key at mh head sampling offset of qth query to 4th key in 7 feature level at th head output projection matrix at th head input query projection matrix mh head input key projection matrix at mhth head input value projection matrix at m head unnormalized 2-d coordinate of in 1th feature level
A Ppmitl
W
U
Vn
W
(i(P)
exp
exponential function
sigmoid function
o
inverse sigmoid function
16


Reviews:
[{"review_id": "fya7XUpEcg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "As a new framework for object detection, DETR is very important. However,  it suffers from slow convergence and limited feature spatial resolution. This paper proposes deformable attention, which attends to a small set of sampling locations rather than all the locations in the original DETR. Besides, the paper applies multi-scale deformable attention for better results.\n\nThe paper is well-written and obtains very impressive results. Traning for only 50 epochs, deformable Detr obtain results similar to DETR which is trained for 500 epochs. By implementing a two-stage detector based on deformable Detr, the paper obtain state-of-the-art object detection results with a very high AP (52.3) on AP.\n\n A few suggestions for improving the paper are given as follows.\n (1) The training and testing times could be reported in the paper, which is useful for other researchers to implement and use this method.\n (2) Some related methods on sparse connected self-attention/transformer [a,b,c] should be cited and discussed.\n\n [a] Representative Graph Neural Network, CVPR 2020\n [b] Dynamic Graph Message Passing Networks, CVPR 2020\n [c] CCNet: Criss-Cross Attention for Semantic Segmentation in ICCV 19 & TPAMI 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LZ9NJonoMnL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes Deformable DETR with multi-scale deformable attention modules to solve the problems of DETR: slow convergence and limited feature spatial resolution.  In particular, it has faster convergence and achieves better performance(especially on small objects) than DETR.\n\nReasons for score:\n\nOverall, I vote for accepting. I like this paper because it solves the main problems suffered by DETR. My major concern is about the clarity of the paper and some additional ablation studies (see cons below). Hopefully the authors can address my concern in the rebuttal period.\n\nPros:\n\n1.This paper solves the main problems suffered by DETR: slow convergence and limited feature spatial resolution. In my opinion, it makes DETR more practical.\n\n2.The proposed Deformable DETR can obtain multi-scale features without a huge cost. In this way, it can be optimized easily and detect objects precisely, especially small objects.\n\n3.This paper also introduces some improvements and variants to boost the performance of Deformable DETR.\n\nCons:\n\n1.In the experiments, focal loss is used for bounding box classification. What is the reason for this choice? In addition, the number of object queries is increased from 100 to 300. Why? In the test, how to choose 100 objects from 300 objects? \n\n2.From Table 1, we can find DETR (500 epochs) has better performance than Deformable DETR on large objects (61.1 vs. 58.0), though the overall performance of Deformable DETR is better. Why?\n\n3.In the Table 1, there is only DETR-DC5+ (50 epochs). Could you provide DETR-DC5+ (500 epochs) ?\n\nQuestions during rebuttal period:\n\nPlease address and clarify the cons above", "extra_scores": {}, "extra_text": {}}, {"review_id": "C2eDl-piNdR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution is a new attention module called deformable attention module. Like deformable convolution, it adds a translation term into the expression of the transformer, allowing a sparse spatial sampling. The resulting model is very interesting in terms of convergence and complexity compared to the original DETR. A Multi-scale deformable attention module is also proposed. it needs to add a scale function in the attention module equation. Experiments shows that it increases the AP detection rate on MSCOCO compared to FasterR-CNN and DETR. \nContributions are clearly state and validated. The complexity study is very interesting and shows the interest of deformable attention module.\nFigure 1 presents a general view of the model. Since the deformable attention module is the core of the contribution, it should be interesting to add a figure dedicated to this component. Combined with eq.2, it will give a better understanding of the method. \nIt seems that the Axial-DeepLab paper presented in ECCV-2020 misses in the references. This paper proposes a simple strategy for attention modules that also reduce complexity. \nResults clearly show that deformable DETR provides better AP than DETR for less training-epochs. Moreover, the convergence is better than for Faster R-CNN (FPN). \nOne of the concerns with deformable convolution is that the computation speed is slower than classical convolution. The same drawback appears with deformable attention modules. Fps decrease from 26 to 19 compared to DETR. It should be interesting to also report fps in the state of the art comparison table 3. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "8In3YKsvOX7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper aims to improve a very recent detection model -- DETR, which suffers from two issues: long training time and limited feature spatial resolution. Targeting these issues, this paper proposes (1) deformable attention (2) multi-scale processing (inputs/attention) for DETR, which have greatly reduced the  training time and improved the performance. Moreover, it develops two additional modules \"iterative bounding box refinement\" and  \"two-stage framework\", which help to achieve the SOTA detection results on COCO.\n\n### Pros\n - This paper is insightful as it studies the two biggest issues of DETR. The vanilla attention is slow but hard to replace as the performance may be harmed. Similarly, it's not super straightforward to incorporate multi-scale processing into DETR due to the novel framework architecture. This paper has addressed these two issues, demonstrated outstanding performance on COCO, and properly studied the effectiveness of each single module. \n - The deformable attention module is a new kind of attention implementation. It samples a fix number of feature points on spatial feature map and thus greatly reduces the complexity. \n - The additional techniques make lots of sense to me, and improve the results greatly. I'm impressed that transformer based model can achieve very competitive  as shown in Tab.3. \n\n### Cons\n - The deformable attention is considered the most important contribution of this paper and thus should be studied more thoroughly. Specifically, comparisons between this module and other \"linear\"/\"efficient\" implementations of attention should be performed. As discussed in related work, baselines like \"pre-defined sparse attention\", \"data-dependent sparse attention\", and \"low-rank property attention\" should be considered. This kind of comparisons will help us to understand better about the proposed module, and also to researchers from other field who are interested in using this.\n  - Another missed baseline is DETR with multi-scale input and attention (vanilla version) for decently long epochs. I would want to know whether the proposed multi-scale thing could help the vanilla DETR.\n - Does the level embedding help? I don't find experiments to support this design choice.\n - If small objects are the issue, why not have a feature map of H/4 x W/4? Why are multi-scale feature maps constructed like in Appendix Fig.3? \n\n### Minor\n - There are lots of notations. It's pretty hard for me to remember \"mlqk, A, W, \\phi, ...\". I would recommend the authors providing a lookup table in Appendix.\n - Personally, I think the discussion about FPN in main paper is distractive. The authors may want to move all of them into Appendix.\n\n### Questions\n - Do the last 5 rows in Tab.3 use \"iterative bounding box refinement\" and  \"two-stage framework\"? The difference between these 5 lines are just the backbone? \n - Fig.2 shows that Deformable DETR keeps improving over time. It doesn't converge at Ep.50. Why stick to this number in most experiments? How many epochs do the models in Tab.3 get trained?\n - In Tab.1, why do \"iterative bounding box refinement\" and \"two-stage Deformable DETR\" have no influence on FPS? Shouldn't the speed become slower as they are iterative and stage-wise. Does this FPS mean training FPS of Deformable DETR only? If that's the case, please also provide the total training time of everything.\n- Please guide me to the definition/explanation of \"DETR-DC5\". Does it mean the backbone in DETE is changed to ResNet-DC5?\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Accept. The paper proposes Deformable DETR that builds on DETR and solves the slow convergence and limited spatial resolution problem while getting impressive results.
The authors should think about comparing with other linear attention mechanisms to show the applicability of the method.
Decision: Accept (Oral)