Title: Deformable DETR: Deformable Transformers for End-to-End Object Detection
Authors: Xizhou Zhu
Weijie Su
Lewei Lu
Bin Li
Xiaogang Wang
Jifeng Dai
Venue: ICLR 2021
Keywords: Efficient Attention Mechanism
Deformation Modeling
Multi-scale Representation
End-to-End Object Detection

Abstract: DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10$\times$ less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach. Code is released at https://github.com/fundamentalvision/Deformable-DETR.


Review discussion:
### Summary
This paper aims to improve a very recent detection model -- DETR, which suffers from two issues: long training time and limited feature spatial resolution. Targeting these issues, this paper proposes (1) deformable attention (2) multi-scale processing (inputs/attention) for DETR, which have greatly reduced the  training time and improved the performance.
Moreover, it develops two additional modules "iterative bounding box refinement" and  "two-stage framework", which help to achieve the SOTA detection results on COCO.
### Pros
- This paper is insightful as it studies the two biggest issues of DETR. The vanilla attention is slow but hard to replace as the performance may be harmed.
Similarly, it's not super straightforward to incorporate multi-scale processing into DETR due to the novel framework architecture. This paper has addressed these two issues, demonstrated outstanding performance on COCO, and properly studied the effectiveness of each single module.
- The deformable attention module is a new kind of attention implementation. It samples a fix number of feature points on spatial feature map and thus greatly reduces the complexity.
- The additional techniques make lots of sense to me, and improve the results greatly. I'm impressed that transformer based model can achieve very competitive  as shown in Tab.3.
### Cons
- The deformable attention is considered the most important contribution of this paper and thus should be studied more thoroughly.
Specifically, comparisons between this module and other "linear"/"efficient" implementations of attention should be performed. As discussed in related work, baselines like "pre-defined sparse attention", "data-dependent sparse attention", and "low-rank property attention" should be considered. This kind of comparisons will help us to understand better about the proposed module, and also to researchers from other field who are interested in using this.
- Another missed baseline is DETR with multi-scale input and attention (vanilla version) for decently long epochs. I would want to know whether the proposed multi-scale thing could help the vanilla DETR.
- Does the level embedding help? I don't find experiments to support this design choice.
- If small objects are the issue, why not have a feature map of H/4 x W/4? Why are multi-scale feature maps constructed like in Appendix Fig.3?
### Minor
- There are lots of notations. It's pretty hard for me to remember "mlqk, A, W, \phi, ...". I would recommend the authors providing a lookup table in Appendix.
- Personally, I think the discussion about FPN in main paper is distractive. The authors may want to move all of them into Appendix.
### Questions
- Do the last 5 rows in Tab.3 use "iterative bounding box refinement" and  "two-stage framework"? The difference between these 5 lines are just the backbone?
- Fig.2 shows that Deformable DETR keeps improving over time. It doesn't converge at Ep.50. Why stick to this number in most experiments? How many epochs do the models in Tab.3 get trained?
- In Tab.1, why do "iterative bounding box refinement" and "two-stage Deformable DETR" have no influence on FPS? Shouldn't the speed become slower as they are iterative and stage-wise. Does this FPS mean training FPS of Deformable DETR only? If that's the case, please also provide the total training time of everything.
- Please guide me to the definition/explanation of "DETR-DC5". Does it mean the backbone in DETE is changed to ResNet-DC5?
We thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.
Q#1: Comparisons between the deformable attention module and other "linear"/"efficient" implementations of attention should be performed.
A#1: Thanks for the great suggestion. It is very valuable to conduct a survey to benchmark these efficient attention mechanisms on object detection.
However, most of these efficient attention mechanisms are developed for the NLP domain, which haven been rarely studied in the image domain.
Thus, conducting this survey requires lots of effort, which is beyond the scope of this paper.
On the other hand, as we discussed in the related works, in the image domain, the designs of efficient attention mechanism are still limited to using pre-defined sparse attention patterns.
Despite the theoretically reduced complexity, such approaches are much slower in implementation than traditional convolution with the same FLOPs (at least 3× slower), due to the intrinsic limitation in memory access patterns. Compared to these efficient attention mechanisms in the image domain, our deformable attention module learns data-dependent sparse attention patterns, and just slightly slower than the traditional convolution under the same FLOPs.
Q#2: "Another missed baseline is DETR with multi-scale input and attention (vanilla version) for decently long epochs."
A#2: Thanks for your suggestion.
However, it is impossible to conduct such an experiment.
In particular, training DETR with multi-scale inputs and vanilla transformer attention modules would require more than 500G of GPU memory (with batchsize of 1), which far exceeds the GPU memory limit. This is because the vanilla transformer attention modules suffer from a quadratic complexity w.r.t the number of input elements, which is very large in the case of multi-scale inputs.
Q#3: Effect of level embedding.
A#3: Adding level embedding is a natural design to identify which feature level each query pixel lies in. It will not introduce any additional computations, but improves the AP with 0.5 points. Due to the page limit, we did not include this ablation experiment, which will be added in the revision.
| Deformable DETR     | Epochs |  AP  | AP50 | AP75 | AP@S | AP@M | AP@L |
| w. level embedding  |   50   | 43.8 | 62.6 | 47.7 | 26.4 | 47.1 | 58.0 |
| w/o level embedding |   50   | 43.3 | 62.3 | 47.2 | 25.9 | 46.5 | 57.6 |
Q#4: Why using multi-scale feature maps instead of a high-resolution feature map (e.g., H/4 x W/4) to mitigate the issue of small objects?
A#4: It is a common practice in object detection, that large objects are detected on low-resolution feature maps while small objects are detected on high-resolution feature maps. As shown in [a], using multi-scale feature maps is more effective than using the single-scale high-resolution feature map in object detection.
[a] Feature pyramid networks for object detection. In CVPR, 2017.
Minor:
Q#5: Add a lookup table for notations in the Appendix. Move the discussion about FPN into Appendix.
A#5: Thanks for the suggestions. We shall consider them in revision.
Questions:
Q#6: Do the last 5 rows in Tab.3 use "iterative bounding box refinement" and "two-stage framework"? The difference between these 5 lines is just the backbone?
A#6: Sorry for the confusion. The differences among the last 5 rows in Tab.3 are just the backbone and TTA. They all use "iterative bounding box refinement" and "two-stage framework". We shall clarify this in revision.
Q#7: Why stick to 50 epochs? How many epochs do the models in Tab.3 get trained?
A#7: Models in Tab.3 are all trained for 50 epochs. The reason we use 50 epochs for most experiments is that it is within an affordable training time.
Otherwise, the experiments would take too much time for us.
Q#8: In Tab.1, why do "iterative bounding box refinement" and "two-stage Deformable DETR" have no influence on FPS? Does this FPS mean training FPS of Deformable DETR only? If that's the case, please also provide the total training time of everything.
A#8: "FPS" in Tab.1 refers to the inference speed as usual.
To be precise, the inference speed of Deformable DETR is 19.4 FPS. Adding "iterative bounding box refinement" causes <0.1 FPS drop.
Further adding "two-stage Deformable DETR" makes the inference speed drop to 18.8 FPS. The total training time of models in Tab.1 are shown as follows (measured on NVIDIA Tesla V100 GPU), which will be added in the revision:
| Model                               | Epochs | Total Training Time (GPU hours) |
| ----------------------------------- | :----: | :-----------------------------: |
| Faster R-CNN + FPN                  |  109   |               380               |
| DETR                                |  500   |              2000               |
| DETR-DC5                            |  500   |              7000               |
| DETR-DC5                            |   50   |               700               |
| DETR-DC5+                           |   50   |               700               |
| Deformable DETR                     |   50   |               325               |
| + iterative bounding box refinement |   50   |               325               |
| ++ two-stage Deformable DETR        |   50   |               340               |
Q#9: Definition/explanation of "DETR-DC5".
A#9: We quote the word of "DETR-DC5" from the original DETR paper (see Technical details in Page 8 of [b]). The backbone of DETR-DC5 is ResNet50 with dilated C5 stage.
[b] End-to-end object detection with transformers. In ECCV, 2020.
The main contribution is a new attention module called deformable attention module.
Like deformable convolution, it adds a translation term into the expression of the transformer, allowing a sparse spatial sampling. The resulting model is very interesting in terms of convergence and complexity compared to the original DETR. A Multi-scale deformable attention module is also proposed. it needs to add a scale function in the attention module equation. Experiments shows that it increases the AP detection rate on MSCOCO compared to FasterR-CNN and DETR.
Contributions are clearly state and validated. The complexity study is very interesting and shows the interest of deformable attention module.
Figure 1 presents a general view of the model. Since the deformable attention module is the core of the contribution, it should be interesting to add a figure dedicated to this component. Combined with eq.2, it will give a better understanding of the method.
It seems that the Axial-DeepLab paper presented in ECCV-2020 misses in the references. This paper proposes a simple strategy for attention modules that also reduce complexity.
Results clearly show that deformable DETR provides better AP than DETR for less training-epochs.
Moreover, the convergence is better than for Faster R-CNN (FPN).
One of the concerns with deformable convolution is that the computation speed is slower than classical convolution. The same drawback appears with deformable attention modules. Fps decrease from 26 to 19 compared to DETR. It should be interesting to also report fps in the state of the art comparison table 3.
We thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.
Q#1: "Since the deformable attention module is the core of the contribution, it should be interesting to add a figure dedicated to this component."
A#1: Thanks for the good suggestion. Due to the page limit, we did not include the figure that demonstrates the deformable attention module. We shall add it in the revision, where one more page is available.
Q#2: "It seems that the Axial-DeepLab paper presented in ECCV-2020 misses in the references."
A#2: Thanks for your kind reminder. We shall add it in the revision.
Q#3: The computation speed of deformable attention modules is slower than classical convolution.
A#3: As we discussed in the Related Work and Section 5.1, the vanilla Transformer attention suffers from the large amount of memory access, meanwhile the previously proposed sparse attention mechanisms suffer from the intrinsic limitation in memory access patterns. Our proposed deformable attention can mitigate these issues, but at the cost of small amount of unordered memory access.
Thus, it is still slightly slower than the traditional convolution under the same FLOPs.
On the other hand, the speed issue is because modern GPUs are equipped with highly optimized hardware and implementations for traditional convolution, while the random memory access required by the deformable attention modules is hardware unfriendly. The improvement in hardware and implementations may mitigate this issue.
Q#4: "It should be interesting to also report fps in the state of the art comparison table 3."
A#4: It is difficult to compare FPS fairly with other methods at system level in table 3, because they use different codebases, DL frameworks (TensorFlow, PyTorch, etc.), and hardware platforms. We would try our best to add this comparison in revision.
Summary:
This paper proposes Deformable DETR with multi-scale deformable attention modules to solve the problems of DETR: slow convergence and limited feature spatial resolution.
In particular, it has faster convergence and achieves better performance(especially on small objects) than DETR.
Reasons for score:
Overall, I vote for accepting. I like this paper because it solves the main problems suffered by DETR. My major concern is about the clarity of the paper and some additional ablation studies (see cons below). Hopefully the authors can address my concern in the rebuttal period.
Pros:
1.This paper solves the main problems suffered by DETR: slow convergence and limited feature spatial resolution. In my opinion, it makes DETR more practical.
2.The proposed Deformable DETR can obtain multi-scale features without a huge cost. In this way, it can be optimized easily and detect objects precisely, especially small objects.
3.This paper also introduces some improvements and variants to boost the performance of Deformable DETR.
Cons:
1.In the experiments, focal loss is used for bounding box classification. What is the reason for this choice?
In addition, the number of object queries is increased from 100 to 300. Why? In the test, how to choose 100 objects from 300 objects?
2.From Table 1, we can find DETR (500 epochs) has better performance than Deformable DETR on large objects (61.1 vs. 58.0), though the overall performance of Deformable DETR is better. Why?
3.In the Table 1, there is only DETR-DC5+ (50 epochs). Could you provide DETR-DC5+ (500 epochs) ?
Questions during rebuttal period:
Please address and clarify the cons above
We thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.
Q#1(a): "In the experiments, focal loss is used for bounding box classification. What is the reason for this choice?
In addition, the number of object queries is increased from 100 to 300. Why?
A#1(a): These two modifications are both general improvements for object detectors, which are also applicable on vanilla DETR. We also include an improved version of DETR (namely "DETR-DC5+") in table 1 for a fair comparison.
Q#1(b): "In the test, how to choose 100 objects from 300 objects?"
A#1(b): For each image, following the standard evaluation protocol for COCO detection, the top-scored 100 detections (across all categories) will be chosen as the final predictions.
Q#2: "From Table 1, we can find DETR (500 epochs) has better performance than Deformable DETR on large objects (61.1 vs. 58.0), though the overall performance of Deformable DETR is better. Why?"
A#2: Indeed, we also find it is interesting that the original DETR is better than our Deformable DETR on large objects.
However, we still don’t know what the reason is.
Further study is needed.
Q#3: "In Table 1, there is only DETR-DC5+ (50 epochs). Could you provide DETR-DC5+ (500 epochs)?"
A#3: Thanks for your suggestion.
However, training DETR-DC5 for 500 epochs is extremely slow. Around 18 days are required on 16 NVIDIA Tesla V100 GPUs. Due to limited computing resources, it is not affordable for us. By the way, from this point, Deformable DETR opens up the possibilities to explore these variants thanks to its fast convergence and efficiency.
As a new framework for object detection, DETR is very important.
However,  it suffers from slow convergence and limited feature spatial resolution. This paper proposes deformable attention, which attends to a small set of sampling locations rather than all the locations in the original DETR.
Besides, the paper applies multi-scale deformable attention for better results.
The paper is well-written and obtains very impressive results. Traning for only 50 epochs, deformable Detr obtain results similar to DETR which is trained for 500 epochs. By implementing a two-stage detector based on deformable Detr, the paper obtain state-of-the-art object detection results with a very high AP (52.3) on AP.
A few suggestions for improving the paper are given as follows.
(1) The training and testing times could be reported in the paper, which is useful for other researchers to implement and use this method.
(2) Some related methods on sparse connected self-attention/transformer [a,b,c] should be cited and discussed.
[a] Representative Graph Neural Network, CVPR 2020
[b] Dynamic Graph Message Passing Networks, CVPR 2020
[c] CCNet: Criss-Cross Attention for Semantic Segmentation in ICCV 19 & TPAMI 2020
We thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.
Q#1: "The training and testing times could be reported in the paper, which is useful for other researchers to implement and use this method."
A#1: Thanks for your good suggestion. The testing time is already reported in the column "FPS" of Table 1. The total training time of each model in Table 1 is shown as follows (measured on NVIDIA Tesla V100 GPU), which will be added in the revision:
| Model                               | Epochs | Total Training Time (GPU hours) |
| ----------------------------------- | :----: | :-----------------------------: |
| Faster R-CNN + FPN                  |  109   |               380               |
| DETR                                |  500   |              2000               |
| DETR-DC5                            |  500   |              7000               |
| DETR-DC5                            |   50   |               700               |
| DETR-DC5+                           |   50   |               700               |
| Deformable DETR                     |   50   |               325               |
| + iterative bounding box refinement |   50   |               325               |
| ++ two-stage Deformable DETR        |   50   |               340               |
Q#2: "Some related methods on sparse connected self-attention/transformer [a,b,c] should be cited and discussed."
A#2: Thanks for your kind reminder. We shall cite and discuss them in revision. By the way, [c] has already been cited.
[a] Representative Graph Neural Network, CVPR 2020
[b] Dynamic Graph Message Passing Networks, CVPR 2020
[c] CCNet: Criss-Cross Attention for Semantic Segmentation in ICCV 19 & TPAMI 2020

Meta-review:
role: PC
data: comment: Accept. The paper proposes Deformable DETR that builds on DETR and solves the slow convergence and limited spatial resolution problem while getting impressive results.
The authors should think about comparing with other linear attention mechanisms to show the applicability of the method.

Decision: Accept (Oral)