Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: As a new framework for object detection, DETR is very important. However,  it suffers from slow convergence and limited feature spatial resolution. This paper proposes deformable attention, which attends to a small set of sampling locations rather than all the locations in the original DETR. Besides, the paper applies multi-scale deformable attention for better results.

The paper is well-written and obtains very impressive results. Traning for only 50 epochs, deformable Detr obtain results similar to DETR which is trained for 500 epochs. By implementing a two-stage detector based on deformable Detr, the paper obtain state-of-the-art object detection results with a very high AP (52.3) on AP.

 A few suggestions for improving the paper are given as follows.
 (1) The training and testing times could be reported in the paper, which is useful for other researchers to implement and use this method.
 (2) Some related methods on sparse connected self-attention/transformer [a,b,c] should be cited and discussed.

 [a] Representative Graph Neural Network, CVPR 2020
 [b] Dynamic Graph Message Passing Networks, CVPR 2020
 [c] CCNet: Criss-Cross Attention for Semantic Segmentation in ICCV 19 & TPAMI 2020

review_title: It solves the slow convergence problem in the recent successful DETR framework and obtains SOTA results

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:

This paper proposes Deformable DETR with multi-scale deformable attention modules to solve the problems of DETR: slow convergence and limited feature spatial resolution.  In particular, it has faster convergence and achieves better performance(especially on small objects) than DETR.

Reasons for score:

Overall, I vote for accepting. I like this paper because it solves the main problems suffered by DETR. My major concern is about the clarity of the paper and some additional ablation studies (see cons below). Hopefully the authors can address my concern in the rebuttal period.

Pros:

1.This paper solves the main problems suffered by DETR: slow convergence and limited feature spatial resolution. In my opinion, it makes DETR more practical.

2.The proposed Deformable DETR can obtain multi-scale features without a huge cost. In this way, it can be optimized easily and detect objects precisely, especially small objects.

3.This paper also introduces some improvements and variants to boost the performance of Deformable DETR.

Cons:

1.In the experiments, focal loss is used for bounding box classification. What is the reason for this choice? In addition, the number of object queries is increased from 100 to 300. Why? In the test, how to choose 100 objects from 300 objects? 

2.From Table 1, we can find DETR (500 epochs) has better performance than Deformable DETR on large objects (61.1 vs. 58.0), though the overall performance of Deformable DETR is better. Why?

3.In the Table 1, there is only DETR-DC5+ (50 epochs). Could you provide DETR-DC5+ (500 epochs) ?

Questions during rebuttal period:

Please address and clarify the cons above
review_title: Official Blind Review #4

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The main contribution is a new attention module called deformable attention module. Like deformable convolution, it adds a translation term into the expression of the transformer, allowing a sparse spatial sampling. The resulting model is very interesting in terms of convergence and complexity compared to the original DETR. A Multi-scale deformable attention module is also proposed. it needs to add a scale function in the attention module equation. Experiments shows that it increases the AP detection rate on MSCOCO compared to FasterR-CNN and DETR. 
Contributions are clearly state and validated. The complexity study is very interesting and shows the interest of deformable attention module.
Figure 1 presents a general view of the model. Since the deformable attention module is the core of the contribution, it should be interesting to add a figure dedicated to this component. Combined with eq.2, it will give a better understanding of the method. 
It seems that the Axial-DeepLab paper presented in ECCV-2020 misses in the references. This paper proposes a simple strategy for attention modules that also reduce complexity. 
Results clearly show that deformable DETR provides better AP than DETR for less training-epochs. Moreover, the convergence is better than for Faster R-CNN (FPN). 
One of the concerns with deformable convolution is that the computation speed is slower than classical convolution. The same drawback appears with deformable attention modules. Fps decrease from 26 to 19 compared to DETR. It should be interesting to also report fps in the state of the art comparison table 3. 
review_title: This paper introduces Deformable DETR: A modified version of the recent DETR paper for end-to-end object detection with transformers

final_score: rating: 7: Good paper, accept
review_content: ### Summary
This paper aims to improve a very recent detection model -- DETR, which suffers from two issues: long training time and limited feature spatial resolution. Targeting these issues, this paper proposes (1) deformable attention (2) multi-scale processing (inputs/attention) for DETR, which have greatly reduced the  training time and improved the performance. Moreover, it develops two additional modules "iterative bounding box refinement" and  "two-stage framework", which help to achieve the SOTA detection results on COCO.

### Pros
 - This paper is insightful as it studies the two biggest issues of DETR. The vanilla attention is slow but hard to replace as the performance may be harmed. Similarly, it's not super straightforward to incorporate multi-scale processing into DETR due to the novel framework architecture. This paper has addressed these two issues, demonstrated outstanding performance on COCO, and properly studied the effectiveness of each single module. 
 - The deformable attention module is a new kind of attention implementation. It samples a fix number of feature points on spatial feature map and thus greatly reduces the complexity. 
 - The additional techniques make lots of sense to me, and improve the results greatly. I'm impressed that transformer based model can achieve very competitive  as shown in Tab.3. 

### Cons
 - The deformable attention is considered the most important contribution of this paper and thus should be studied more thoroughly. Specifically, comparisons between this module and other "linear"/"efficient" implementations of attention should be performed. As discussed in related work, baselines like "pre-defined sparse attention", "data-dependent sparse attention", and "low-rank property attention" should be considered. This kind of comparisons will help us to understand better about the proposed module, and also to researchers from other field who are interested in using this.
  - Another missed baseline is DETR with multi-scale input and attention (vanilla version) for decently long epochs. I would want to know whether the proposed multi-scale thing could help the vanilla DETR.
 - Does the level embedding help? I don't find experiments to support this design choice.
 - If small objects are the issue, why not have a feature map of H/4 x W/4? Why are multi-scale feature maps constructed like in Appendix Fig.3? 

### Minor
 - There are lots of notations. It's pretty hard for me to remember "mlqk, A, W, \phi, ...". I would recommend the authors providing a lookup table in Appendix.
 - Personally, I think the discussion about FPN in main paper is distractive. The authors may want to move all of them into Appendix.

### Questions
 - Do the last 5 rows in Tab.3 use "iterative bounding box refinement" and  "two-stage framework"? The difference between these 5 lines are just the backbone? 
 - Fig.2 shows that Deformable DETR keeps improving over time. It doesn't converge at Ep.50. Why stick to this number in most experiments? How many epochs do the models in Tab.3 get trained?
 - In Tab.1, why do "iterative bounding box refinement" and "two-stage Deformable DETR" have no influence on FPS? Shouldn't the speed become slower as they are iterative and stage-wise. Does this FPS mean training FPS of Deformable DETR only? If that's the case, please also provide the total training time of everything.
- Please guide me to the definition/explanation of "DETR-DC5". Does it mean the backbone in DETE is changed to ResNet-DC5?


review_title: Good paper; solid results on established benchmark; improved a recent detection model; some evaluations are missing;

Meta-review: Accept. The paper proposes Deformable DETR that builds on DETR and solves the slow convergence and limited spatial resolution problem while getting impressive results.
The authors should think about comparing with other linear attention mechanisms to show the applicability of the method.
Decision: Accept (Oral)