Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes methods to further improve the efficiency of Deformable DETR. It observes that the encoder queries referenced by the decoder account for only 45% of the total, and the detection accuracy does not deteriorate significantly even if only the referenced queries are polished in the encoder block. To this end, this paper proposes Sparse DETR that selectively updates only the queries expected to be referenced by the decoder. An auxiliary detection loss in the encoder also improves the performance. Experiments prove the effectiveness of the method. 
main_review: Overall, I think this is a good paper that has 

a. reasonable motivations and backgrounds

b. reasonable and effective solutions

c. supportive benchmark results and sufficient ablation studies

d. good written and easy to follow. 

It focuses on a specific problem in current DETR based methods and proposes a reasonable solution/direction to further improve the efficiency of the model. 

There are some suggestions to improve the literature: 

a. There is an argument that “the encoder queries referenced by the decoder account for only 45% of the total”. However, I do not observe any statistical analysis and support in the main paper. A more detailed analysis is recommended to add to better support the paper. 

b. The paper uses the word “encoder queries”. It doesn’t have problems. However, the encoder queries (i.e. backbone features) are key/values in the decoder, not the queries. In DETR and Deformable DETR, they all use the word “feature maps”, because from the perspective of the entire Transformer, the query is the learnable object embeddings. I recommend the authors revise this part to make the presentation more clear. 

summary_of_the_review: Overall, this is a good paper that has good motivations and reasonable solutions. The experiments are thorough. I recommend to accept the paper and hope the authors can revise the paper according to my suggestions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to solve the problem of expensive computation on the encoder of Deformable DETR. The hypothesis is the encoder inputs a large number of image feature queries, but only a small number of them are actually referred by the decoder. The paper has shown that with a help with some selection mechanism, the encoder and only inputs partial of the feature queries, such that the computation can be reduced. With some other components, e.g. “top-k decoder queries” and “encoder auxiliary loss”, the Sparse DETR can achieve slightly better performances at fewer computations.
main_review: -The motivation is interesting somehow. The computational and memory complexity of Deformable DETR are indeed some problems. It would be interesting to reduce its complexity but remain accuracy.

-The techniques to achieve the goal, e.g. adding a saliency head or using the decoder saliency map, are kind of empirical and less interesting.

-I am quite confused about the two additional components in Sec. 3.4. I don’t get their motivations and they seem to be irrelevant to the main motivation of this paper. What are their effects? Don’t see detailed ablations on them in the experiments.

-The improvements on both accuracy and speeds are reasonably good, but not very impressive.

-The writing/presentation needs to be improved. There are quite a few confusing/unclear points in the paper, which are difficult to understand.


Some detailed comments:

-in the abstract, it says only 45% queries are referred and not updating them don’t drop the accuracy. How is this implemented? A very naïve baseline? Don’t see it in the experiments.

-The auxiliary loss was first used in [A], instead of GoogleNet.

-For the query selection, once the queries are selected, they are active for all encoders or each encoder has its own selection?

-For the inactive queries, are they inactive for all encoders or they can be inactive for 1st encoder but active for the 2nd?

-For the inactive queries, aren't they not involved in the computation graph at all? how could they be referenced when updating the selected queries?

-Fig. 2 is quite difficult to read.

-For DAM, if you already use the decoder for saliency, how can you not compute the encoder? Aren’t the encoder’s outputs the decoder’s inputs?

-Why need another 4-layer scoring network? Don’t you already use decoder to compute the saliency?

-How to add auxiliary loss to encoder? There are no object queries as input to the encoder, so no detections for encoder.

-In table 1, what’s the deformable detr with a check mark? And what’s Deformable DETR+?


[A] Deeply-Supervised Nets, AISTATS 2015
summary_of_the_review: The motivation is somehow interesting; the techniques are kind of empirical and less interesting; the improvements over Deformable DETR are not very surprising on both accuracy and speeds.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A modification to the recently proposed Transformer-based object detector, DETR, is proposed. The authors are motivated by low performance of DETR on small objects, and high computational cost of self-attention on large feature maps. To improve both, they propose to sparsify input feature maps by learning a classifier whether to include an input feature or not. Supervision for this classifier comes from Transformer decoder attention weights. The authors apply their approach to Deformable DETR and evaluate on the COCO bounding box detection benchmark, and show that it allows to trade detection performance for computational efficiency by tuning the classifier threshold.
main_review: Overall the method seems fairly simple and should be straightforward to implement. The sparisification contributions are well supported experimentally, with ablations and insights.

Contribution #3, the encoder auxiliary loss, is not well supported. The original DETR paper shows that the encoder can be scaled to 12 layers and show improved performance over 6 layers (+1AP), table 2. An additional investigation on what could be the reason behind convergence issues in deep Deformable DETR is required. Training deep Transformer encoder models is extensively studied in NLP, with well known practices like BERT init and pre-norm block structure. The contribution needs to be backed up by convincing evidence on why it would be preferable over the common practices.

It would be beneficial to show results with instance or panoptic segmentation, eq with SOLQ which was build on top of Deformable DETR as well. Would in this case more topk features be required to match performance, would be proposed method still be a useful method to improve computational efficiency? Would Objectness Score be a better sparsification target than DAM in case of masks?

Suggestions to improve writing:
- In table 1, results for R50 with SCRL init could be moved to appendix as they are not directly support the claims.
- The first sentence of the abstract where the authors say "DETR demonstrates competitive performance but low computational efficiency" is confusing, since in table 1 DETR actually has the lowest FLOP count/fastest runtime. I guess what the authors meant was that "DETR demonstrates competitive performance but low computational efficiency on high resolution feature maps". Overall the abstract needs to be significantly improved, it has undefined terms and is very difficult to read. For example, in the sentence "Using the multiscale feature to ameliorate performance.." neither multiscale nor feature is mentioned before.
- Figures 4-7 lack captions explaining what the reader is supposed to understand from them.
- Figures 2-3 would benefit from highlighting inputs and outputs
- The authors use word "polish" when describing Transformer layers applied to feature maps, which is slightly confusing. Perhaps there is a better word for it.
- notation $x_\text{bb}$ is confusing, since $\text{bb}$ typically means "bounding box", and not "backbone".
- Section 3.1: "decoder takes both the refined queries ... " Transformer encoder output is typically called "memory", it would disambiguate encoder output from object queries.


summary_of_the_review: The paper suggests a novel approach for improving computational efficiency of Transformer-based object detectors. Most contributions are supported by experiments, ablations and insights. The contribution with auxiliary losses in encoder requires more investigation. Overall I recommend accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an efficient end-to-end object detection architecture based on Deformable DETR, called Sparse DETR. Its main contribution is solving the computation budget by selecting only a few tokens as queries in attention computing. Sparse DETR can use only 10%-50% of the original encoder query while achieving comparable results. This is done by several components. Firstly, the authors add a subnet named Scoring Net before encoder to predict saliency scores for input. And only the tokens which have top-k scores will be chosen as queries in the encoder attention computing. Then, to train the Scoring Network, this paper use binarized attention weight from cross attention, named DAM, as pseudo labels. It also adds a detection head to predict the class scores for outputs of the encoder. The scores are used to select the appropriate token as the decoder`s query. The extensive experiments show the effectiveness of the proposed method.
main_review: Strengths:
The method is novel and technically sound. As we all know that the queries or tokens are somewhat redundant for DETR-based object detection, this work brings us some new insight of how to efficiently condense queries without performance dropping. Compared to PnP and other dynamic ViTs, this work is more effective and efficient. The writing of this work is also clear and easy to follow.

Weakness:
In table 1, the 50% ratio Sparse DETR is higher (46.3 vs 46.0) than Deformabel DETR under the same training setting. In my understanding, the full queries version of Deformabel DETR is equal to a 100% ratio version of sparse DETR. If so, why the performance is even higher when we only keep 50% of queries. I think this result is deserved for more analysis. Or are there any additional components that can improve the total performance of the full version of the detector? 

summary_of_the_review: Overall, this paper provides valuable insight into reducing queries for DETR-based detectors and I prefer to accept it.
review_title: null

Meta-review: This paper proposes to modify DETR, a recent Transformer-based architecture for object detection.
More precisely,  they propose to sparsify input feature maps by learning an extra classifier to select which input features (few of them) will be used in the attention module. The supervision of this classifier is guided by second extra module  coming from the Transformer decoder attention weights.
The resulting framework, called Sparse DETR, is an efficient end-to-end object detection architecture that allows to overcome the main computational bottleneck of DETR. Sparse DETR can use  only 10%-50% of the original encoder query while achieving DETR comparable results.

Authors tried to answer to all the questions raised during the rebuttal. 
Even if the final scores are still contrasted, most of reviewers are very positive. 
Overall, this paper provides valuable insight into reducing computational complexity (by reducing the number of queries) for DETR-like detectors. The proposed method is novel and technically sound. Even if there are some tricks to make the whole working well, this work is more effective and efficient than previous propositions to handle this complexity problem,  bringing us some new insight of how to sparsify queries without performance dropping.  
All these elements lead me to propose this paper for publication at ICLR.
Decision: Accept (Poster)