Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity
Authors: Byungseok Roh
JaeWoong Shin
Wuhyun Shin
Saehoon Kim
Venue: ICLR 2022 Poster (2022)
Keywords: Transformer Query Sparsification Mechanism
Efficient End-to-End Object Detection

Abstract: DETR is the first end-to-end object detector using a transformer encoder-decoder architecture and demonstrates competitive performance but low computational efficiency. The subsequent work, Deformable DETR, enhances the efficiency of DETR by replacing dense attention with deformable attention, which achieves 10x faster convergence and improved performance. Using the multiscale feature to ameliorate performance, however, the number of encoder queries increases by 20x compared to DETR, and the computation cost of the encoder attention remains a bottleneck. We observe that the encoder queries referenced by the decoder account for only 45% of the total, and find out the detection accuracy does not deteriorate significantly even if only the referenced queries are polished in the encoder block. Inspired by this observation, we propose Sparse DETR that selectively updates only the queries expected to be referenced by the decoder, thus help the model effectively detect objects. In addition, we show that applying an auxiliary detection loss on the selected queries in the encoder improves the performance while minimizing computational overhead. We validate that Sparse DETR achieves better performance than Deformable DETR even with only 10% encoder queries on the COCO dataset. Albeit only the encoder queries are sparsified, the total computation cost decreases by 38% and the frames per second (FPS) increases by 42% compared to Deformable DETR. Code will be released.


Published as a conference paper at ICLR 2022
SPARSE DETR: EFFICIENT END-TO-END OBJECT DE- TECTION WITH LEARNABLE SPARSITY
Byungseok Roh' Jae Shin2*1, Wuhyun Shin ' Saehoon Kim KaaaaBrrain
Lunit
(peter. roh, aider. hsin, sam. kin com twoong shin@lunit.
ABSTRACT
DETR is the first end-to-end object detector using transformer encoder-decoder architecture and demonstrates competitive performance but low computational efficiency high resolution feature maps. The subsequent work, Deformable DETR, enhances the efficiency DETR by replacing dense attention with de- formable attention, which achieves 10x faster convergence and improved perfor- mance. Deformable DETR uses the multiscale feature to ameliorate performance, however, the number of encoder tokens increases by 20x compared to DETR, and the computation cost of the encoder attention remains a bottleneck. In our prelim- inary experiment, we observe that the detection performance hardly deteriorates even if only part of the encoder token is updated. Inspired by this observation, we propose Spar se DETR that selectively updates only the tokens expected be referenced by the decoder, thus help the model effectively detect objects. In ad- dition, we show that applying an auxiliary detection loss on the selected tokens the encoder improves the performance while minimizing computational overhead. We validate that Sparse DETR achieves better performance than Deformable DETR even with only 10% encoder tokens the COCO dataset. Albeit only the encoder tokens are sparsified, the total computation cost decreases by 38% and the frames per second (FPS) increases by 42% compared to Deformable DETR. Code is
at Nlttess//gEtpp..co///kkkesssssss
detr
INTRODUCTION
In recent years, we have witnessed the dramatic advancement and the success of object detection in deep learning. Diverse object detection methods have been proposed, but the existing algorithms that perform positive matching with the ground truth as heuristic way require mo--maximmm suppres- sion (NMS) post-processing of near-duplicate predictions. Recently, Cariio et al.) (2020) has intro- duced fully end-to-end detector DETR by eliminating the need for NMS post-processing through a set-based objective. The training objective is designed by employing the Hungarian algorithm that considers both classification and regression costs, and achieves highly competitive performance. However, DETR is unable to use multi-scale features such as feature pyramid networks (Lin et al.) 2017]. which are commonly used object detection to improve the detection of small objects. The main reason is increased memory usage and computation by adding Transformer (Vaswani et al.) 2017) architecture. As a result, its ability to detect small objects relatively poor.
To address this problem, Zhu et al. (2021) has proposed a deformable-attemee deformable convolution (Dai et al. (017) and reduced the quadratic complexity to linear complex- ity through key sparsification in the attention module. By using deformable attention, deformable DETR addresses the slow convergence and high complexity issue of DETR, which enables the en- coder to multi-scale features small objects. However, using the multi-scale features as an encoder input increases the number of tokens to be processed by about 20 times. Eventually, despite efficient computation for the same
inspired by the
input and significantly improves performance on detecting
"qqual contribution. 'Corresponding author. Work is done during an internship KakaoBrain.

---

Published as a conference paper at ICLR 2022
token length, the overall complexity increases back again, making the model inference slower even than vanilla DETR.
In general, natural images often contain large background regions irrelevant to the objects of interest, and accordingly, in end-to-end detectors, the tokens corresponding to the background also occupy a significant portion. In addition, the importance of each regional feature is not identical, which has been proven by the two-stage detectors successfully do their job by focusing only on the foreground. suggests that there exists considerable regional redundancy that can be reduced in the detection tasks and seeking to devise an efficient detector focusing on the salient regions is necessary and natural direction. In our preliminary experiments, we observe the following: (a) during inference of cully-converged Deformable DETR model on the COCO validation dataset, the encoder tokens referenced by the decoder account for only about 45% of the total, and (b) retraining new detector while updating only the encoder tokens preferred by the decoder from another fully-trained detector, barely suffers performance loss(0.1 1 AP degradation). See Appendix (A. for the details.
Inspired by this observation, we propose learnable decoder cross-attention map predictor to spar- sify encoder tokens. In the existing methods (Carion et al. 2020; Zhuu et al. [202T). ,. the encoder takes all the tokens, i.e. the backbone features combined with corresponding positional embeddings, as input without discrimination. Meanwhile, our approach distinguishes encoder tokens to be refer- enced later in the decoder and considers only those tokens in self-attention. Therefore, this can significantly reduce the number of encoder tokens involved in the computation and reduce the total computational cost. We further propose the encoder auxiliary loss for selected encoder tokens to improve detection performance while minimizing computational overhead. The proposed auxiliary loss not only improves performance, but also allows training a larger number of encoder layers.
Extensive experiments on the COCO 2017 benchmark (Lin et al. 2014) demonstrate that Sparse DETR effectively reduces computational cost while achieving better detection performance. With- out bells and whistles. Sparse DETR using Swin-T (Liu et al. (021) backbonn achieves 48.2 AP with 38% reduction of the entire computational cost compared to the 48.0 AP bascline and 49.2 AP with 23% reduction. In the case of the experiment that achieves 48.2 AP using only 10% of encoder tokens, the computational cost of the transformer encoder block reduced by approximately 82%.
We summarize our contributions as follows:
We propose encoder token sparsification method for an efficient end-to-end object detector, by which we lighten the attention complexity in the encoder. This efficiency enables stack- ing more encoder layers than Deformable DETR, leading to performance improvement within the same computational budget.
We propose two novel sparsification criteria to sample the informative subset from the en- tire token set: Objectness Score (OS) and Decoder cross-Attention Map (DAM). Based on the decoder cross-altention map criterion, the sparsified model preserves detection perfor- mance even when using only 10% of the whole tokens.
We adopt an encoder auxiliary loss only for the selected tokens. This additional loss not only stabilizes the learning process, but also greatly improves performance, with only marginally increased training time.
2 RELATED WORK
Efficient computation in vision transformers. It a well-known problem that the attention com- putation in Transformers incurs the high time and memory complexity. The vision transformers need digest even bigger token sets as input so that large body of works (Parmar et al.) 2018 Child
let al. 2019a| Ho et al. 2019 Wang et al.) 2020: Katharopoulos et al.) 2020 Choromanski et al. Kitaev et al.J (020)) has been proposed lightweight attention mechanisms for them. Most of those works shed light on the complexity that resides only in single-scale attention module, which hinders direct extension to the multi-scale features generally required in object detection.
2021
One of the promising approaches for the lighter transformer attention is input-dependent token spar- sification. Dyn (Rao et al. 2021) and IA-RED2 (Pan et al., 2021). similar to our work, both propose jointly-learned token selectors generating the sparsity patterns to be overlaid on the input tokens. Those approaches mainly focus on sparsifying a backbone network evaluated on the classification tasks, while our interest lies in sparse encoder of the end-to-end object detectors.
2

---

Published as a conference paper at ICLR 2022
On the other hand, there has been line of works sharing the spirit with ours in that they aim at sparse transformers in the DETR-based framework. Deformable DETR (Zhu et al. (021)) conducts sparse attention computation by sampling only fraction of the entire key set with learnable 2-d off- sets, which enables to use multi-scale feature maps with a reasonable computational cost. It can be viewed as key sparsification method but with dense queries, while our approach further reduces the query set pursuing even more sparsity. PnP-DETR (Wang et al. (202I) shortens the token length of the transformer encoder by introducing the Polling and Pull (PnP) module sample the foreground tokens and condense the background tokens into a smaller set. However, their method cannot naively be integrated with Deformable DETR, since their sparsification breaks the 2d spatial structure of the token set assumed in the deformable attention. On the contrary. Sparse DETR preserves the 2d sample space of the set and can be seamlessly combined with the deformable attention, which facil- itates handling the multi-scale features. Thus, our approach gets benefits from both the deformable key sampling and the proposed query sparsification. Most of all, we propose explicit objectives for the token selection network, whereas the aforementioned works have explicit objective implying their beliefs in good selection strategy, merely relying on the final detection objective.
Auxiliary Loss. Auxiliary loss (Lee et al. 2015 Szegedy et al. 2015) is widely adopted to deliver gradients to the early layers of deep networks. DETR variants employ auxiliary Hungarian matching objectives at the end of every decoder layer with extra FFN heads so that each decoder layer directly lears detect the correct number of objects out of the decoder's outputs. Unlike the decoder's object queries whose number is relatively small(e.g. 300), the number of encoder's ' tokens has much larger scales when using multi-scale features. Thus, extending the layerwise auxiliary loss to the multi-scale encoder increases the training time cost by feeding too many tokens to the attached FFN heads. In Sparse DETR, thanks to the sparsity already induced in the encoder, we can instantly economize that cost while enjoying the auxiliary gradients in wider range of intermediate layers.
APPROACH
In this section, we present our main contributions: (a) formulating generalized saliency-based token sparsification scheme for the encoder, (b) proposing the effective saliency criteria with which that scheme can practically work, and (c) employing the encoder auxiliary losses and the top-k decoder query selection to improve the performance. Before describing the details, we revisit the key components of DETR (Carion et al. 2020) and Deformable DETR (Zhu et al. 2021.
3.1 PRELIMINARY
DETR. DETR takes the flattened spatial feature map Xteat E R NxD from a backbone network into the transformer encoder, where N denotes the number of tokens (i.e. features) and D denotes token dimension. The encoder iteratively updates Xiea by several vanilla self-attention modules. Then, the transformer decoder takes both the refined encoder tokens (i.e. encoder output) and M learnable object queries {q: i as inputs and predicts a tuple of a class score E [0, 1J and a bounding box b E [0, 1] for each object query qi, denoted { = {(c,, b,)). where C denotes the number of classes. All components including the backbone network are jointly trained by performing the bipartite matching between the ground truth } and predictions {
Deformable DETR. Deformable DETR replaces the vanilla dense attention, which is the main computational bottleneck in DETR, with deformable attention module. This significantly reduces the computational cost and improves the convergence. Suppose that we have the same size of set of queries (denoted as 22y) and set of keys (denoted as Ω2), which means |2,11 = [4 N. The conventional dense attention computes the attention weight Agk for every pair ((g, k) q E S2.. 94). resulting in quadratic complexity with respect to N. Deformable attention reduces this quadratic complexity into the linear one by only considering relevant keys for each query. Specifically, deformable attention computes attention weight Agt for all queries and small set of keys: {(g k)) E P.y. i E Pigt }. where Pyyk C 9 and [Rap = K & N.
Due to this key sparsification, Deformable DETR is able to the multi-scale features of the back- bone network, improving the detection performance of small objects significantly. Paradoxically, us- ing the multi-scale feature increases the number of tokens in the transformer encoder by about x0x
3

---

Published as a conference paper at ICLR 2022
Key
Key
Query
Query
Query
(a) DETR
(b) Deformable DETR
Sparse DETR
Figure 1: Attention complexity. The circles in the square matrix represent the attention between keys and queries. The gray/white circles correspond to preserved/removed connection respectively, and darker gray the diagonal positions means where the token attends to itself. (a) Dense attention in DETR takes quadratic complexity. (b) Deformable DETR uses key sparsification, thus takes linear complexity. (c) Sparse DETR further uses query sparsification. Attention in Sparse DETR also takes linear complexity, but is much lighter than Deformable DETR's.
compared to DETR, making that the encoder becomes the computational bottleneck of deformable DETR This motivates us develop sparsification method to reduce the number of tokens in the encoder aggressively, which is described in the next sections.
3.2 ENCODER TOKEN SPARSIFICATION
In this section, we introduce our token sparsification scheme that the encoder module selectively refines small number of encoder tokens. This encoder token subset obtained from the backbone feature map Xfeat with a certain criterion, which is described in the subsequent section. For features that are not updated in this process, the values of Xfent are passed through the encoder layers without being changed.
Formally. suppose that we have a scoring network g Rd + R. that measures saliency of each token in Xiea- We then define p-salient regions Di as the top-p% tokens with the highest scores, for a given keeping ratio p. i.e. S [2g] p. |12g1 < |S2g1 N. Then, the i-th encoder layer updates the features x;-1 by:
X LN(FFN(Z() + +z)) j Sg where z LN(DefAtm(x(_ + x/_1),
j$ D2
x/-
(1)
where DefAttn refers to deformable attention, LN to layer normalization (Ba et al. 2016). and FFN feed-forward network. Even in the case of unselected tokens, the values still passed through the encoder layer, so they can be referenced as keys when updating the selected tokens. This means that the unselected tokens can hand over information the selected tokens without losing the value of themselves while minimizing the computational cost. Here, we use deformable attention for refining tokens in 12g, but the proposed encoder token sparsification applicable regardless of which attention method the encoder uses.
Complexity of Attention Modules in Encoder. Deformable DETR reduces the attention com- plexity through key sparsification, and we further reduce the attention complexity through query sparsification, as shown in Fig. [ Conventional dense attention in DETR requires quadratic com- plexity O(N2), where N is the query length. Deformable attention requires linear complexity O(NK), where K < N is the number of keys for each query. Sparse attention requires only O(SK), where S & N is the number of salient encoder queries.
3.3 FINDING SALIENT ENCODER TOKENS
In this section, we introduce how to find a salient token set Qg from backbone feature Xfeat- We propose a method for determining saliency using a cross attention map from the transformer de- coder. Before presenting objectness scores obtained from a separate detection head. The limitation of this simple approach motivates us develop an advanced one, which is described in the following paragraph.
approach, we first discuss simple yet effective method based on the
4

---

Published as a conference paper at ICLR 2022
Salient Token
Deformable Cross-Attention
Prediction
BCE
Scoring Network
Botten Bingrize
Reference
Sum
pib
Attention weight
DAM
Detection Pipeline
Hungarian poriian Loss
Input Image
Backbone
Sparsify
Encoder
Decoder
Donse
Sporse Tokens
Decoder
Tokens
Queries
Figure 2:
Illustration on how to learn a scoring network by predicting binarized Decoder cross- Attention Map (DAM), where dashed orange arrow means a backpropagation path. The bottom box shows the forward/backward passes in Sparse DETR, and the top boxes present how to construct DAM for learning the scoring network. See Appendix A.7 for implementation details of scoring net.
Objectness Score. Measuring objectness per each input token (i.e. feature Xtea) of encoder is very natural to determine which ones from backbone feature should be further updated in the transformer encoder. It is widely known that feature map from a pretrained backbone network is able to find the saliency of objects, which is why the region proposal network (RPN) has been successfully adopted many object detectors (Ren et al.] 2015 Dai et al. 2016 He et al.) (017). Inspired by this observation, we introduce an additional detection head and Hungarian loss to the backbone feature map, where the structure of the newly added head is the same as the one of the final detection head in the decoder. Then, can select the top-p% encoder tokens with the highest class scores as a salient token set S2g. This approach is effective to sparsify encoder tokens, but we believe that it is sub-optimal the transformer decoder, because the selected encoder tokens from the separate detection head are not explicitly considered for the decoder.
Decoder Cross-Attention Map. We consider another approach to select subset of encoder to- kens that are highly relevant to the decoder in a more explicit manner. We observe that the cross- attention maps from the transformer decoder could be used for measuring the saliency, because the decoder gradually attends subset of encoder output tokens that are favorable to detect objects as training continues. Motivated by this, we introduce a scoring network that predicts a pseudo ground- truth of the saliency defined by decoder ross-attention maps, and use it to determine which encoder tokens should be further refined on the fly. Fig. 2 summarizes how to train a scoring network, and details presented below.
To determine the saliency of each input token of encoder Xfcats we have to aggregate the decoder cross-attentions between all object queries and the encoder output. This procedure produces single map of the same size as the feature map from the backbone, which is defined as Decoder cross- Attention Map (DAM). the case of the dense attention, DAM can be easily obtained by summing up attention maps from every decoder layer. In case of deformable attention, for each encoder token, the corresponding value of DAM can be obtained by accumulating the attention weights of decoder object queries whose attention offsets are directed toward the encoder output tokens. Refer to the Appendix A.2 for the details in the DAM creation.
To train the scoring network, we binarize DAM so that the top-p% (by attention weights) of encoder tokens is only retained. This is because our goal is to find a small subset of encoder tokens that decoder references the most, rather than precisely predicting how much each encoder token will be referenced by the decoder. This binarized DAM implics the one-hot target that indicates whether each encoder token is included in the top-p% most referenced encoder tokens. Then, consider a 4-layer scoring network to predict how likely a given encoder token is included in the top-p % most referenced tokens, and the network is trained by minimizing the binary cross entropy (BCE) loss between the binarized DAM and prediction:
Ldam N
(gD(g(Xtea)) DAM
5

---

Published as a conference paper at ICLR 2022
Detection Head
Backbone
(2)
x/
Decoder
Aux. Head
Aux. Head
xN.
Encoder
FFN
Aux. Head
Positional
Def. Cross-Attn.
Bipartite Matching
Encoding
FFN
Scoring
Def. Self-Attn.
Self-Artn.
Network
cls,, reg. loss
Figure 3: Sparse DETR architecture. Sparse DETR introduces three additional components: (a) the scoring network, (b) auxiliary heads in the encoder, and (c) the auxiliary head select the top-k tokens for the decoder. Sparse DETR measures the saliency of encoder tokens by using the scoring network, and selects the top-p% tokens, which is referred as (I) the diagram. After refining only the selected tokens in the encoder blocks, the auxiliary head selects the top-k tokens from the encoder output, which served as the decoder object queries. This process is referred to as (2) in the diagram. In addition, we remark that additional auxiliary heads in each encoder block play key role in achieving improved performance. Only sparsified encoder tokens are passed to the encoder auxiliary heads for efficiency. All auxiliary heads in the encoder and decoder are trained with Hungarian loss as described Deformable DETR (Zuu et al. 202T).
where DAM means the binarized DAM value of the ith encoder token.
One may say that since DAM in the early phase of training is not accurate, pruning the encoder tokens based on the result in the decoder degrades the final performance or hurts the convergence. However, we empirically observe that the optimization is very stable even in the early phase of training, and achieves better performance compared to the method based on objectness score. We describe detailed comparisons in the experiments section.
3.4 ADDITIONAL COMPONENTS
In this section, we introduce two additional components: (a) auxiliary losses on the encoder tokens and (b) top-k decoder queries selection. We empirically observe that these greatly help improve final performance and stabilize the optimization. The overall architecture of Sparse DETR including these components is depicted in Fig. 3
Encoder Auxiliary Loss. In DETR variants, auxiliary detection heads attached to decoder layers, but not to encoder layers. Due to a significantly larger number of encoder tokens (about 18k tokens) compared to decoder tokens (about 300), encoder auxiliary heads will heavily increase the computational cost. In Sparse DETR, however, only part of encoder tokens are refined by the encoder, and adding auxiliary heads only for sparsified encoder tokens is not big burden.
We empirically observe that applying an auxiliary detection head along with Hungarian loss on the selected tokens stabilizes the convergence of deeper encoders by alleviating the vanishing gradient issue and even improves the detection performance. We conjecture that, following the analysis in Sun al. (02T), applying Hungarian loss the intermediate layers helps distinguish the confusing features in the encoder, which contributes to the detection performance in the final head.
Top-k Decoder Queries. In DETR and Deformable DETR, decoder queries are given by only learnable object queries or with predicted reference points via another head after the encoder. In Efficient DETR (Yao et al. 2021). decoder takes a part of encoder output as input, similar to Rol Pooling (Ren et al. 2015). Here, auxiliary detection head is attached to the encoder output Xene and the head calculates the objectness (class) score of each encoder output. Based on the score, the top-k encoder outputs passed decoder queries, similar to objectness score- based encoder token sparsification. Since this outperforms the methods based on learnable object queries or the two-stage scheme, include this top-k decoder query selection in our final architecture.

---

Published as a conference paper at ICLR 2022
Table 1: Detection results of Sparse DETR on COCO 2017 val set. Top-k & BBR denotes that we sample the top-k object queries instead of using the learned object queries (ao et al. 2021). and perform bounding box refinement in the decoder block (Zhu al.) (021). respectively. Note that the proposed encoder auxiliary loss is only applied to Sparse DETR. FLOPs and FPS are measured the same way as used Zhu et al. (2021). The results marked by t, are the reported ones from Zhu et al.
2021)
and Wang et al. [202] respectively.
Keeping ratio (p)
Top-k & BBR
Method
Epochs
AP
AP50
AP75
APs
APM AP,
params
FLOPs
FPS
ResNet-50 backbone:
F-RCNN-FPN
109
N/A
42.0
62.1
45.5
26.6
45.4
53.4
42M
180G
26
DETRI
500
100%
42.0
62.4
44.2
20.5
45.8
61.1
41M
86G
28
DETR-DC5!
500
100%
43.3
63.1
45.9
22.5
47.3
61.1
41M
187G
12
500
33%
41.1
43.7
20.8
60.0
PaP-DETR!
500
50%
41.8
21.2
45.3
60.8
500
33%
42.7
62.8
45.1
22.4
46.2
60
PaP-DETR-DC51
500
50%
43.1
63.4
45.3
22.7
46.5
61.1
50
100%
43.9
62.8
47.8
26.1
47.4
58.0
40M
173G
19.1
Deformable- DETR
100%
28.2
49.1
61.0
41M
177G
18.2
50
10%
45.3
65.8
49.3
28.4
48.3
60.1
4IM
105G
25.3
50
20%
45.6
65.8
49.6
28.5
48.6
60.4
41M
113G
24.8
Sparse-DETR
50
30%
46.0
65.9
49.7
29.1
49.1
60.6
41M
121G
23.2
50
40%
46.2
66.0
50.3
28.7
49.0
61.4
41M
128G
21.8
50%
46.3
66.0
29.0
60.8
41M
136G
20.5
Swin-7 backhone:
DETR
500
100%
45.4
66.2
48.1
22.9
49.5
65.9
45M
92G
26.8
50
100%
45.7
65.3
49.9
26.9
49.4
61.2
40M
180G
15.9
Deformable-DEr
100%
48.0
$2.0
30.3
63.7
41M
185G
15.4
50
10%
48.2
69.2
52.3
29.8
51.2
64.5
41M
113G
21.2
50
20%
48.8
69.4
53.0
30.4
51.9
64.8
41M
121G
20.0
Dearse-DETR
50
30%
49.1
69.5
53.5
31.4
52.5
65.1
41M
129G
18.9
50
40%
49.2
69.5
53.5
31.4
$2.9
64.8
41M
136G
18.0
50%
49.3
69.5
53.3
32.0
52.7
64.9
41M
144G
17.2
4 EXPERIMENTS
We compare Sparse DETR with the conventional object detectors, including the recently proposed ones in the DETR family. In addition, conduct ablation study to support claims in Sec- tion 3) presenting the performance comparison between token selection criteria (OS vs. DAM), the effectiveness of the encoder auxiliary loss, and the dynamic sparsification during inference.
Implementation Details. We use ResNet-50 (He et al. 2016) and Swin Transformer (Liu et al., 2 as pre-trained backbone networks, where Swin Transformer is one of the state-of-the-art chitecture in the ViT family. We stack 6 encoder and 6 decoder layers, each with an auxiliary head at the end. We train the model on V100 GPU machine with total batch size of 16, for 50 epochs, where the initial learning rate is 0.0002 and decayed by 1/10 at the 40 epoch. Unless otherwise specified, we use the same hyperparameters as Deformable DETR.
4.1 COMPARISON WITH OBJECT DETECTION BASELINES
Baselines. We compare Sparse DETR with Faster RCNN with FPN (Lin et al., 2017). DETR (Car- ion et al. 2020), Deformable DETR (Zhu et al. (02T). and PnP DETR (Wang et al. 202Th. We also compare with DETR and Deformable DETR that uses Swin-Time Tiny (Liu et al. (021) Here, for brevity, we denote Deformable DETR with the top-k object query selection and bound- ing box refinement, keeping ratios of 10%, 20%, 30% 40%, and 50%, using DAM criterion. We demonstrate detection performance and inference costs COCO val2011 dataset.
backbone.
Deformable DETR+ In Sparse DETR, encoder tokens sparsified with
Result. Table [ shows the results Sparse DETR and the other baselines on COCO va12017 set. Remarkably, the ResNet-50 backbone, Sparse DETR with keeping ratio over 30% outper-

---

Published as a conference paper at ICLR 2022
Mm
46
0.68
0.92
W
COCO mAP
45
48
0.90
DAM
0.60
0.88
Full
DMM
0.86
05
43
45
0.48
10%
20%
50%
10%
20%
30%
50%
20
40
20
40
Keeping Ratio P
Keeping Ratio
Training Epochs
Training Epochs
(a) ResNet-50
(b) Swin-T
(a) Keeping ratio 10%
(b) Keeping ratio 40%
Figure 4: Selection criteria. Comparison of the performance with respect to encoder token selection criteria for different backbones.
Figure 5: Correlation graph.
Correla-
tion graphs of OS and DAM during training.
forms all the baselines while minimizing the computational cost. Even with the keeping ratio re- duced down to 10%, Sparse DETR still performs better than most baselines except for Deformable DETR+ More surprisingly, on the Swin-T backbone, Sparse DETR only with the keeping ratio 10% outperforms all the baselines with no exception, while improving FPS by 38% compared to Deformable DETR+
Remark that, compared to the most competitive baseline, Deformable DETR+ the improvement in APL is relatively noticeable on the Swin-T backbone even under the extreme sparsity of 10%, while the performance gap on the ResNet-50 backbone comes evenly from different sizes of objects. We conjecture that it is because single token in Swin-T can hold wider region of information than the one in ResNet-50, so even we aggressively sparsify the encoder token, the network seems to have enough information to detect objects.
4.2 COMPARISON BETWEEN TOKEN SELECTION CRITERIA
Baselines. To verify the benefits of the proposed saliency criteria, compare three token spar- sification criteria: random, Objectness Score (OS), and Decoder cross-Attention Map (DAM). The random baseline samples fixed ratio of arbitrary tokens. Note that the proposed encoder auxiliary loss is applied for all the methods.
As illustrated in Fig. the random strategy suffers noticeable performance degradation. On the other hand, the DAM-based model outperforms the OS-based model every ratio and almost catches up with the non-sparse baseline when using 50% of encoder tokens. See the Appendix A.4 for detailed results this experiment.
Result.
To analyze the reason that DAM-based model outperforms its counterpart, measure the overlap between the encoder tokens referred by the decoder and the tokens refined by the encoder. As metric, compute scalar correlation Corr
Corr = ExERonor DAM,
Exerg DAM,
where Up is the encoder token set referred by the decoder and DAM, is the DAM-value corre- sponding to token æ. This Corr metric indicates the ratio of tokens polished by the encoder among the tokens referred by the decoder.
Fig. demonstrates that Corr of based model rises higher than that of OS-based model. This result implies that DAM-based model is more suitable sparsification method for the decoder, because the tokens referenced by the decoder are explicitly refined in the encoder, which achieves better detection performance. See the Appendix f.4 detailed results of this experiment.
8

---

Published as a conference paper at ICLR 2022
49.5
280
40% Full
20h
49.0
COCO mAp
49
20%
1on
COCO mAP
48
GFLOP
200
-51%
48.5
Training
A
10%
20%
w/ enc aux. loss
46
loss
120
106
20%
40%
100
20
40%
0
47.0
50%
12
# Encoder Layers
# Encoder Layers
10
20
30
40
50
Inference Keeping Ratio p
(a) mAP
(b) GFLOPs
Figure 6: Ablation # encoder layers.
Figure 7: Dynamic sparsification.
4.3 EFFECTIVENESS OF ENCODER AUXILIARY LOSS
Owing to the sparsified token set in our model, we can apply the auxiliary loss to the encoder layers without sacrificing too much computational cost. Apart from improved efficiency and performance, find another benefit of the encoder auxiliary loss that allows safely stack more encoder layers without failing to converge.
As shown in Fig. 6 the encoder auxiliary loss only enhances detection performance, but also consistently increases detection performance as the encoder layers doubled to 12. However, we ob- serve that the training without its assistance utterly fails when using 12 encoder layers. We argue that gradient propagated through decoder cross-attention vanishes as we stack more encoder lay- ers, thus intermediate gradients from the auxiliary loss are required. The observations reported in Appendix supports this assertion and detaiil the results of Fig. 6
4.4 DYNAMIC SPARSIFICATION FOR INFERENCE STAGE
To deploy the models in various hardware conditions real-world applications, one often should retrain them at different scales according to the performance-computttion trade-off required. We evaluate if our model trained with fixed sparsity can adapt well to dynamic sparsity at inference time to check out Sparse DETR avoid that hassle. Figure 7 shows the performance under the var- ied keeping ratio (p) during inference when the model trained using the Swin-T backbone and 30% of encoder tokens with the DAM- method. When the inference keeping ratio is small, the per- formance of dynamic sparsification is slightly degraded, but the overall performance is satisfactory various keeping ratios given that only single model is used.
PnP DETR introduces dynamic ratio training to achieve similar performance to the fixed keeping ra- tio counterpart. However, without the additional trick, it suffers significant performance degradation, for instance, 5.0 AP drop when training/inference keeping ratio is 0.33/0.5, despite the increased number of encoder tokens. On the contrary, Sparse DETR achieves 0.2 AP improvement in simi- keeping ratio is 0.3/0.5. To conclude, our method shows better robustness compared to PnP DETR without further treatment, showing a greater potential of dynamic adaptation to different hardware environments. Note that any technique such as dynamic ratio training is orthogonal to our method and introducing may bring even more robustness.
lar condition where the training/inferenn
CONCLUSION
In this paper, we have presented encoder token sparsification algorithm that lowers the computational cost of the encoder, which is computational bottleneck in the DETR and Deformable DETR. By doing so, the proposed Sparse DETR architecture outperforms the Deformable DETR even when using only 10 % of the encoder token, and decreases the overall computation by 38%, and increases the FPS by 42% compared to the Deformable DETR. We hope that our proposed method will provide insights effectively detect objects in the transformer structure in the future.

---

Published as a conference paper at ICLR 2022
REFERENCES
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint
arXiv:
arXiv:1607.066: (600/6/0065555 06750, 2016.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling, In ICLR (Poster). OpenReview.net, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904.10500 2019a.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1 abs/1904. 10509, 2019b.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with per- formers. In ICLR. OpenReview.net, 2021.
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-FCN: object detection via region-based fully convolutional networks. In NIPS, pp. 379-387, 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. ICCV, 2017.
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, (bbs/1606004155, 2016.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. CoRR, abs/19 12180. 2019.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 5156-5165. PMLR, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR. OpenReview.net, 2020.
Chen-Yu Lee, Saining Xie. Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- Supervised Nets. In Guy Lebanon and S. V. N. Vishwanathan (eds.), Proceedings of the Eigh- feenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceed- ings of Machine Learning Research, pp. 562-570, San Diego, California, USA, 09-12 May 2015. PMLR.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
10

---

Published as a conference paper at ICLR 2022
Bowen Pan, Yifan Jiang, Rameswar Panda, Zhangyang Wang, Rogério Feris, and Aude Oliva. IA-RED2: Interpretabiltty-awaaa redundancy reduction for vision transformers. arXiv preprint arXiv:2106. 12620, 2021.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML,, volume 80 of Proceedings of Machine Learning Research, pp. 4052-4061, PMLR, 2018.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. DynamicViTI efficient vision transformers with dynamic token sparsification. arXiv preprint arXiv:2106.02034
2021.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS, pp. 91-99, 2015.
Byungseok Roh, Wuhyun Shin, Ildoo Kim, and Sungwoong Kim. Spatilly consistent representation learning. In CVPR. IEEE, 2021.
Peize Sun, Yi Jiang, Enze Xie, Wenqi Shao, Zehuan Yuan, Changhu Wang, and Ping Luo. What makes for end-to-end object detection? ICML, volume 139, pp. 9934 9944, 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierr Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pp. 1-9. IEEE Computer Society, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan L. Yuille, and Liang-Chied Chen. Axial-decplab: Stand-alone axial-attention for panoptic segmentation. In ECCV (4), volume 12349 of Lecture Notes in Computer Science, pp. 102. Springer, 2020.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia Chao. Learning deep transformer models for machine translation. In ACL (1), pp. 1810-1822. Associa- tion for Computational Linguistics, 2019.
Tao Wang, Li Yuan, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. PnP-DETR: towards efficient visual analysis with transformers. In ICCV, 2021.
Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient DETR: improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In ICLR, 2021.

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 IMPLEMENTATION DETAILS OF THE SCORING NETWORK
The scoring network consists of 4 linear layers where Layer Normalization (Ba et al. 2016) comes before the first layer and every layer except for the last one is followed by GELU (Hendrycks) & Gimpel 2016) activation. The output dimension of the 1st layer is 256 and halved to 128 and 64 at the 2nd and 3rd layers. The last layer outputs 1-d logit for the BCE loss. Since the network locally processes the input tokens in a rosen-wise manner, the final decisions may overlook global statistics without additional treatment. To remedy this issue, we aside half of the output dimension of the first layer as a global feature, and average them across the whole token set, then concatenate it with each of the remained local features to maintain the original dimension. We also exclude the tokens that correspond to the zero-padded area when selecting top-p% scores, thereby we can prevent those tokens from participating in Hungarian matching process and getting meaningless gradients from the detection objective.
A.2 DAM CREATION IN DEFORMABLE ATTENTION
As attention offset calculated deformable attention fractional position, deformable attention uses bilinear interpolation to get values. Thus, we also bilinear interpolation obtain DAM.
Assume that, one of the attention offsets, weights and the reference point of decoder object query calculated p, A and r, respectively. Then, deformable attention takes values
Σ A G(a, + v(z)
where x enumerates all integral spatial locations in the feature map, G(:,)) is the bilinear interpo- lation kernel defined as G(a, b) = max(0, 1 1- la-a----0 bal) max(0, 1 Jay byl) and is the values. Similarly, we accumulate DAM-value for location as:
Σ A G-G(xr++p) (p.4.r)
Then, accumulate DAM over every decoder object query.
A.3 ALTERNATIVE OBJECTIVES FOR DAM-BASED MODEL
As training objective of the scoring network using DAM, we can consider other alternatives as long as they can encourage the predicted scores represent the relative saliency of the encoder tokens. One the naive alternatives is the regression loss by which the scoring network directly predicts the values DAM. The ranking loss can be another choice with which the network focuses more on learning the relativeness rather than estimating the set of salient tokens.
49.0
COCO mAp
48.5
48.0
BCE
Ranking Smoothed
Figure 8 shows the default BCE loss outperforms the alterna- tives. First, it is well-known that the regression problem is much harder than classification. Furthermore, since the value DAM changes during training. the regression loss to predict the ac- curate value more difficult. In case the pairwise ranking loss, ranking the DAM elements may also be unstable as DAM gradually evolves. Meanwhile, the BCE loss may reduce those element-level noises down to the sev-level in that its binary (keep or drop) targets retain more consistency compared to the exact values or ranks. Refer to Table to
47.5 10%
30% Keeping Ratio p
20%
40%
Figure -AM loss ablation
the exact values of the points represented in Figure g
12

---

Published as a conference paper at ICLR 2022
Table Comparision between the alternative objectives for DAM-based scoring network.
Loss
Keeping ratio (p)
AP
AP
AP75
APs
AP MM
APL
10%
47.8
68.9
51.7
29.8
50.9
64.0
20%
48.4
69.0
52.5
31.1
51.4
64.7
smoothed I.1
30%
48.6
69.0
52.6
31.1
51.9
64.7
40%
48.6
69.3
52.9
33.4
51.8
64.5
10%
48.0
69.1
52.1
29.9
51.4
64.6
20%
48.7
69.5
53.0
31.1
51.8
65.1
ranking
30%
48.8
69.2
52.8
31.4
52.0
64.9
40%
48.9
69.3
53.1
31.5
52.2
64.7
10%
48.2
69.2
52.3
29.8
51.2
64.5
20%
48.8
69.4
53.0
30.4
51.9
64.8
BCE
30%
49.1
69.5
53.5
31.4
52.5
65.1
40%
49.2
69.5
53.5
31.4
52.9
64.8
Table 3: Comparison between token selection criteria.
Scoring
Keeping ratio (p)
method
AP
AP30
AP7s
APs
AP M
APL
params
FLOPs
FPS
ResNet-50 backbone:
100%
46.3
65.8
50.1
29.0
49.4
61.7
41M
177G
18.2
N/A
0%
42.2
63.0
45.6
25.9
45.3
56.5
36M
91G
35.0
10%
43.6
64.3
47.2
26.7
46.9
58.4
41M
102G
27.7
20%
44.0
64.8
47.8
27.3
47.0
58.4
41M
110G
25.6
random
30%
44.0
64.9
47.5
27.4
47.4
58.2
41M
117G
24.1
40%
44.5
65.1
48.0
27.3
47.8
59.8
41M
125G
22.5
50%
44.4
64.8
48.0
27.8
47.4
59.2
41M
133G
21.1
10%
44.9
65.2
48.7
27.9
47.8
60.4
41M
106G
26.6
20%
45.5
65.5
49.3
28.7
48.3
60.5
41M
114G
24.7
OS
30%
45.7
65.8
49.5
29.7
48.5
60.8
41M
121G
23.2
40%
45.8
65.5
49.8
29.1
48.8
60.5
41M
129G
21.8
50%
46.0
65.9
49.8
28.8
48.9
60.6
41M
136G
20.6
10%
45.3
65.8
49.3
28.4
48.3
60.1
41M
105G
26.5
20%
45.6
65.8
49.6
28.5
48.6
60.4
41M
113G
24.8
DAM
30%
46.0
65.9
49.7
29.1
49.1
60.6
41M
121G
23.2
40%
46.2
66.0
50.3
28.7
49.0
61.4
41M
128G
21.8
50%
46.3
66.0
50.1
29.0
49.5
60.8
41M
136G
20.5
Swin-T backbone:
100%
49.4
69.4
53.5
31.9
52.6
65.1
41M
185G
15.4
N/A
0%
43.7
65.8
46.9
27.0
46.7
60.0
37M
96G
26.5
10%
45.5
67.6
48.8
28.4
48.5
62.2
41M
110G
22.1
20%
45.6
67.5
49.2
28.6
49.1
62.2
41M
118G
20.8
random
30%
46.2
68.1
49.7
29.5
49.7
63.0
41M
125G
19.7
40%
46.5
68.2
50.0
29.9
49.8
63.0
41M
133G
18.7
50%
47.2
68.3
50.9
29.1
50.4
63.9
41M
141G
17.7
10%
48.0
69.1
52.1
29.9
51.1
64.4
42M
114G
21.4
20%
48.3
69.1
52.5
30.4
51.6
64.2
42M
122G
20.2
OS
30%
48.6
69.2
53.0
31.0
52.0
64.6
42M
129G
18.6
40%
48.9
69.4
53.1
33.0
51.9
64.5
42M
137G
18.2
50%
49.0
69.2
53.5
31.2
52.4
65.0
42M
145G
17.2
10%
48.2
69.2
52.3
29.8
51.2
64.5
41M
113G
21.2
20%
48.8
69.4
53.0
30.4
51.9
64.8
41M
121G
20.0
DAM
30%
49.1
69.5
53.5
31.4
52.5
65.1
41M
129G
18.9
40%
49.2
69.5
53.5
31.4
52.9
64.8
41M
136G
18.0
50%
49.3
69.5
53.3
32.0
52.7
64.9
41M
144G
17.2
13

---

Published as a conference paper at ICLR 2022
Post-L ((defaut))
Pre.LN
Post-LN[defauly
Pro-LN
3.5
3.5
Gradient norm
3.0
2.0
Gradient
1.5
1.0
0.5
0.5
0.0
0.0
84
E4
E6
Backbone and encoder layers
Backbone and encoder layers
(a) DETR /ResNet-50 6 encoder layers
(b) DETR ResNet-50 encoder layers
Podt-LN((efault]
Pre-LN
Post-LN EncAuxx(urs)
Gradient norm
4.5
3.0
1.5
0.0
Backbone and encoder layers
(c) Deformable- -DabR Swin-T/ 12 encoder layers
Figure 9: Layerwise gradient norm DETR variants. observation of the vanishing gradi- ent problem on DETR variants with different backbones by measuring e-norm of gradients in layerwise manner. The first letter in r-axis label represents module name, specifically, 'B' for the backbone and 'E' for the encoder, and the second number represents i-th layer in that module. (a), (b): Layerwise gradient norm of DETR with ResNet-50 backbone. With the default settings(Post- LN), the gradient scale generally decreases more encoder layers are stacked. while the Pre-LN technique preserves gradient magnitude even in deeper early layers. (c) : Layerwise gradient norm of Deformable-DETR with Swin-T backbone. In this case, the Pre-LN fails to prevent vanishing gradient while the encoder auxiliary loss(denoted as Post-LN + EncAux) proposed in this paper effectively resolves this issue.
A.4 EXPERIMENTAL DETAILS DIFFERENT TOKEN SELECTION CRITERIA
Table containo the specific values used to plot (a) ResNet-50 and (b) Swin-T backbone in Figure Additionally, they also include a lower-bound baseline that has scoring method with keeping ratio 0%, meaning that the entire encoder block removed and the backbone features are directly passed to the decoder. Even with the lowest keeping ratio 10%, all the scoring methods including random criterion outperform this lower- bound baseline. Note that all experiments reported in Table except for the keeping ratio 0% use the encoder auxiliary loss for training.
A.5 VANISHING GRADIENT PROBLEM IN THE DEEP END-TO-END DETECTORS
As shown in Section 4.2 in Carion et al. (220)), they observe that the performance of DETR gradu- ally improves with more encoder layers. To reproduce this result, we used the default settings the official code, but only changed the number of encoder layers. However, we fail to train the DETR model when using more than 9 encoder layers, which is probably due to different hyperparameters from the ones used in their experiments. Interestingly, verges stably with the Pre-LN architecture 2019) that is known be better choice than the canonical Post-LN when the number of layers of
also found that the DETR model con- & Auli 2019 Child et al., 2019b; Wang et al.]
14

---

Published as a conference paper at ICLR 2022
Table 4: Effectiveness of the encoder auxiliary loss using Swin-T. When the number of encoder layers is more than 9, the model training fails, but if the encoder auxiliary loss is adopted, the model training is feasible regardless of the number of encoder layers, and accuracy is improved.
# of encoder
Keeping
Aux. loss
ratio (p)
AP
AP30
APTs
APs
A
M
APL
params
FLOPs
FPS
100%
48.0
68.0
52
30.3
51.4
63.7
41M
185G
15,4
100%
49.4
69.4
53.5
31.9
52.6
65.1
41M
185G
15.4
10%
46.8
68.0
50.6
29.7
49.7
63.3
41M
113G
21.2
20%
47.5
68.3
51.4
31.4
50.4
64.4
41M
121G
20.0
30%
47.6
67.9
51.4
29.9
51.1
63.9
41M
129G
18.9
6
40%
47.6
68.2
51.5
30.3
50.8
64.0
41M
136G
18.0
10%
48.2
69.2
52.3
29.8
51.2
64.5
41M
113G
21.2
20%
48.8
69.4
53.0
30.4
51.9
64.8
41M
121G
20.0
30%
49.1
69.5
53.5
31.4
52.5
65.1
41M
129G
18.9
40%
49.2
69.5
53.5
31.4
52.9
64.8
41M
136G
18.0
100%
49.7
69.4
54.1
32.4
52.9
65.4
44M
220G
12.8
100%
50.1
69.6
54.6
32.2
53.4
65.8
46M
261G
11.0
10%
49.0
69.5
53.5
31.6
52.2
65.2
46M
128G
19.2
12
20%
49.4
69.6
53.5
31.9
52.8
65.4
46M
143G
17.5
30%
49.3
69.4
53.6
31.7
52.5
65.6
46M
158G
15.6
40%
49.8
69.8
54.3
33.1
53.4
65.4
46M
173G
14.6
transformer increases. We used the pre.norm option the authors already have implemented in their code.
Figure (a) and 9(b illustrate that gradient norm of each layer from bottom to top in 6 and 12 encoder layers when using Post-LN and Pre-LN, respectively. We compute the e-norm of the gradients for all parameters in particular layer, as if they concatenated into single vector. To training dynamics in the early stage training, we track the gradients computed on a fixed set of training data and average them the first 150 steps with batch size of 2. Note that applied Pre-LN only to the encoder module for a fair comparison between only the early modules although could be used in any other transformer modules, e.g. backbone decoder.
We found that the vanishing gradient issue is generally observed regardless of the encoder size. When we double the size of the encoder, as one can expect, the gradient in the early layers ends up with an even smaller scale, which may have caused the convergence failure. Meanwhile, the Pre- LN technique seems to significantly alleviate this issue even for the deeper encoder by maintaining the gradient scale evenly through the encoder layers and conveying strong training signal to the backbone layers.
Deformable-DETR suffers from the same problem of vanishing gradient and even the Pre-LN technique does not help in this case. Meanwhile, the encoder auxiliary loss proposed in our paper drastically amplifies the gradient magnitude in the early layers by providing aggressive intermediate objectives for each encoder layer. Note that it also creates good synergy with our main sparsification strategy owing to reduced training cost. We claim that this observation supports our motivation introducing the encoder auxiliary loss.
On the other hand, as shown in Figure {C)
A.6 EXPERIMENTAL DETAILS FOR EFFECTIVENESS THE ENCODER AUXILIARY LOSS
Table presents the detailed values of Figure 4 As shown in the Table and discussed Section A.5 training of a deeper encoder of more than 9 layers fails without the auxiliary loss, but if it is adopted, the convergence becomes feasible as intermediate gradients provided to the early encoder layers augment the vanishing gradient back-propagated from the decoder module.
15

---

Published as a conference paper at ICLR 2022
Table 5: Detection results of Sparse DETR with SCRL initialization using ResNet-50. The same environment and hyperparameters as Experiments section are used, except for initializing the backbone with SCRL (Roh al., 2021) model. The results marked by mean that the backbone network is initialized by SCRL instead of the ImageNet (Deng et al. 2009) pre-trained one.
Keeping
Method
ratio (p)
AP
3P
50
AP
APs S
APM M
APL
params
FLOPs
FPS
10%
45.3
65.8
49.3
28.4
48.3
60.1
41M
105G
25.3
20%
45.6
65.8
49.6
28.5
48.6
60.4
41M
113G
24.8
Sparse-DETT -ETS
30%
46.0
63.9
49.7
29.1
49.1
60.6
41M
121G
23.2
40%
46.2
66.0
50.3
28.7
49.0
61.4
41M
128G
21.8
50%
46.3
66.0
50.1
29.0
49.5
60.8
41M
136G
20.5
10%
46.9
67.2
51.0
30.2
49.7
62.3
41M
105G
25.3
20%
47.3
67.1
51.4
29.7
50.3
62.7
41M
113G
24.8
Sparse-DETR'
30%
47.4
67.3
51.4
30.1
50.5
62.4
41M
121G
23.2
40%
47.7
67.4
51.6
30.0
50.8
62.9
41M
128G
21.8
50%
47.9
67.5
52.1
30.5
51.2
63.2
41M
136G
20.5
Table Performance of Sparse DETR with Swin-B. The same environment and hyperparameters as Experiments section are used, except for changing the backbone to larger scale. Note that Aux. loss means only the ones applied to the encoder layers.
Keeping ratio (p)
Aux. loss
Backbone
AP
AP50
AP75
APS
APM M
AP,
params
FLOPs
FPS
100%
48.0
68.0
52.0
30.3
51.4
63.7
41M
185G
15.4
10%
48.2
69.2
52.3
29.8
51.2
64.5
41M
113G
21.2
Swin-T
20%
48.8
69.4
53.0
30.4
51.9
64.8
41M
121G
20.0
30%
V
49.1
69.5
53.5
31.4
52.5
65.1
41M
129G
18.9
40%
49.2
69.5
53.5
31.4
52.9
64.8
41M
136G
18.0
100%
52.5
72.9
56.9
34.7
56.5
69.6
101M
400G
7.6
10%
52.2
73.5
57.0
34.0
56.3
70.3
10IM
335G
8.8
Swin-B 20%
V
53.1
73.8
57.9
34.6
56.9
70.6
101M
343G
8.6
30%
53.2
73.7
57.7
35.3
56.8
70.8
101M
350G
8.4
40%
53.3
73.4
58.0
36.3
57.2
70.9
101M
358G
8.2
A.7 EFFECTIVENESS OF USING A DENSE REPRESENTATION BACKBONE INITIALIZATION
Recently, many self-supervised learning methods through contrastive learning have been studied, and in particular, methods for obtaining dense representations with better performance for localiza- tion downstream tasks such object detection in the spotlight. In order to check whether our proposed method is effective even when such dense representation is used, the backbone network is self-
initialized with the SCRL (Roh al. 202T) model that aims learn dense representations supervised way instead of initializing with the ImageNet (Deng et al. (009)) pre-trained one. Just as the SCRL model outperformed the ImageNet pre-trained model various localization downstream tasks, our proposed method, Sparse DETR, also shows better performance in all keeping ratios(p) without the influence of encoder token sparsification as shown Table
A.8 USING LARGER TRANSFORMER BASED BACKBONE(SWIN-B)
We perform experiments Sparse DETR with Swin-Base al. 202 backbone to see if our method shows similar efficiency and performance gain even when using heavier transformer-be backbone. Table 6 illustrates comparison of COCO detection performance between Swin-T and Swin-B backbone under the varied sparsity. Due to the increased capacity, using Swin-B backbone significantly boosts up the baseline AP up to 52.5 (+4.5) but with 2.4x parameters and 2. 1x com-
16

---

Published as a conference paper at ICLR 2022
p=0.45
200
# of Images
150
100
50
0.6
0.8
Ratio of Non-zero Values of Decoder Attention Map
Figure 10: Distribution of the ratio of non-zero values of DAM on COCO 2017 set.
Table 7: Two-stage encoder token sparsification with varied keeping ratio. COCO detec- tion performance when the encoder tokens are sparsified at the later stage with the top-p% binarized DAMs pre-computed from the former stage. All models Deformable-DETR- with Swin-T back- bone and the encoder auxiliary loss is not applied. Note that the performance of the 50% model(47.9 AP) hardly degenerates compared to the baseline(48.0 AP).
Keeping ratio (p)
AP
AP50
AP75
APs
APM
M
APL
100%
48.0
68.0
52.0
30.3
51.4
63.7
10%
44.0
66.0
47.2
26.9
46.8
61.0
20%
44.9
66.3
48.3
28.2
48.2
61.4
30%
46.5
67.3
50.2
30.9
49.7
62.3
40%
47.3
67.9
51.3
30.7
50.7
63.4
50%
47.9
67.8
52.0
29.8
51.4
63.7
putational cost. With the keeping ratio of 40% and the encoder auxiliary loss, the performance gap remains similar level(+4.1). We also observe consistent performance gains as the keeping ratio gets higher while the increasing gap converges more quickly than Swin-T. It may be because single visual token of Swin-B can incorporate a wider range of information due the deeper tention hierarchies and smaller number of tokens is required to fully represent all the objects in an image. Note that the efficiency of backbone network behind the scope of this paper. Our work is orthogonal to the backbone sparsification approaches, e.g. Dynamic ViT (Rao et al. 2021). and we leave the integration with those works future work.
A.9 THE PRELIMINARY EXPERIMENTS: WHY PURSUE SPARSE ENCODER?
Using a model trained with Deformable-DETT tokens referenced by the decoder's s object query. Unlike using the bilinear interpolation described in the appendix A.2 to generate DAMs for training with pseudo-labels, in this analysis, do not bilinear interpolation to calculate how many encoder tokens are directly referenced by the decoder object query. To analyze the non-zero values of DAM, use Deforamble DETR model trained with Top-k sampling strategy Yao et al.) (02T) and bounding box refinement (Zhu et al. 2021) using ResNet-50 backbone. Fig. 10 illustrates the distribution of the ratio of non-zero values of
have analyzed the number of encoder output
DAM COCO va12017 dataset. As shown in Fig. 10 on average, only 45% of encoder tokens were referenced by object queries.
This observation naturally raises question: Can we preserve the detection performance even if we focus, in the first place, only on the encoder tokens that the decoder might have preferred? As a preliminary experiment to answer this question, trained the detector restricting token updates to the subset to which the decoder could have referred there had been no such restriction. To this end, performed the two-stage learning as follows: (i) We first obtained the DAMs of the entire
17

---

Published as a conference paper at ICLR 2022
training data by feeding them to fully-trained Deformable-DETR model. (ii) Then, retrained another model from the scratch by updating only a subset tokens determined by the binarized DAM preserving top-p% of the clements(refer S.ction for more details). Table 7 shows the performance on COCO detection for different keeping ratio / We found that the two-stage model almost catches up with the baseline(p 100%) the keeping ratio is raised close to 45%, namely the percentage of non-zero values in DAM computed on the validation dataset earlier.
These observations have strongly motivated presented in the main text. Note that our main algorithm differs from this preliminary experiment some aspects: (a) DAM is obtained from the jointy learning decoder, from the separately trained decoder, and (b) binarized DAM is utilized as prediction target of the scoring network rather than used directly as sparsification mask.
develop the encoder token sparsification method
A.10 VISUALIZATIONS OF SELECTED ENCODER TOKENS
We visualize selected encoder tokens and top-k decoder queries for each criterion, OS, and DAM. In the first row, selected encoder tokens from the backbone feature map are visualized as yellow regions, whereas unselected tokens are visualized as purple regions. In the second row, selected top- decoder queries from encoder output are visualized the same manner. the final row, DAM values and Corr metrics are visualized. Corr is measured as in Section 4.2
Interestingly, the DAM-basedd selection seems to better capture the objects than OS-based selection. The os-based selection also captures objects well, but it typically focuses on the high- feggfrreyyncc edges that are not only in the foreground but also in the background. On the other hand, the DAM- based selection captures the boundary of the objects and also their inner from the background edges. We analyze that DAM focuses on the boundary of objects to lower the regression loss, and attends to the inside of objects lower the classification loss. Finally, the scoring network predicts such a DAM well, and refining the encoder tokens according to it finally helps achieve better detection performance.
and is less distracted
18

---

Published as a conference paper at ICLR 2022
Image
Image
Bocktare
Enccadas
Exccede R0 Soo
cu,
Dabo
con::
(a) OS based model (p 0.1)
(b) DAM-based model (p 0.1)
Imagi
Muliseale
Mcale Mutiscele #2
Meliscale 44
Imags
Muliscale
Mullocate #2
Mutissale #2
Muliscalo
Bodloong
tolp 3585
Tel 3565
Enccoder tob 300
Enccodes
100 3000
DAV
DAW
corr: 0.74
com:: 0.43
corr: 0.87
com
cor:: 1.00
corr: 0.86
com: 0.74
corr: 0.90
com
con: 1.00
(c) OS-based model (p 0.2)
(d) DAM-based model (p 0.2)
Mulibccle Mullstate
Mutiocale #s
Muliinall
Mulibcale Muliatate
Matiscale a2
Muliscale 44
Badksone
top S343
Enccoder
000 300
Encodes
000 300
DANK
DAN
cor:: 0.84
con:: 0.67
corr: 0.96
com 0.98
con:: 1.00
cor:: 0.94
com: 0.87
corr: 0.97
cor 1.00
con:: 1.00
(c) OS-based model 0.3)
(f) DAM-based model (p 0.3)
Multiccale
Mutiicall #3
Mutticcale
Mutticale
Muliicaale
Mutticcle
Bardbone
100 7128
Sandboree
Tag 71
Encoser
t0e 500
Eicooter
50 500
CON
Dow
corn: 0.91
0.79
cor:: 0.98
com: 1.00
1.00
corn: 0.94
com 0.90
0.95
com 1.00
1.00
(g) OS-based model (p 0.4)
(h) DAM-based model (p 0.4)
Figure 11: Visualization of selected tokens and DAM for COCO validation image #289960
19

---

Published as a conference paper at ICLR 2022
Muliscale
Backbotee
Backbooe
RAD 1730
Toc 200
D8W
0.87
(a) OS-hased model (p 0.1)
(b) DAM-based model (p 0.1)
Image
Mutticale
Picalle
Mutticaale #3
Mutticcale
Image
Mutticaae
Miltscale
Mutiscale #3
Mutticale
top 3460
Tol 3460
Emcooder top 300
Eiccodor
100 3:00
DAN
DAW
cor:: 0.75
con:: 0.32
carr: 0.74
cor:: 0.99
con: 1.00
cor:: 0.91
con:: 0.75
carr: 0.91
cor:: 1.00
con: 1.00
(c) OS-based model (p 0.2)
(d) DAM-based model (p 0.2)
Mulibeale Mill stalee
Mutiscale as
Muliicale #4
Mulibcale Mulistate #2
Mutiscale as
Muliscale 44
Bacdhowe
100 5.89
Racihone
too $.00
Ercossn
t0p 500
Ercopdor
tpp 500
DAN
DAN
cor:: 0.89
0.63
0.85
com 0.99
1.00
cor:: 0.91
0.81
0.95
com 1.00
1.00
(e) OS-based model (p 0.3)
(1) DAM-based model (p 0.3)
tap 9915
top
Encodee top 300
Encodeer Top 200
D451
gg) OS-hased model (p = 0.4)
(h) DAM-based model (p = 0.4)
Figure 12: Visualization of selected tokens and DAM for COCO validation image #22396
20

---

Published as a conference paper at ICLR 2022
Image
Image
Mutiiscal
Backtare
Bocktonn Cop 1763
Enccdda
tep 2:00
cor::
(a) OS based model (p 0.1)
(b) DAM-based model (p 0.1)
Imago
Muliseale #1
Mul scatte
Mutissale #2
Huliscale 44
Image
Muliscale #1
Muliocate #2
Mutissale as
Mulizzale
Bedkoorn
top 3585
Bockporn
tol 20933585
Enccdder 100 300
Encoder hop 2000
DAN
DAN
cor:: 0.84
com: 0.70
corr: 0.92
com
cor:: 1.00
corr: 0.94
com: 0.92
corr: 0.94
com
cor:: 1.00
(c) OS-based model (p 0.2)
(d) DAM- model (p 0.2)
Mulibcale Mull scaate #
Mutiocale as
Multicale 94
Mulibcale Muliitate #2
Matissale a2
Muliicale a4
Badksone
top 5243
Beckseen
telp St4i
Enccoder
00 200
Encoderr
to0 200
DANA
DAN
cor:: 0.91
com: 0.84
corr: 0.96
cor:: 0.97
con:: 1.00
cor:: 0.97
com:: 0.96
corr: 0.90
corn: 0.98
cor:: 1.00
(c) OS-based model 0.3)
(f) DAM- model (p 0.3)
Muttocale
Mutiscall #3
Mutticcale
Mutticale Multicale
Muliiccale
Mutticale
Sardbone
100 7128
Bartbore
tap 7129
Encoserr
t2 500
Eirocerr
Se 500
CON
corn: 0.96
com 0.92
cor:: 0.98
com: 1.00
1.00
corn: 0.98
com: 0.97
0.98
com: 1.00
1.00
(g) OS OS-based model (p 0.4)
(h) DAM-based model (p 0.4)
Figure 13: Visualization of selected tokens and DAM for COCO validation image #46252
21

---

Published as a conference paper at ICLR 2022
Eroas Backbore
Image
MuBscale x1
Nuriscale 42
Mult
NuBscale 24
Encoueer Bardkerre
top 2045
Image
Nuttscak Milsaae
Muliccale #3
Nutticale
t00 2010
DAN
CoFF: 0.55
con:: 0.28
cof:: 0.91
corr: 0.80
con:: 0.93
cor::
com: 0.79
cor:: 0.91
com 0.95
com: 0.99
(a) AS-based model (p 0.1)
(b) DAM-based model (p 0.1)
Encodder Bardhooe
000 300 top 4050
Image
Murticcae #1
M.Ficale #2
Mutiicalee #3
Hutticale #4
Encoverr Bardkeree
top 40:50
Image
Nurtscale
Muliscalee #2
Muliicalee #3
Nuttscals 04
100 2000
DAVI
DAW
con: CoF:: 0.57
cor:: 0.96
com
cor:: 1.00
cor:: 0.95
com:: 0.94
corr: 0.94
com
con: 1.00
(c) OS-based model (p = 0.2)
(d) DAM-based model (p 0.2)
Encosen Bacibove
tie 500 1op 6176
Mulibccle
Mull Scate
Mutiocale as
Miliieale
Encossn Bacibove
THE 500 Top $174
Mulibcale Muliotate
Matiscale a3
Muliscal
DADA
DADH
CoT:: com: 0.76
corr: 0.98
com 1.00
cor:: 1.00
cor:: 0.97
com:: 0.97
corr: 0.98
com 0.99
con:: 1.00
(e) OS-bosee based model (p = 0.3)
(1 DAM- model (p = 0.3)
Encesee Eaccbene
tig 300 top 8179
Mullsccal Muul
Mutiicale #s
Muliscale
Encoserr Bacobere
Tig 590 Top $.7.
Multscale Mullscare
Mutiscale
Muliscala
OAAY
com:: 0.93
con:: 0.88
corr: 0.90
cor:: 1.00
com:: 1.00
cor:: 0.98
con:: 0.98
corr: 0.98
com:: 1.00
com 1.00
(g) based model (p = 0.4)
(h) DAM-based model (p 0.4)
Figure 14: Visualization of selected tokens and DAM for COCO validation image #6040
22

---

Published as a conference paper at ICLR 2022
Muliccale
Muliacalle
Mullbicate
Mulibicate
Muliszale
Muliscall
Kullscall Nuliccale
+6n
Encocoe foop 300
DAN
COnT: 0.57
corr: 0.14
com 0.74
com 0.96
cor:: 0.98
cor:: 0.70
cor::
Cor:: 0.77
cor:: 0.92
con::
(a) OS -s-based model (p 0.1)
(b) DAM-based model (p 0.1)
Image
Multicaai
Mutiscals #2
Nuticcll #3
Nutsscal a4
Image
Nutsccai
Nutiscla #2
Nuticcale #3
Nutscale
Bucksone
tap 3565
Entoveer tog 300
Encoveor
logp 300
DAN
DMH
con:: 0.75
con:: 0.42
con:: 0.86
con:: 0.99
corr: 1.00
con:: 0.86
cor:: 0.68
con:: 0.80
con:: 0.99
con:: 1.00
(c) OS-based model (p 0.2)
(d) DAM-based model (p 0.2)
Muttinzale Mullicaale
Nutincale
Muliccale #6
Mulasale Nuliaalle
Nubiccale
Mubacale
Barkbens
top 5341
Encoeerr Iep 1000
Encocerr
too 100
DWM
con:: 0.81
con:: 0.53
con: 0.91
con:: 1.00
carr: 1.00
con:: 0.89
con:: 0.83
con: 0.86
con:: 0.95
carr: 0.98
(c) OS OS-based model (p 0.3)
(f) 3AM-based model (p 0.3)
image
Multicali A1
Muticcals #2
Nutiscals #3
Mutiscale #4
Image
Nutsccal Al
Nuticcll #2
Mutiicale #3
Muticclle
Buclbenc
top 1128
Enrovler log: 300
10x
Encoveor
lopp 300
DAN
DAM
con:: 0.85
con:: 0.65
con: 0.89
cof:: 1.00
con:: 1.00
cor:: 0.96
con:: 0.95
con: 0.94
con:: 1.00
con: 1.00
(g) OS based model (p = 0.4)
(h) Dasebased model (p 0.4)
Figure 15: Visualization of selected tokens and DAM for COCO validation image #1 117799
23


Reviews:
[{"review_id": "txNPyrVSpbL", "reviewer": "Reviewer_sTnV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-The motivation is interesting somehow. The computational and memory complexity of Deformable DETR are indeed some problems. It would be interesting to reduce its complexity but remain accuracy.\n\n-The techniques to achieve the goal, e.g. adding a saliency head or using the decoder saliency map, are kind of empirical and less interesting.\n\n-I am quite confused about the two additional components in Sec. 3.4. I don’t get their motivations and they seem to be irrelevant to the main motivation of this paper. What are their effects? Don’t see detailed ablations on them in the experiments.\n\n-The improvements on both accuracy and speeds are reasonably good, but not very impressive.\n\n-The writing/presentation needs to be improved. There are quite a few confusing/unclear points in the paper, which are difficult to understand.\n\n\nSome detailed comments:\n\n-in the abstract, it says only 45% queries are referred and not updating them don’t drop the accuracy. How is this implemented? A very naïve baseline? Don’t see it in the experiments.\n\n-The auxiliary loss was first used in [A], instead of GoogleNet.\n\n-For the query selection, once the queries are selected, they are active for all encoders or each encoder has its own selection?\n\n-For the inactive queries, are they inactive for all encoders or they can be inactive for 1st encoder but active for the 2nd?\n\n-For the inactive queries, aren't they not involved in the computation graph at all? how could they be referenced when updating the selected queries?\n\n-Fig. 2 is quite difficult to read.\n\n-For DAM, if you already use the decoder for saliency, how can you not compute the encoder? Aren’t the encoder’s outputs the decoder’s inputs?\n\n-Why need another 4-layer scoring network? Don’t you already use decoder to compute the saliency?\n\n-How to add auxiliary loss to encoder? There are no object queries as input to the encoder, so no detections for encoder.\n\n-In table 1, what’s the deformable detr with a check mark? And what’s Deformable DETR+?\n\n\n[A] Deeply-Supervised Nets, AISTATS 2015", "summary_of_the_paper": "This paper tries to solve the problem of expensive computation on the encoder of Deformable DETR. The hypothesis is the encoder inputs a large number of image feature queries, but only a small number of them are actually referred by the decoder. The paper has shown that with a help with some selection mechanism, the encoder and only inputs partial of the feature queries, such that the computation can be reduced. With some other components, e.g. “top-k decoder queries” and “encoder auxiliary loss”, the Sparse DETR can achieve slightly better performances at fewer computations.", "summary_of_the_review": "The motivation is somehow interesting; the techniques are kind of empirical and less interesting; the improvements over Deformable DETR are not very surprising on both accuracy and speeds.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qX94ASfFHpp", "reviewer": "Reviewer_YsVT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the method seems fairly simple and should be straightforward to implement. The sparisification contributions are well supported experimentally, with ablations and insights.\n\nContribution #3, the encoder auxiliary loss, is not well supported. The original DETR paper shows that the encoder can be scaled to 12 layers and show improved performance over 6 layers (+1AP), table 2. An additional investigation on what could be the reason behind convergence issues in deep Deformable DETR is required. Training deep Transformer encoder models is extensively studied in NLP, with well known practices like BERT init and pre-norm block structure. The contribution needs to be backed up by convincing evidence on why it would be preferable over the common practices.\n\nIt would be beneficial to show results with instance or panoptic segmentation, eq with SOLQ which was build on top of Deformable DETR as well. Would in this case more topk features be required to match performance, would be proposed method still be a useful method to improve computational efficiency? Would Objectness Score be a better sparsification target than DAM in case of masks?\n\nSuggestions to improve writing:\n- In table 1, results for R50 with SCRL init could be moved to appendix as they are not directly support the claims.\n- The first sentence of the abstract where the authors say \"DETR demonstrates competitive performance but low computational efficiency\" is confusing, since in table 1 DETR actually has the lowest FLOP count/fastest runtime. I guess what the authors meant was that \"DETR demonstrates competitive performance but low computational efficiency on high resolution feature maps\". Overall the abstract needs to be significantly improved, it has undefined terms and is very difficult to read. For example, in the sentence \"Using the multiscale feature to ameliorate performance..\" neither multiscale nor feature is mentioned before.\n- Figures 4-7 lack captions explaining what the reader is supposed to understand from them.\n- Figures 2-3 would benefit from highlighting inputs and outputs\n- The authors use word \"polish\" when describing Transformer layers applied to feature maps, which is slightly confusing. Perhaps there is a better word for it.\n- notation $x_\\text{bb}$ is confusing, since $\\text{bb}$ typically means \"bounding box\", and not \"backbone\".\n- Section 3.1: \"decoder takes both the refined queries ... \" Transformer encoder output is typically called \"memory\", it would disambiguate encoder output from object queries.\n\n", "summary_of_the_paper": "A modification to the recently proposed Transformer-based object detector, DETR, is proposed. The authors are motivated by low performance of DETR on small objects, and high computational cost of self-attention on large feature maps. To improve both, they propose to sparsify input feature maps by learning a classifier whether to include an input feature or not. Supervision for this classifier comes from Transformer decoder attention weights. The authors apply their approach to Deformable DETR and evaluate on the COCO bounding box detection benchmark, and show that it allows to trade detection performance for computational efficiency by tuning the classifier threshold.", "summary_of_the_review": "The paper suggests a novel approach for improving computational efficiency of Transformer-based object detectors. Most contributions are supported by experiments, ablations and insights. The contribution with auxiliary losses in encoder requires more investigation. Overall I recommend accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "oHUQN5ajajT", "reviewer": "Reviewer_jT45", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe method is novel and technically sound. As we all know that the queries or tokens are somewhat redundant for DETR-based object detection, this work brings us some new insight of how to efficiently condense queries without performance dropping. Compared to PnP and other dynamic ViTs, this work is more effective and efficient. The writing of this work is also clear and easy to follow.\n\nWeakness:\nIn table 1, the 50% ratio Sparse DETR is higher (46.3 vs 46.0) than Deformabel DETR under the same training setting. In my understanding, the full queries version of Deformabel DETR is equal to a 100% ratio version of sparse DETR. If so, why the performance is even higher when we only keep 50% of queries. I think this result is deserved for more analysis. Or are there any additional components that can improve the total performance of the full version of the detector? \n", "summary_of_the_paper": "This paper proposes an efficient end-to-end object detection architecture based on Deformable DETR, called Sparse DETR. Its main contribution is solving the computation budget by selecting only a few tokens as queries in attention computing. Sparse DETR can use only 10%-50% of the original encoder query while achieving comparable results. This is done by several components. Firstly, the authors add a subnet named Scoring Net before encoder to predict saliency scores for input. And only the tokens which have top-k scores will be chosen as queries in the encoder attention computing. Then, to train the Scoring Network, this paper use binarized attention weight from cross attention, named DAM, as pseudo labels. It also adds a detection head to predict the class scores for outputs of the encoder. The scores are used to select the appropriate token as the decoder`s query. The extensive experiments show the effectiveness of the proposed method.", "summary_of_the_review": "Overall, this paper provides valuable insight into reducing queries for DETR-based detectors and I prefer to accept it.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Au9dLEvSoxX", "reviewer": "Reviewer_u6Zb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this is a good paper that has \n\na. reasonable motivations and backgrounds\n\nb. reasonable and effective solutions\n\nc. supportive benchmark results and sufficient ablation studies\n\nd. good written and easy to follow. \n\nIt focuses on a specific problem in current DETR based methods and proposes a reasonable solution/direction to further improve the efficiency of the model. \n\nThere are some suggestions to improve the literature: \n\na. There is an argument that “the encoder queries referenced by the decoder account for only 45% of the total”. However, I do not observe any statistical analysis and support in the main paper. A more detailed analysis is recommended to add to better support the paper. \n\nb. The paper uses the word “encoder queries”. It doesn’t have problems. However, the encoder queries (i.e. backbone features) are key/values in the decoder, not the queries. In DETR and Deformable DETR, they all use the word “feature maps”, because from the perspective of the entire Transformer, the query is the learnable object embeddings. I recommend the authors revise this part to make the presentation more clear. \n", "summary_of_the_paper": "This paper proposes methods to further improve the efficiency of Deformable DETR. It observes that the encoder queries referenced by the decoder account for only 45% of the total, and the detection accuracy does not deteriorate significantly even if only the referenced queries are polished in the encoder block. To this end, this paper proposes Sparse DETR that selectively updates only the queries expected to be referenced by the decoder. An auxiliary detection loss in the encoder also improves the performance. Experiments prove the effectiveness of the method. ", "summary_of_the_review": "Overall, this is a good paper that has good motivations and reasonable solutions. The experiments are thorough. I recommend to accept the paper and hope the authors can revise the paper according to my suggestions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to modify DETR, a recent Transformer-based architecture for object detection.
More precisely,  they propose to sparsify input feature maps by learning an extra classifier to select which input features (few of them) will be used in the attention module. The supervision of this classifier is guided by second extra module  coming from the Transformer decoder attention weights.
The resulting framework, called Sparse DETR, is an efficient end-to-end object detection architecture that allows to overcome the main computational bottleneck of DETR. Sparse DETR can use  only 10%-50% of the original encoder query while achieving DETR comparable results.

Authors tried to answer to all the questions raised during the rebuttal. 
Even if the final scores are still contrasted, most of reviewers are very positive. 
Overall, this paper provides valuable insight into reducing computational complexity (by reducing the number of queries) for DETR-like detectors. The proposed method is novel and technically sound. Even if there are some tricks to make the whole working well, this work is more effective and efficient than previous propositions to handle this complexity problem,  bringing us some new insight of how to sparsify queries without performance dropping.  
All these elements lead me to propose this paper for publication at ICLR.
Decision: Accept (Poster)