Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
Authors: Jiacheng Cheng
Xiwen Yao
Xiang Yuan
Junwei Han
Venue: ICML 2025 poster (2025)
Keywords: detection transformer; efficient model; token merging
Primary area: applications->computer_vision
TL;DR: We introduce Dynamic DETR, a novel approach that adaptively controls token density via importance-aware distribution and applies multi-level sparsification.

Abstract: The substantial computational demands of detection transformers (DETRs) hinder their deployment in resource-constrained scenarios, with the encoder consistently emerging as a critical bottleneck. A promising solution lies in reducing token redundancy within the encoder. However, existing methods perform static sparsification while ignoring the varying importance of tokens across different levels and encoder blocks for object detection, leading to suboptimal sparsification and performance degradation. In this paper, we propose **Dynamic DETR** (**Dynamic** token aggregation for **DE**tection **TR**ansformers), a novel strategy that leverages inherent importance distribution to control token density and performs multi-level token sparsification. Within each stage, we apply a proximal aggregation paradigm for low-level tokens to maintain spatial integrity, and a holistic strategy for high-level tokens to capture broader contextual information. Furthermore, we propose center-distance regularization to align the distribution of tokens throughout the sparsification process, thereby facilitating the representation consistency and effectively preserving critical object-specific patterns. Extensive experiments on canonical DETR models demonstrate that Dynamic DETR is broadly applicable across various models and consistently outperforms existing token sparsification methods.

Not All Tokens Matter All The Time: Dynamic Token A Aggregation Towards Efficient Detection Transformers
Jiacheng Cheng'
Xiwen Yao!
Xiang Yuan
Junwei Han
Abstract
Backhone
E Neck
Encoder
Decoder
m Head
The substantial computational demands of detec- tion transformers (DETRs) hinder their deploy- ment resource-constrained scenarios, with the encoder consistently emerging as critical bottle- neck. A promising solution lies in reducing token redundancy within the encoder. However, exist- ing methods perform static sparsification while ignoring the varying importance of tokens across different levels and encoder blocks for object de- tection, leading to suboptimal sparsification and performance degradation. In this paper, we pro- pose Dynamic DETR (Dynamic token aggrega- tion for DEtection TRansformers), a novel strat- egy that leverages inherent importance distribu- tion to control token density and performs multi- level token sparsification. Within each stage, we apply a proximal aggregation paradigm for low- level tokens to maintain spatial integrity, and a holistic strategy for high-level tokens to capture broader contextual information. Furthermore, we propose
120
FL.OPs())
80
70.36
40
6.15
10.5
4.15
10.5
4.15
D-DETR
DAB-D-DETR
DINO
(b)
BBVO
(d)
Syam
Lax
24
15
30
50.6
50.2
BAR-S-BRTE
20
Spece
D-BRTT
200
10
126
150
1770
160
175
200
220
100
200
FLOPs (G)
regularization to align the distribution of tokens throughout the sparsifica- tion process, thereby facilitating the representa- tion consistency and effectively preserving critical object-specific patterns. Extensive experiments canonical DETR models demonstrate that Dy- namic DETR is broadly applicable across various models and consistently outperforms existing to- ken sparsification methods.
Figure I: (a) Calculation distribution of D-DETR (Zhu et al., 2021), DAB-D-DETR (Liu al., 2022), and DINO (Zhang et al., 2022), where the encoder emerges the primary contributor to the overall computational load. The compar- isons between various sparsification strategies on D-DETR (b) and DINO (c) in terms of FLOPs and FPS, while (d) exhibits the promotion our method applying to DAB-D- DETR, DN-D-DETR (Li et al., 2022), and H-D-DETR (Jia et al., 2023), where the AP of each method is underlined. All results are obtained using ResNet-50 as the backbone.
1. Introduction
transformers. Deformable DETR (Zhu et al., 2021) further advances this paradigm with a multi-level deformable at- tention mechanism that accelerates convergence and boosts detection performance. Building on this foundation, vari- ous DETR- models (Zhu et al., 2021; Liu et al., 2022; Li et al., 2022; Zhang et al., 2022) have been proposed, driving both the performance and versatility of object de- tection frameworks. However, the growing computational and memory demands of DETRs remain a significant obsta- cle to their practical deployment, particularly in resource- constrained environments that necessitate both efficiency
Detection Transformer (DETR) (Vaswani, 2017; Carion al., 2020; Meng al., 2021; Yao et al., 2021) has marked transformative shift in object detection by incorporating
'School of Automation, Northwestern Polytechnical Univer- sity, Xi an, China. Correspondence to: Xiwen Yao <yaoxi- wen nwpu.cdu.cn>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
tion weights across different levels and encoder blocks in Figure 2, and find that we move deeper into the encoder blocks, the importance ranking R of lower-level tends to decrease, while that of higher-level increases. This indicates that token importance across levels is dynamic but grad- ually migrates from lower to higher levels as the encoder depth increases. However, current approaches (Zheng et al., 2023; Wang et al.. 2021: Roh et al., 2022). typically employ static, coarse-grained token discarding strategies based on global patches (Pu et al., 2022). Worse still, they resort to compensatory modules to mitigate information loss and fail to capture the hierarchical characteristic of tokens. Inspired by this insight, we propose Dynamic DETR, a framework that adapts token sparsification based on their evolving im- portance across encoder stages.
1.0
Leve-1
Progertion 0.27
R:
Properdoo: 0.20
R:2
Progentow 0.25
R
Leul-2
0.6
Pregarsion: 0.49
Preporalas: 0.18
R:
Propardlon:
Leve--3
0.4
Proportion::
Propartion:
Proparzion:
0.2 We apply token merging for token sparsification, which has been widely validated across the fields of transformers (Bolya et al., 2023; Liang et al., 2021; Long et al., 2023; Lee et al., 2024). Yet, its application in DETRs often overlook the diverse relationships between tokens and their respective levels, leading to suboptimal matching and significant per- formance degradation. Low-level tokens primarily capture detailed local information, with key features often concen- trated in specific spatial regions (Cao et al., 2021). To preserve these details, merging should prioritize spatially proximal tokens, ensuring the retention of critical localized information. At higher levels, tokens with border recep- tive filed, related tokens may be distributed across different spatial locations (Lin et al., 2022). Holistic aggregation fa- cilitates the direct integration of the most important tokens, without being constrained by local neighboring regions. To this end, we propose strategy that applies distinct aggre- gation techniques across different levels, ensuring optimal token aggregation and preserving both local and global in- formation effectively.
Leve-4
Progertion: 0.03
R:1
Propertion: 0.04
R:1
Progertion: 0,90
R:4
0.0
Figure 2: Visualization of attention weights of tokens across different encoder blocks. Noting that larger weights con- tribute more to the optimization and are deemed to be im- portant in the attention mechanism according to pioneering efforts (Raganato& & Tiedemann, 2018). We highlight the top 75% weights in the current block with pink-shaded mask, while the proportion of each level comes Proportion. Moreover, the term importance ranking (R) is introduced to intuitively represent the distribution of important tokens across different hierarchies. It can be seen that important tokens tend to migrate from shallow to deeper pyramidal levels as the depth of the encoder blocks increases.
and low-latency inference (Chen et al., 2022b; Zhang al., 2023; Chen et al., 2022a).
Furthermore, we hypothesize that sparse and non-sparse features fed into the detection heads should maintain an in- dependent and identically distributed (i.i.d.) alignment (Gao et al., 2024). However, the varying inputs at encoder block introduce deviations from this ideal distribution. To address this issue, we propose a representational center-distance reg- ularization to encourage the convergence toward the desired i.idd. distribution.
To investigate the underlying computational burden, we an- alyze the floating point operations (FLOPs) for three preva- lent DETR variants: Deformable DETR (Zhu al., 2021), DAB DETR (Liu et al., 2022), and DINO (Zhang et al., 2022). As illustrated in Figure (a)), the backbone and coder contribute the most to the overall computational load. Although recent work has predominantly focused on opti- mizing the backbone (Rao et al., 2021; Chang et al., 2023; Liang et al., 2021; Long et al., 2023; Bolya et al., 2023), the encoder remains significant bottleneck. This oversight is problematic, as efforts reduce backbone complexity, such with Dynamic VIT (Rao et al., 2021; Liu et al., 2024), lead to only modest reductions in FLOPs (approximately 12%). while causing substantial drop in accuracy (4.6%).
To further validate the effectiveness and generality of Dy- namic DETR, we evaluate it on COC02017 using five DETR variants (Zhu et al., 2021; Liu et al., 2022; Zhang et al., 2022; Li et al., 2022; Jia et al., 2023). Results show that Dynamic DETR significantly reduces computational costs with minimal performance drops, outperforming other token sparsification methods (Figure 1). To sum up, the contributions of this paper three-fold:
Token sparsification has emerged as an effective solution for reducing the number of tokens based on their importance. To employ the importance of tokens, we visualize the atten-
Proposing Dynamic DETR, dynamic token aggregation
2

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
pipeline for detection transformers that efficiently sparsi- fies tokens across encoder stages, achieving computation reduction with least performance loss.
While effective, these methods impose substantial computa- tional overhead, rendering them less suitable for deployment in detection transformers.
. Devising a novel token aggregation strategy to handle multi-level token sparsification: a proximal aggregation paradigm for low-level tokens to preserve spatial details and a holistic merging strategy for high-level tokens to leverage their global receptive field. To the best of our knowledge, this is the first application of token aggregation techniques in detection transformers.
Merging-base based methods simplify computations by consoli- dating similar tokens. ToMe (Bolya et al., 2023) and CTS (Lu et al., 2023) merge tokens based on similarity, while EViT (Liang et al., 2021) and Long et al. (Long et al., 2023) use attention scores to aggregate discarded tokens into "su- per tokens". MCTF (Lee et al., 2024) extends merging by integrating multi-criteria for better information preservation. However, existing merging methods often lack hierarchical adaptability, and overly simplistic strategies risk entangling critical information.
Introducing the representation center-distance regulariza- tion that aligns the centroids of token distributions before and after sparsification. This improves feature consistency and enhances detection performance.
3. Method
2. Related Work
We propose a general paradigm for token sparsification in Detection Transformer (DETR) models, compatible with a wide range of architectures, including Deformable DETR (Zhu et al., 2021), DAB-DETR (Liu et al., 2022), DINO (Zhang et al., 2022), and other prominent variants (Li et al., 2022; Jia et al., 2023). Based on the analysis presented in Section 1, our method applies token merging selectively at specific stages within the encoder, as illustrated in Fig- ure 3. Detailed descriptions of the proposed methodology are provided in the following subsections.
2.1. Advancing Detection Transformer Designs
Recent advancements in Detection Transformers (DETRs) have aimed at improving detection accuracy and reducing computational costs.
On the performance front, Deformable DETR (Zhu et al., 2021) introduced multi-scale deformable attention to en- hance convergence and accuracy. while Anchor DETR (Wang et al., 2022) and DAB-DETR (Liu et al., 2022) in- corporated anchor-based priors to bridge the gap with tra- ditional detectors. DN-DETR (Li et al., 2022) and DINO (Zhang et al., 2022) leveraged noisy ground-truth boxes to improve bounding box prediction. Despite these improve- ments, their high computational cost remains a barrier for real-world deployment.
3.1. Preliminaries
Deformable DETR (Zhu et al., 2021) significantly enhances the performance of DETR (Carion et al., 2020) and has served as the foundation for several subsequent variants (Liu et al., 2022; Li et al., 2022: Zhang et al., 2022; Jia et al., 2023). For clarity and reproducibility, we adopt Deformable DETR as the baseline to describe our method.
Efficiency-focused DETR variants address this challenge. Efficient DETR (Yao et al., 2021) reduced encoder layers by initializing decoder queries with priors, while Lite DETR (Li et al., 2023) simplified token levels. Sparse approaches like Sparse DETR (Roh et al., 2022), PnP DETR (Wang et al., 2021), and Focus DETR (Zheng al., 2023) opti- mized token updates by scoring salient tokens. However, these methods often rely on auxiliary modules and loss terms, complicating integration and limiting generality.
Let {<1)í_ represent the multi-level input feature maps, where x Hi denotes the representation at level I, with Hi, st, and C' corresponding to the height, width, and hidden states dimension, respectively. To meet the input requirements of the DETR encoder, each feature map is flattened into a sequence of embeddings Zi E R N.C. where N₁- H2 x 81 denotes the total number of tokens level I, which are subsequently fed into the encoder.
2.2. Token Sparsification Algorithms
The total number of tokens across all levels is N = Σl= Nj. The computational complexity of encoder blocks is primarily dominated by the self-attention mechanism. In the case of the deformable multi-scale attention module, the complexity is O22N, + min C2, N2KK))), where N, is the number of tokens updated in the current block and K is the number of sampled keys (Yang et al., 2021). Thus, the primary issue for decreasing the overall computational cost lies in the reduction of No.
Token sparsification has become central to accelerating vi- sion transformers (ViTs) (Dosovitskiy et al., 2020), primar- ily comprising pruning-based and merging- methods.
Pruning-based methods predict token importance scores us- ing auxiliary modules. For instance, Dynamic ViT (Rao et al., 2021) and STViT-R (Chang et al., 2023) rank tokens based on their relevance, while A-ViT (Yin et al., 2022) uses an accumulative mask to discard tokens below a threshold.
3

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
Stage! 1 x N₁
Stage II x N2
Stage III x N
Proximal Aggregation
Proximal Aggregation
Proximal Aggregation
Flatten
R R
R
R,
Holistic Aggregation
Holistic Aggregation
Holistic Aggregation
Backbone
Decoder
Proximal Aggregation
Holistic Aggregation
Tokens
Reshape
Flatten
Thresholding
200
4
Untmmporaant Tokens
"__ Wy
Proximal Relationship
Ranking
Affinity Matching and Merging
Allocating
Figure 3: Dynamic Token Aggregation Pipeline for Detection Transformers (Dynamic DETR). The 6 encoder blocks divided into three stages with N2 2, N2 3, and N3 1. At each stage, tokens are merged based dynamic importance ranking RD of different levels. At lower levels, a proximal aggregation paradigm is employed, which first maps tokens to there original spatial grids and aggregates them within local windows. (e.g., at level-2 with a window size of n = 2, each window contains tokens.) Then the proximal relationship, denoted as Wm. is quantified to inform the aggregation decision. At higher levels, we devise holistic aggregation strategy, where tokens are divided into important ones and unimportant ones according their importance scores. Then unimportant tokens are aggregated into its most affine counterparts with the proposed affinity matching and aggregation mechanism. Finally, the sparsified features are constrained by Lalign to preserve their fidelity by aligning the representation centroids between the non-sparsified and sparsified feature spaces.
1001.0%
3.2. Dynamic Token Aggregation
Stage 2
Stage 3
The core of our aggregation mechanism lies in the dynamic evolution of the importance criterion for tokens across vari- ous hierarchical features and encoder blocks. Therefore, be- fore delving into the proposed paradigm, we first investigate how the importance shifts within the encoder architecture.
83.25
1300
72.66%
$100005
3200
Notation. Hereafter, we use the following notation for clarity and conformity. Given the importance ranking at s- th stage Ri > Rig > >..>> Rog, we first introduce ranking index as follows:
1300
8.59%
7.2%%
6.7%
I = (i 12, in., where S S L, + ( k). (1)
2.5%
2.2%
2.1%
888%
1224
2a14
$124
1442
$612
152)
1348
Index
Then, the aforementioned ranking can be formulated with:
Figure The distribution of importance ranking index
R" ="=iRij: Re....., Rig) R-
(2)
(namely I) across various pyramidal levels three stages. Note that the statistics are captured the COCO val -set images (Lin et al., 2014).
Firstly, we divide the conventional six encoder blocks in the pipeline into three stages with N1 2, N2 = 3, and N3 1. Then, we procure the distribution of importance ranking for each level E {l..... L.. at three stages in Figure 4. It is evident that the importance rankings are distributed

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
differently across the various encoder stages. In particular, during the first stage, we observe R! (R1, , R2, Ra, Ra), indicating that the model initially places greater emphasis on lower-level feature maps. As the stage depth increases, however, higher-level representations-
erally, the aggregation of tokens at level / is formalized as:
9p Z Ih (1)
if S 1-1
2,
(4)
otherwise,
as those from levels / = or = 4-start to account for more of the impor- tant tokens, demonstrating clear migration of importance. This finding aligns precisely with our observations in the Introduction (see Figure 2): namely, that the criterion for token-level importance within the encoders is dynamic and evolving across different stages.
where 2, illustrates the aggregated tokens at level I.
Low-level: Proximal Aggregation. The idea behind this paradigm is simple: tokens at lower pyramids are spatially abundant but semantically sparse, hence aggregating seman- tically similar proximal tokens builds path to reduce the overall redundancy. Enlightened by previous works (Liu et al., 2021; Zhang et al., 2024), we introduce a window- based setup to achieve the optimal balance between com- pression efficiency and semantic integrity.
Inspired by this observation, we therefore propose an dy- namic token aggregation mechanism to dynamically reduce the number of tokens updated at each stage, thereby sub- stantially diminishing the computational complexity while maintaining competitive performance. In contrast to pio- neering works (Wang et al., 2021; Roh et al., 2022; Zheng et al., 2023) that fix uniform retention ratio across all levels, we leverage statistic-based importance ranking to dy- namically derive this key parameter for token sparsification at each level, thereby more accurately accommodating the evolving distribution of critical tokens within the encoder. To be specific, first initialize the retention ratios for L pyramidal levels with p (P1, P2...., PL) E R. Then for stage s, its retention ratio P. is formulated by reordering based on the ranking index I", namely:
To be specific, we first map the tokens at level I, Z,, back to the original spatial dimensions C Hi x st. Then, it will be partitioned into M local patches with each containing tokens, where n 2 stands for the window size level I. Intuitively, such setup fully leverages the resolution relationships between adjacent feature maps in the FPN. And most importantly, is hyperparamter--fee.
Next, to better investigate the local similarity of these patches, we capture the proximal relationship matrix for each patch, based on fully connected bipartite graph con- structed among tokens within the patch, which is formulated as the edge matrix E,, ERaxn where m E fl.... MM. Concretely, supposing z; and zj belong to the identical proximal patch, then the edge weight Gi between these two nodes is demonstrated as follows:
p² p|="] (pi,, i Pig.....
PP..)
(3)
In this way, the number of tokens after sparsification for level I is derived K₁ N₁ x pf. To go step further, the static ratio setting can be viewed as special case of our dynamic strategy, wherein = up. by reducing the number of tokens at each stage in an evolving manner, our approach achieves substantial compu- tational savings, allowing more resources to be dedicated to processing the most critical token interactions
2i 2
Ci =
(5)
PL- To sum
| |
where 21, Zy RCx1 (with i, {1...., Ni) and i + j) represent the i-th and j-th tokens of Zi, while stands for the inner product, and || ||a is the Fuclidean norm
To comprehensively quantify the semantic coherence of all proximal patches at level I, we capture the average weight sequence Wi- (WI, W22 ... ,,uM) ), in which:
3.3. Multi-level Token Aggregation
Our solution is grounded in the observation that the distribu- tion of important tokens across different hierarchies of Fea- ture Pyramid Network (FPN) (Lin et al., 2017) progressively shifts as the encoder blocks deepen. Therefore, the sparsifi- cation process is dynamically executed through a multi-level framework, incorporating two complementary aggregation paradigms: (a) Proximal Aggregation (gp) applied to low- level tokens (1 < L 1), which groups and merges tokens within spatially proximal patches. This paradigm not only preserves spatial coherence but also retains the fine- -neiird details that are essential for discriminative representation. (b) Holistic Aggregation (gh) applied to (1 = L) tokens, which leverages a global perspective for aggregation. It efficiently integrates long-range contextual patterns, thereby facilitating compact and semantically rich features. Gen-
1
Wmm
(6)
n n
Next, we rank Wi with decreasing order and filter top-M' by indexing the set of selected indices as follows:
S =
(M')),
(7)
in which _ = argsort( Wi, descending). To go step fur- ther, the number of remaining tokens after sparsification is M' + (M - M') - n², which is consistent with that K₁ = N₁ x pi (see Sec. 3.2). In this way, M' essentially comes as follows:
N₁ (1 pi)
M'
(8)
n2 -
5

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
Algorithm 1 Affinity Matching and Aggregation.
To preserve token relationships for further processing, we employ affinity matching strategy. Each unimportant token is paired with its most similar important tokens, and the corresponding visual patterns are aggregated into these counterparts. Then, the import-to-unimpontt trix Ag2a E - is derived from A. For each unimportant token, the top-7 similar tokens are identified by selecting the second-larged similarity value from each row of Aizu- We conduct ablations to procure the optimal setup of T, and further details are provided in Sec. A.
Input: Important token set V {2, 1, 2, ................................ Kr}: Unimportant token set U {z, i ..2. Ni
Ki}; import-o--nnimprrtan affinity matrix Asu E R(N-K)XNN
affinity ma-
Output: Updated important tokens 2, = {2: l
1.2..... Ki}
Initialize & < 2,, {i = 1, ..., Ki).
for to N₁ K2 do
ids, affs SorrReverse(A12u Delete [d][ [ff for 1 to T do
After identifying the top-I' tokens for each unimportant token, we verify whether these tokens belong to the set of important tokens V. If they do, the representation of the unimportant token aggregated into its corresponding important tokens. Note such one-to-many matching strategy can optimally embed the information from tokens in U. Moreover, the overall matching and aggregation pipeline is summarized in Algorithm 1.
if' Widssj] E {i = 1, k then
أزر=ي + Merge E V. 2, E
end if end for end for
Thereafter, within each qualified patch, the n² tokens are merged to form super token which replaces the bottom- right one of current patch. Then, the aggregated token set P= = {z/li = (11) - (()) nz...... .......(MM)))... (MM') n²²), while the remaining unmerged patches are flattened into T. The final output tokens include both the aggregated super tokens and unmerged ones which can be formulated as follows:
This token merging process not only preserves the seman- tic richness of the discarded tokens but also enhances the expressiveness of the important tokens by enriching them with complementary information.
3.4. Representational Center-distance Regularization
Let represent the standard forward-pas representation and Ż the corresponding sparsified pattern. In an ideal scenario, the distribution of predictions based representa- tions between the pre- (Ppro)) and post-sparsification (ppost) stages should exhibit minimal deviation, thereby effectively reducing sparsificationinnueed information loss. Unfortu- nately, this is infeasible in practical implementations due to inherent limitations.
2, PUT.
(9)
Thanks to the rigorous evaluation of semantic integrity across all patches, our proximal aggregation strategy en- sures the integration of semantically meaningful tokens, thereby achieving an optimal balance between computa- tional efficiency and representational quality.
High-level: Holistic Aggregation. Inspired by the bipartite soft matching mechanism (Bolya et al., 2023), we propose tailored strategy to aggregate high-level tokens from holistic fashion.
To mitigate this thorny issue without introducing auxiliary burdens, we propose representational center-distance reg- ularization, which aligns the centroids of sparsified and non- con-spariified representations:
Given high-level tokens Z = f f i Ni). we first construct the affinity matrix E RMXM by capturing the cosine similarity between all tokens with Eq. Subse- quently, the importance score for the i-th token is defined as follows:
Lalign = || - Ppost 1|2,
(11)
in which Upre = Eppce [Z) and 1/post Eppen If 2
stand for the representation centroids of the non-sparsified and sparsified models, respectively. Then, the overall objec- tives can be formulated as follows:
N
(10)
No
j=ljih ei
To go step further, tokens exhibiting lower similarity to others are assigned higher importance scores, as they are presumed to encapsulate more distinctive information. Next, we rank the importance sequence in descending order. Ac- cording to Eq. 3. the top K2 tokens are designated as im- portant tokens V = {z. i = 1, 1,2.... K2), while the remaining (Nr Ki) tokens are identified as unimportant
Listal Lede + X Lallgnn
(12)
where Laet is the detection loss of the original model and X balances the contributions of different loss terms which is empirically default to 0.1 in our experiments. This regu- larization promotes alignment between centroids, ensuring consistent representation quality across sparsification levels and enhancing both robustness and generalization.
ones 11 = {2, j = 1, 1,2,... .......................... N1- K:).
6

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
Table 1: Comparison of Dynamic DETR with DETR variants and other efficient DETR algorithms on the COC02017 wal-set. DETR, DAB DETR, DN DETR, Conditional DETR and Anchor DETR use a single DC5 feature, while the remaining detectors the comparison employ multi-level features. "D-DETR" refers to Deformable DETR. Noting the lop-performing results are highlighted in Red while the second- best ones are in Blue.
Model
Paradigm
Epochs
AP
APpo
AP
Ts
APs
APM
APL
Params(M)
FLOPs(G)
FPS
DETR (Carion et al. 2020)
500
43.3
63.1
45.9
22.5
47.3
61.1
41,0
187.0
11.2
DAB DETR (Liu et al., 2022)
50
44.5
65.1
47.7
25.3
48.2
62.3
44.0
203.0
DN DETR (Li et al., 2022)
50
46.3
66.4
49.7
26.7
50.0
64.3
44.0
202.0
Conditional DETR (Yao et al., 2021)
108
45.1
65.4
48.5
25.3
49.0
62.2
44.0
195.0
11.5
Anchor DETR (Wang et al., 2022)
44.2
64.7
47.5
24.7
48.2
60.6
37.0
171.5
25.0
Original
47.0
66.1
50.9
30.0
49.8
61.9
41.0
179.0
19.1
Sparse (Roh al., 2022)
46.0
65.9
49.7
29.1
49.1
60.6
41,0
121.0
23.2
D-DETR (Zhu et al,, 2021)
50
Lite al., 2023)
45,8
65.1
49.3
27.7
49.1
61.1
41.0
1M07%
24.0
Dynamic
46.0
65.1
50.1
29.0
48.9
60.6
41.0
107.9,977% po07%
251299
Original
50.9
68.9
55.3
34.6
54.1
64.6
47.0
244.5
14.4
Sparse (Roh al., 2022)
48.2
65.9
52.5
30.4
51,4
63.1
47.0
152.0
24.2,40.99
DINO (Zhang al., 2022)
Lite (Li al., 2023)
36
50.4
54.6
33.5
53.6
65.5
47.0
151.0 151.0 p.w.3%
y3.2
Focus (Zheng et al., 2023)
50.4
68.5
55.0
34.0
53.5
64.4
48.0
154.0
20.0
Dynamic
50.2
69.2
54.7
33.6
53.4
64.4
47.0
141.7
23.2 (6..15
DAB-
Original
46.9
66.0
50.8
30.1
50.4
62.5
44.0
235.4
14.8
(Liu et al., 2022)
D-DETR
Dynamic
45,8
64.4
49.7
29.0
49.3
60.2
44.0
131.8
22.615279
DN-
Original
48.6
67.4
52.7
31.0
52.0
63.7
48.0
231.3
18.5
D-DETR eti 2022)
Dynamic
47.7
66.5
51.8
30.3
50.6
62.6
48.0
139 139.81m96
23.0
N-
Original
50,0
32.9
52.7
65.3
48.0
234.8
6.7
D-DETR (Jia ul., 2023)
Dynamic
49.1
65.7
54.0
32.7
52,9
64.3
48.0
123.6147.48
9.3-388%
4. Experiments
4.2. Experimental Results
4.1. Implementation Details
4.2.1. MAIN RESULTTS
All experiments are implemented using the detrex frame- work (Ren al., 2023). Training is performed on 4 Nvidia GPUs with a batch size of 2. To ensure fair comparisons, we strictly follow the configurations of baseline models, including hyperparameters, network architectures, and loss functions.
Comprehensive results are summarized in Table 1, contrast- ing DETR variants, efficient DETR algorithms, and the enhanced versions with our Dynamic DETR. When applied to Deformable DETR, Lite DETR reduces FLOPs by 58.0G (32.4%) with a minor 1 1.0% drop in accuracy. In compari- son, Dynamic DETR achieves a more substantial reduction of FLOPs (1.39.7%), while outperforming Lite DETR with comparable FLOPPs and an additional 0.2% gain in precision. Sparse DETR achieves the same accuracy but incurs more FLOPs (13.IG) than our method. For DINO, Dynamic DETR reduces the overall computational cost by 42.0%, with only a 0.7% drop AP. contrast, Lite DETR and Focus DETR reduce FLOPs by 38.2% and 37.0%, respec- tively, with slightly smaller performance drops (0.5% AP). Dynamic DETR achieves the best trade-off by offering higher reduction in computational cost while maintaining competitive accuracy.
The primary experiments are performed on the COCO 2017 dataset, which contains 118K training images and 5K vali- dation images. We adopt ResNet-50 as the backbone, and use 6 encoder blocks and 4 pyramid levels by default. To evaluate the generalizability of our approach, we addition- ally conduct experiments on LVIS v1.0 (Gupta et al., 2019) and PASCAL VOC 2007 (Everingham et al.) within the DINO framework. LVIS v.0 (Gupta et al., 2019) presents large-vocabulayy setting with long-tailed category distribu- tion, challenging models detect rare objects. In contrast, PASCAL VOC 2007 (Everingham et al.) is smaller-scale dataset that focuses on simpler object layouts and limited training samples. Moreover, we evaluate our method with the Swin-T backbone (Liu et al., 2021), a hierarchical vision transformer, to assess its effectiveness.
We further evaluate Dynamic DETR on three advanced DETR variants: DAB-D-DETR, DN-D-DETR, and H-D- DETR. Dynamic DETR can reduce the models' compu- tational cost by varying amounts (ranging from 39.6% to 47.4%), while only incurring minimal performance loss (about 1.0%). Dynamic DETR demonstrates superior com- putational efficiency and accuracy across a wide range of DETR architectures. This, makes it an good choice for real-time and resource-constrained applications.
7

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
4.2.2. ADDITIONAL RESULTS
Table shows that Dynamic DINO consistently surpasses other variants across most metrics. It achieves the lowest FLOPs and the highest FPS, validating that the dynamic token aggregation strategy scales well with multi-stage, patch-based architectures. Compared to Sparse DINO, Dy- namic DINO improves AP by 0.3 points while reducing computational cost by over 9%, indicating better accuracy- efficiency trade-off.
To further assess the generalizability and robustness of our approach, we conduct supplementary experiments across multiple datasets and architectures. All models are trained for 12 epochs with consistent settings unless otherwise spec- ified.
Results on LVIS v1.0. Table 2 presents the results on LVIS v1.0, a large-scale benchmark featuring over 1200 categories with long-tailed distribution (Gupta et al., 2019). Dynamic DINO achieves strong trade-off between performance and efficiency. While Focus DINO yields the highest overall AP and AP,, Dynamic DINO delivers a competitive AP of 23.4 and achieves the best y Notably, our method reports the lowest FLOPs and the highest FPS among all variants, highlighting its efficiency advantage in long-tailed detection scenario.
Table 4: Performance comparison using the Swin-T back- bone.
Model
AP
APTs
APs
APM
APL
FLOPP(G)
FPS
DINO
$1.5
70.2
56.5
34.6
54.5
67.0
252.3
14.0
Sparse DINO
49.6
68.4
54.1
32.1
52.6
65.3
137,0
18.0
Lite DINO
48.3
66.1
52.8
30.3
$1.5
64.0
151.0
16.8
Focus DINO
49.9
68.2
54.3
32,9
52.8
65.1
156.9
15.3
Dy mumic DINO
49.9
68.8
54,3
32.8
52,9
65.4
149.4
18.2
Table 2: Detection performance on LVIS v1.0.
4.2.3. ABLATION STUDIES
Model
AP
APso
Alts
AP,
AP.
APr
FLOPs(G)
FTS
We perform all the ablation studies on DINO to evaluate the effectiveness of the proposed components. The results are summarized in Table 5.
DINO
34,5
36.1
247.1
Spurse DINO
22.9
32.0
24.2
8,4
21.3
30.9
151.7
21.2
DINO
28.0
21.4
17.5
30.8
160.0
16.0
Focus DINO
23.7
32.9
25.2
10.2
21.7
31.9
168.2
20.4
Dynamic DINO
31.8
25.0
7.7
20.8
33.4
146.6
Effect of the Dynamic Token Aggregation Mechanism.
To assess the impact of the Dynamic Token Aggregation (DTA) module, we replace it with static variant using a fixed token retention ratio of 0.3 across all pyramid levels. Under this setting, the overall AP achieved is 48.2%, which is 1.4 points lower than the full dynamic version. This performance gap highlights the limitations of static aggre- gation, which fails to adaptively prioritize token relevance, and demonstrates the effectiveness of our DTA strategy in selectively retaining informative tokens.
Results on PASCAL VOC 2007. As shown in Table 3, we evaluate our method on PASCAL VOC 2007, a com- pact dataset with 20 object classes and relatively simple scenes (Everingham et al.). Dynamic DINO attains the highest mAP (63.8%) among all lightweight variants, while consuming only 135.2 GFLOPs. Compared to the original DINO, our method maintains competitive accuracy with just 56% of the computation, and significantly outperforms Sparse DINO and Focus DINO in both accuracy and speed. Despite its lightweight design, Lite DINO significantly un- derperforms compared to its counterparts, potentially due to the necessity of prolonged training achieve satisfac- tory convergence. To sum up. the promising results further underscore the generality of our design in efficient object detection.
Effeet of Multi-level Token Aggregation. We further in- vestigate the role of Multi-level Token Aggregation (MTA), Disabling token aggregation across stages by randomly se- lecting and retaining tokens with dynamic ratios results in a significant performance drop to 46.6% AP, despite the reduc- tion in computational cost 134.7G FLOPs. Additionally, assess the role of affinity matching within the holistic to- ken aggregation process. Removing affinity matching yields an AP of 49.8%, while enabling it improves performance to 50.2%, with a minor increase of 1.5G FLOPs. These results demonstrate the effectiveness of hierarchical aggregation and highlight the advantages of incorporating semantic sim- ilarity matching in holistic token aggregation, achieving a balanced accuracy-efficiency trade-off.
Table 3: Detection performance on PASCAL VOC 2007.
Model
mAP
FLOPs(G)
FPS
DINO
65.7
241.6
15.5
Sparse DINO
62.5
141.4
19.6
Lite DINO
38.1
151.0
21.3
Focus DINO
51.4
152.1
20.2
Dynamic DINO
63.8
135.2
21.1
Effect of Representational Center-distance Regulariza- tion. Finally, we evaluate the impact of the Representa- tional Center-distance Regularization (RCDR). Incorporat- ing RCDR results in a significant performance improvement of 0.6% AP (from 49.6% to 50.2%) with no additional computational overhead. This result indicates that RCDR
Results on Swin-T Backbone. We also assess our method with Swin-T, T, a hierarchical vision Transformer (Liu et al., 2021), to examine its compatibility with modern backbones.
8

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
Table 5: Ablation studies of the proposed components on COCO 2017. We evaluate the individual and combined
contributions the Dynamic Token Aggregation (DTA), Multi-level Token Aggregation (MTA) with and without Affinity Matching (AM), and Representational Center-distance Regularization (RCDR).
DTA
MTA
RCDR
AP
50
AP75
APs
APM
APL
FLOPs(G)
X
X
46.6
68.3
50.1
29.9
49.9
63.1
134.7
(with AM)
X
48.2
64.5
52.9
30.9
52.1
63.1
140.6
/ (w/o AM)
V
49.8
69.9
54.2
33.1
53.4
65.4
140.2
(with AM)
X
49.6
68.1
54.2
32.0
52.9
65.2
141.7
(with AM)
50.2
69.2
54.7
33.6
53.4
64.4
141.7
Stage-1
Stage-2
Stage-3
features are preserved, reflecting the model's ability to pri- oritize salient information during the aggregation process.
Leveb-1
5. Conclusion
In this work, we proposed Dynamic DETR (dynamic token aggregation for detection transformers) to tackle the compu- tational bottlenecks inherent in DETRs. By leveraging token redundancy at multiple levels and across different encoder blocks, Dynamic DETR dynamically adjusts token repre- sentations while preserving essential semantic information. Through extensive experiments various DETR variants, Dynamic DETR demonstrates state-of-the-art performance in terms of speed and accuracy, surpassing existing token sparsification competitors. Its flexibility allows to be in- tegrated seamlessly into diverse architectures, paving way for efficient DETRs. Future work will explore adaptive sparsification strategies tailored to varying input character- istics and task requirements. Additionally, aim to extend Dynamic DETR to other Transformer-basee architectures, broadening its applicability to a wider range of tasks in computer vision and beyond.
Levb2
Leve-3
Leve-4
Figure 5: The distribution of important tokens, in which
unimportant ones are masked.
A Acknowledgements
This work supported in part by the National Natural Sci- ence Foundation of China under Grant 62471398 and Grant 621 62136007, in part by the National Science and Technology Major Project of China under Grant 2022ZD01 11222D011900 19005.
effectively enhances feature discrimination and improves model generalization through lightweight yet principled regularization scheme.
4.2.4. VISUALIZATION AND QUALITATIVE ANALYSIS
Impact Statement
We visualize the distribution of important tokens across three stages Figure 5. Consistent with our design, earlier stages retain higher proportion of tokens at lower levels, while the focus gradually shifts toward higher levels in later stages. This demonstrates that critical information is effec- tively preserved throughout the merging process. At lower levels (e.g., Levels 1-3), proximal tokens corresponding to spatially similar regions, are merged. In contrast, at higher levels, token aggregation is guided by the global semantic relationships. Notably, tokens with unique or distinctive
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your vit but faster. In

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
International Conference on Learning Representations (ICLR), 2023.
transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp- 15741- 15750, 2024.
Cao, Z., Fu, C., Li, B., and Li, Y. Hift: Hierarchical feature transformer for aerial tracking. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15457-15466, 2021.
Li, F., Zhang, H., Liu, S., Guo, J., Ni, M., and Zhang, L. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 13619- 13627, 2022.
Carion, Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. European Conference on Computer Vi- sion (ECCV), pp. 213-229. Springer, 2020.
Li, F., Zeng, A., Liu, S., Zhang, H., Li, H., Zhang, L., and Ni, I. M. Lite detr: An interleaved multi-scale encoder for ef- ficient detr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), PP. 85558-18567, 2023.
Chang, S., Wang, P., Lin, M., Wang, F., Zhang, D. J., Jin, R., and Shou, M. Z. Making vision transformers efficient from a token sparsification view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. $195-6205. 2023.
Liang, Y., Chongjian, G., Tong, Z., Song, Y., Wang, J., and Xie, P. Evit: Expediting vision transformers via token reorganizations. 2021.
Chen, P., Zhang, M., Shen, Y., Sheng. K., Gao, Y., Sun, X., Li, K., and Shen, C. Efficient decoder-free object detection with transformers. In European Conference on Computer Vision (ECCV), pp. 70-86. Springer, 2022a.
Lin, S., Xie. H., Wang, B., Yu, K., Chang, X., Liang, X., and Wang, G. Knowledge distillation via the target-aware transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition (CVPR), pp. 10915-10924, 2022.
Chen, X., Wei, F., Zeng. G., and Wang, J. Conditional detr v2: Efficient detection transformer with box queries. arXiv preprint arXiv:2207.0000 089144 2022b.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. European Conference on Computer Vision (ECCV), pp. 740-755. Springer, 2014.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.., Heigold, G., Gelly, S., et al. An image is worth 16x 16 words: Transformers for image recognition scale. In International Conference Learning Representations (ICLR), 2020.
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2117-2125, 2017.
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal- html.
Liu, S., Li, F., Zhang, H., Yang, X., X., J., and Zhang, L. Dab-detr: Dynamic anchor boxes are better queries for detr. International Conference on Learning Representations (ICLR), 2022.
S, Su, H., Zhu,
Gao, Y., Shen, L., Liu, 1., Cao, Z., Tao, D., Ma, H., and Kato, N. Neural-aware decoupling fusion based person- alized federated learning for intelligent sensing. ACM Transactions Sensor Networks, 20(6): 1-21, 2024.
Liu, Y., Gehrig, M., Messikommer, N., Cannici, M., and Scaramuzza, D. Revisiting token pruning for object de- tection and instance segmentation. In IEEF/CVF Winter Conference on Applications of Computer Vision ( (WACV), pp. 2658-2668, 2024.
Gupta, A., Dollar, P., and Girshick, R. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), 2019.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 10012-10022, 2021.
Jia, D., Yuan, Y., He, H., Wu, X., Yu, H., Lin, W., Sun, L.,, Zhang, C., and Hu, H. Detrs with hybrid match- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19702-19712, 2023.
Long, S., Zhao, Z., Pi, J., Wang, S., and Wang. J. Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10334-10343, 2023.
Lec, S., Choi, J., and Kim, H. J. Multi-criteria token fu- sion with one-step-ahead attention for efficient vision
10

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
Lu, C., de Geus, D., and Dubbelman, G. Content-aware token sharing for efficient semantic segmentation with vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 23631-23640 2023.
Yin, H., Vahdat, A., Alvarez, J. M., Mallya, A., Kautz, J., and Molchanov, P. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10809-10818 2022.
Meng, D., Chen, X., Fan, Z., Zeng, G., Li. H., Yuan, Y. Sun, L.,, and Wang, J. Conditional detr for fast training con- vergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 3651-3660, 2021.
Zhang, G., Luo, Z.,, Tian, Z.,, Zhang. J., Zhang, X., and Lu, S. Towards efficient use of multi-scale features in transformer based object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6206-6216 June 2023.
Pu. M., Huang, Y., Liu, Y. Guan, Q. and Ling, H. Edter: Edge detection with transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1402-1412, 2022.
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H.,, Zhu, J., Ni. L., and Shum, H.-Y. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In Interna- tional Conference on Learning Representations (ICLR), 2022.
Raganato, A. and Tiedemann, J. An analysis of encoder representations in transformer-baseee machine translation. In Proceedings of the 2018 EMNLP workshop Black- boxNLP: analyzing and interpreting neural networks for NLP, pp. 287-297, 2018
Zhang, I.., Li, Y., Zhou, X., Zhao, X., and Gu, S. Transcend- ing the limit of local window: Advanced super-resolution transformer with adaptive token dictionary. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2856-2865. 2024.
Rao, Y. Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsich, C.-J. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Information Processing Systems (NeurlPS), 2021.
Zheng, D., Dong, W., Hu, H., Chen, X., and Wang, Y. Less is more: Focus attention for efficient detr. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.
Ren, T., Liu, S., Li, F,, Zhang, H., Zeng, A., Yang, J., Liao, X., Jia, D., Li, H., Cao, H., Wang, J., Zeng, 7., Qi, X., Yuan, Y., Yang, J., and Zhang, L. detrex: Benchmarking detection transformers, 2023.
2023.
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai. J. De- formable detr: Deformable transformers for end-to-end object detection. In International Conference on Learn- ing Representations (ICLR), 2021.
Roh, B., Shin, J., Shin, W., and Kim, S. Sparse detr: Effi- cient end-to-ent object detection with learnable sparsity. In International Conference on Learning Representations (ICLR), 2022.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Wang, T., Yuan, L., Chen, Y., Feng, J., and Yan, S. Pnp-detr: Towards efficient visual analysis with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4661-4670, 2021.
Wang, Y., Zhang, X., Yang, T., and Sun, J. Anchor detr: Query design for transformer-base detector. In Pro- ceedings of the AAAI conference artificial intelligence (AAAI), volume 36, pp. -575,,275, 2022.
Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L., and Gao, J. Focal self-attention for local-global interactions in vision transformers. arXiv preprint 2rXiv:2107.006411 2021.
Yao, Z., Ai, J., Li, B., and Zhang, C. Efficient detr: improv- ing end-to-end object detector with dense prior. arXiv preprint arXiv:2 104.00 0/318, 2021.

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
A. Discussions
To gain deeper insights into the effectiveness and robustness our method, we conducted series experiments to analyze the impact of various hyperparameters and design choices.
Choice of Token Retention Ratios. We present systematic investigation into the impact of dynamic token retention ratios on model performance through comprehensive experimental framework. Our methodology evaluates retention ratio configurations by exploring an upper bound range of 0.4 to 0.8 and a lower bound range of 0. to 0.5, with detailed results documented in Table 6. As retention ratios increase, both computational complexity (measured FLOPs) and model accuracy improve. For example, increasing retention ratios from (0.4, 0.3, 0.2, 0.1) to (0.8, 0.7, 0.6, 0.5) results a substantial 29.7% increase in computational requirements, with FLOPs rising from 107.9G 140.0G for Deformable DETR. Concurrently, the AP metric exhibits modest improvement (from 46.0% 46.8%). This empirical evidence highlights critical trade-off: while higher retention ratios consistently enhance the overall performance, the marginal gain in accuracy is disproportionately small compared to the significant increase in computational overhead. Based on a comprehensive analysis of these metrics, we propose an optimized configuration strategy that carefully balances token selection ratios for each model. This maximizes the efficiency-accuracy trade-off, ensuring optimal performance across evaluation metrics while maintaining computational feasibility.
Table 6: Performance comparison of different token retention ratios on different models. We include the purple results in Table
D-DETR (Zha et al., 2021)
DINO (Zhang et al., 2022)
DAB DEDER (Liu al., 2022)
DN D-DETR (Li et al., 2022)
H D.DEER (Jin et al., 2023)
FLOPs())
FLOPs(G)
AP
FLOPs(G)
FLOPs()
AP
FLOPs(G)
0.2.0.3.
46.0
107.9
49.9
132.6
45.4
120.1
41.5
128.4
49.1
123.6
0...0..4)333.0...
[0.6. 0.5, 0.4.0,
46.5
122.9
50.5
159.3
46.1
146.2
47.9
154.5
49.4
149.6
10.7. 0.6. 0.4)
131.5
174.5
46.2
49.5
[0.8. 0.7, 0.6,0.676600000 0.5)
46.8
140.0
50.8
189.5
46.3
176.4
48.3
184.7
49.7
179.8
Choice of Options for Token Aggregation. We demonstrate the optimal choices of token retention ratios on various models. For consistency and convenience in further discussions, all subsequent experiments are conducted on DINO (Zhang et al., 2022).
Table 7: Performance comparison across different experimental settings. All experiments conducted DINO. We
include the purple results in Table and Table 6.
Method
AP
AP30
AP7s
FLOPs(G)
Retain
49.7
68.2
53.8
141.7
Sum
49.9
68.5
54.1
141.7
Max
50.0
69.6
54.5
141.7
Mean
50.2
69.2
54.7
141.7
(a): Different token aggregation strategies.
AP
APto
AP7s
FPS
FLOPs(G)
À
AP
AP
AP7s
FLOPs(G)
50.1
69.5
54.5
25.2
141.7
0.01
49.7
69.4
54.1
141.7
50.2
69.2
54.7
23.3
141.7
0.03
49.8
69.8
54.3
141.7
50.2
69.4
54.6
18.4
141.7
0.1
50.2
69.2
54.7
141.7
50.2
69.6
54.6
12.1
141.7
0.8
49,9
69.9
54.4
141.7
50.3
69.6
54.6
7.9
141.7
1
49.4
69.6
53.5
141.7
(b): The number of aggregation tokens.
(c): Varying setups of the regularization coefficient X.
As outlined in the Proximal Aggregation section, the tokens within each eligible patch are merged to form a super token, which retained the bottom-right corner of the current patch. Subsequently we examine how various aggregation
12

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
strategies affect model performance during unified representation generation, including: (a) Retain: Preserving the token located the bottom-right corner of n region; (b) Sum: Summing all tokens within the current patch; (c) Max: Indexing the maximum value among all tokens within the current patch; (d) Mean: Calculating the average of all tokens within the current patch. The results summarized in Table 7(a) demonstrate that all three merging strategies outperform the non-merging baseline (Retain), highlighting the benefits of token aggregation for efficient modeling. Among these strategies, Mean achieves the best performance, with 50.2% AP, 69.2% S0 and 54.7% AP7s. We attribute the superior performance of Mean to its ability to evenly distribute information across tokens during aggregation. This approach preserves semantic consistency while reducing redundancy, making it particularly effective for compressing token representations without sacrificing discriminative power.
The Number of the Most Similar Tokens in Holistic Aggregation. We investigate the effect of varying the number of most similar tokens, T, set to 1, 3, 5, 7, and 9. The results, shown in Table 7(b), indicate that propagating information into a larger number of tokens generally enhances performance compared to using only one or three tokens. This improvement stems from the robustness of distributing information across multiple tokens. Specifically, aggregating into multiple tokens mitigates the impact of noise or erroneous information in individual tokens, as the other tokens are more likely to contain accurate and complementary information. However, the performance gains become marginal when T exceeds 3. Additionally, increasing T' beyond this point causes decline in inference speed, from 23.3 FPS to 7.9 FPS. Considering the trade-off between effectiveness and efficiency, we select T = 3 as the optimal setting for the number of the most similar tokens in our experiments.
Coefficient of Representational Center-distance Regularization Term. We analyze the effect of the regularization coefficient À in the representational center-distance regularization term model performance. A range of X values, from 0.01 to 1, is tested, and the results are summarized in Table 7(c). The results show that X = 0.1 achieves the best balance between representation alignment and model performance, resulting in an AP of 50.2%. When À is set too high (e.g.. À 0.8 or 1). the model becomes over-regularized, focusing too heavily on aligning representation centroids at the cost of learning meaningful task-specific features, which leads to a decline in overall accuracy. On the other hand, when À is too low (e.g.. À= = 0.01 or 0.03), the regularization effect is insufficient, leading to weak alignment between sparsified and non-sparsified representations, which results in suboptimal sparsification performance. Based on these observations, we select 0.1 as the optimal value.
B. A Additional Visualization
Due to the application of our multi-level token aggregation strategy across various stages, direct visual comparison with existing coarse-gras coarse-grained token discarding approaches poses a challenge. To ensure a fair and interpretable evaluation, we visualize the top 300 decoder queries extracted from the encoder outputs of the vanilla DINO model (Zhang et al., 2022), Sparse DETR (Roh et al., 2022), and our proposed Dynamic DETR. These queries are directly responsible for the detection results, as illustrated in Figure 6.
In Figure 6, selected decoder queries are visualized as yellow regions, while unselected tokens are marked in purple. Sparse DETR predominantly focuses on inner areas of the most salient objects, effectively isolating them. However, it neglects background regions and other objects that are critical for accurate detection, particularly at higher levels. In contrast, our proposed Dynamic DETR incorporates a more comprehensive approach by preserving the intrinsic relationships among tokens, leading to token distribution that more closely resembles the original DINO. This alignment ensures that both salient objects and essential contextual information retained, enhancing the robustness and accuracy of the detection process.
13

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
Level-1
Level-2
Level-3
Level-4
Level-1
Level-2
Level-3
Level-4
DINO
Sparse DETR
Dynamic DETR
Level-1
Level-2
Level-3
Level-4
TUNNSS PIRGNUM WINNTTOKE
HAGNUM
WINSTONE
MINES STL
HAGGNUM
SWINSTANE RINOS FIAGNUM
WWIRSTONE
DINO
LOIN STUL
MIN SOU
TWIMSTONE
MIN HAGNUM WINSTONEE
JOM STL MAGNUM WINSTONE
Sparse DEIR
BOVES STI PIAGNUM WINBTONE
MNN STIT MAGNUM AW AWINSTONE
m STD
AOOUUM w NONNE
IAGNUM STONES
Dynamic DETR
14

---

Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers
Level-1
Level-2
Level-3
Level-4
DINO
Sparse DETR
Dynamic DETRR
Level-1
Level-2
Level-3
Level-4
DINO
Sparse DETR
Dynamic DETRR
Figure 6: Visualization of the Top-300 decoder queries from the encoder output. Note the selected queries are highlighted with yellow.
15


Reviews:
[{"review_id": "lvKBomgF2P", "reviewer": "Reviewer_sTgo", "summary": "The paper **\"Not All Tokens Matter All The Time: Dynamic Token Aggregation Towards Efficient Detection Transformers\"** proposes a novel framework called **Dynamic DETR**, aiming to address the computational efficiency bottleneck in **Detection Transformers (DETRs)**. DETRs require high computational resources, especially in the encoder, which becomes a major bottleneck. Existing methods generally adopt **static token sparsification strategies**, ignoring the differences in token importance across different layers and encoder blocks, leading to performance degradation. Dynamic DETR significantly reduces computational costs while maintaining high detection accuracy by dynamically adjusting token density and incorporating a multi-level token sparsification strategy. The main contributions include:\n\n**Dynamic Token Aggregation:** Dynamically adjusts token density to reduce redundancy and computational complexity.\n\n**Multi-level Token Aggregation:** Employs neighbor-based aggregation at lower levels to preserve spatial details and global aggregation at higher levels to capture contextual information.\n\n**Representational Center-distance Regularization:** Ensures consistency in feature distribution before and after sparsification through regularization, improving detection performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your selfless dedication and thoughtful comments. Below we response to these concerns.\n\n## W1&Q1: The paper does not discuss the real-world deployment performance of Dynamic DETR, such as its effectiveness in real-time detection tasks\n\n**Response**: \nTo verify the potential of our dynamic token aggregation in real-time detection, we investigate the performance of several efficient detectors when integrated a lightweight backbone MobileNet-V2, and the results are shown in **Tab. T1**. \n\nTab. T1. Performance of DINO and various efficient solutions with MobileNet-V2 on the COCO val-set, where the output channels are set to 256 for convergence and trained for 12 epochs.\n|Model|AP|AP$_{\\mathrm{50}}$|AP$_{\\mathrm{75}}$|FLOPs (G)|FPS|\n|-|:-:|:-:|:-:|:-:|:-:|\n|DINO|25.9|38.7|27.5|172.6|24.0|\n|Sparse DINO|19.8|33.8|20.7|67.5|27.2|\n|Lite DINO|18.8|29.2|19.9|78.4|**36.2**|\n|Focus DINO|20.0|31.8|20.7|79.8|24.7|\n|Dynamic DINO|**21.7**|**33.1**|**23.1**|**61.5**|36.0|\n||\n\nDynamic DINO significantly boosts the inference performance from 24.0 to 36.0 while the accuracy also outperforms its counterparts by a large margin. This competitive result shows the potential adaptability of our method for practical scenarios.\n\nMoreover, to enable a fair and comprehensive comparison, our experiment setups strictly align with pioneering works [R1, R2, R3], which primarily focus on benchmark evaluations and do not explicitly consider deployment performance. Future work could explore lightweight adaptations and hardware-friendly implementations to further bridge the gap between research and practical deployment.\n- [R1] Roh, B., et al. Sparse detr: Efficient end-to-end object detection with learnable sparsity. ICLR, 2022.\n- [R2] Li, Feng, et al. Lite detr: An interleaved multi-scale encoder for efficient detr. CVPR, 2023.\n- [R3] Zheng, Dehua, et al. Less is more: Focus attention for efficient detr. ICCV, 2023.\n\n## W2&Q2: Although Dynamic DETR performs well on COCO, its performance on other datasets (e.g., Pascal VOC) remains unverified.\n\n**Response**: To demonstrate the generalizability and robustness of our Dynamic DETR, as suggested by your valuable comment, we perform experiments on VOC and LVIS datasets, two of the most commonly used benchmarks beyond COCO for object detection. The results are as exhibited as **Tab. T2** and **Tab. T3**. Note that all the models are with a ResNet-50 as the backbone and trained for a bunch of 12 epochs. \n\nTab. T2. Performance of DINO and various efficient solutions on the VOC2007 val-set.\n| Model| mAP|FLOPs (G)|FPS|\n|-|:-:|:-:|:-:|\n| DINO|65.7| 241.6|15.5|\n| Sparse DINO|62.5|141.4|19.6|\n| Lite DINO |38.1|151.0|**21.3**|\n| Focus DINO|51.4| 153.6|20.2|\n| Dynamic DINO|**63.8**|**135.2**|21.1|\n||\n\nTab. T3. Performance of DINO and various efficient solutions on the LVIS-1.0 val-set.\n| Model | AP  | AP$_{\\mathrm{50}}$  | AP$_{\\mathrm{75}}$  | AP$_{\\mathrm{r}}$  | AP$_{\\mathrm{c}}$  | AP$_{\\mathrm{f}}$   | FLOPs (G) | FPS  |\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| DINO|26.1|34.5|27.5|8.3|24.1|36.1|247.1|19.8|\n| Sparse DINO| 22.9 | 32.0|24.2|8.4|21.3|30.9|151.7|21.2|\n| Lite DINO| 20.2 | 28.0|21.4|3.0|17.5|30.8|160.0|16.0|\n| Focus DINO| **23.7** | **32.9** | **25.2**|**10.2**|**21.7**|31.9|168.2|20.4|\n| Dynamic DINO| 23.4 | 31.8|25.0 |7.7|20.8|**33.4**|**146.6**|**22.5**| \n||\n\nConsistent with the performance on the COCO dataset, the proposed dynamic token aggregation significantly reduces the parameters of the baseline model (DINO), while also exhibiting a smaller performance loss compared to other efficient solutions. Specifically, as exhibited in **Tab. T2**, our Dynamic DINO scores 63.8% AP on the VOC dataset, which is 1.9% points lower than the baseline DINO but with a 36% improvement in FPS, and excels other competitors by a large margin both in accuracy and efficiency. For the LVIS results in **Tab. T3**, our Dynamic DINO lags Focus DINO slightly by 0.3% points, but outperforms it in inference speed by 2.1 FPS.\n\nIn summary, the results on VOC and LVIS datasets further showcase the superiority and generality of our Dynamic token sparisification strategy.\n\n## Q3: Is Dynamic DETR applicable to other Transformer architectures (e.g., ViT)? Are there plans to explore this aspect?\n\n**Response**: To further explore the generalizability of Dynamic DETR, we conducted additional experiments using Swin Transformer as the backbone. The results in **Tab. T4** demonstrate that our approach remains effective across different Transformer architectures, highlighting its adaptability.\n\nTab. T4. Performance of DINO and various efficient solutions with  Swin-Transformer on the COCO val-set.\n| Model | AP  | AP$_{\\mathrm{50}}$  | AP$_{\\mathrm{75}}$|FLOPs (G) | FPS  |\n|-|:-:|:-:|:-:|:-:|:-:|\n|DINO|51.5|70.2|56.5|252.3|14.0|\n|Sparse DINO|49.6|68.4|54.1|137.0|18.0|\n|Lite DINO|48.3|66.1|52.8|151.0|16.8|\n|Focus DINO|**49.9**|68.2|54.3|156.9|15.3|\n|Dynamic DINO|**49.9**|68.8|54.3|**149.4**|**18.2**|\n||", "claims_and_evidence": "The claims made in the paper are well-supported by extensive experiments. The authors conducted experiments on the **COCO2017 dataset**, demonstrating the effectiveness of **Dynamic DETR**. The results show that **Dynamic DETR significantly reduces FLOPs (by 39.7%-53.9%)** across multiple DETR variants while incurring only **a slight performance drop (AP decrease of 0.5%-1.0%)**. Additionally, ablation studies confirm the effectiveness of dynamic token aggregation, multi-level aggregation strategies, and representational center-distance regularization.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are reasonable and well-suited for object detection tasks. **Dynamic DETR** effectively reduces redundant tokens while maintaining detection accuracy through dynamic token density adjustment and multi-level aggregation. The experiments are conducted on the **COCO2017 dataset**, with evaluation metrics including **AP, AP50, and AP75**, ensuring comprehensive assessment.", "theoretical_claims": "The paper does not propose theoretical proofs, so there is no need to verify the correctness of theoretical claims.", "experimental_designs_or_analyses": "The experimental design and analyses are reasonable and effective. The authors evaluate Dynamic DETR on **multiple DETR variants**, demonstrating its **generalizability and effectiveness**. Additionally, ablation studies confirm the contribution of each component, ensuring the reliability of the experimental results.", "relation_to_broader_scientific_literature": "This work is closely related to existing **DETR improvement methods** and **token sparsification algorithms.** Existing DETR variants (e.g., **Deformable DETR, DAB-DETR**) improve detection performance but still have **high computational costs**. **Dynamic DETR** addresses this limitation by **introducing a dynamic token aggregation strategy**, significantly reducing computational costs while maintaining high detection accuracy, filling the gap left by previous methods.", "essential_references_not_discussed": "The paper cites a large number of relevant references covering DETR variants and token sparsification algorithms. No critical missing references were identified.", "other_strengths_and_weaknesses": "**strengths:**\n\n- **Dynamic DETR effectively reduces computational costs** while maintaining high detection accuracy by dynamically adjusting token density.\n\n- **Multi-level token aggregation strategy and representational center-distance regularization** enhance model robustness and generalization ability.\n\n- **Comprehensive experimental design**, validating the method's effectiveness and generalizability across multiple DETR variants.\n\n**Weaknesses:**\n\n- The paper **does not discuss the real-world deployment performance** of Dynamic DETR, such as its effectiveness in **real-time detection tasks.**\n\n- Although **Dynamic DETR performs well on COCO**, its performance on **other datasets (e.g., Pascal VOC) remains unverified.**", "other_comments_or_suggestions": "- The experimental results and analyses are comprehensive, but the authors could further discuss **the potential of Dynamic DETR in practical applications.**\n\n- It is suggested that the authors explore **Dynamic DETR's applicability in other Transformer architectures** in future work to verify its generalizability.", "questions_for_authors": "1. **How does Dynamic DETR perform in real-world applications, such as real-time detection tasks? Are there any plans to conduct related experiments?**\n\n    - **Impact**: Understanding Dynamic DETR’s real-world performance helps assess its deployment potential.\n\n2. **How does Dynamic DETR perform on other datasets (e.g., Pascal VOC)? Are there plans for cross-dataset validation?**\n\n    - **Impact**: Cross-dataset validation can further demonstrate the generalizability and robustness of Dynamic DETR.\n\n3. **Is Dynamic DETR applicable to other Transformer architectures (e.g., ViT)? Are there plans to explore this aspect?**\n\n    - **Impact**: Investigating Dynamic DETR’s applicability in other Transformer architectures can further verify its **generalizability and scalability.**", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4bxZ5xoGHY", "reviewer": "Reviewer_6br4", "summary": "It is known in the object detection literature that the detection transformers are notorious for their long and computationally demanding training requirements. To partially address this issue, this work proposes a novel token aggregation strategy for detection transformers based on the recent token merging strategies. In particular, the work aims to exploit the redundancy of the tokens at different feature levels dynamically by merging them. Experimental results on COCO2017 aim to highlight the efficiency gains of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comments. Below we response to these concerns.\n\n## C1: Comparable performance with Lite DETR\n\n**Response**: \nFirst of all, we are sorry for the incorrect parameter descrition about DINO and Dynamic DINO in the initial submission, while the corrected verison is as follows. Our anonymous code please refers to [here](https://anonymous.4open.science/r/Dynamic-DETR-4D7F)\n\nTab. T1. Corrected efficiency of DINO, Lite DINO, and Dynamic DINO on the COCO val-set.\n|Model|AP|FLOPs (G)|FPS|\n|-|:-:|:-:|:-:|\n|DINO|50.9|244.5|14.4|\n|Lite DINO|**50.4**|151.0|**23.2**|\n|Dynamic DINO|50.2|**141.7**|**23.2**|\n||\n\nTo throughly explore the advantages of our Dynamic strategy to Lite DETR, we take DINO as the baseline detector and compare the performance between these two methods as follows. \nOur method outperforms Lite-DETR across different datasets and backbone networks, with even greater advantages under a shorter training schedule, further demonstrating its efficiency.\n\n - On COCO val-set, Swin-T, 12 epochs.\n|Model|AP|FLOPs (G)|FPS|GPU Memory (G)|Training Hours (h:m)|\n|-|:-:|:-:|:-:|:-:|:-:|\n|DINO|51.5|252.3|14.0|27892|**23:27**|\n|Lite DINO|48.3|151.0|16.8|**26371**|24:20|\n|Dynamic DINO|**49.9**|**149.4**|**18.2**|27764|23:48|\n||\n\n - On LVIS val-set, ResNet-50, 12 epochs.\n|Model|AP|FLOPs (G)|FPS|GPU Memory (G)|Training Hours (h:m)|\n|-|:-:|:-:|:-:|:-:|:-:|\n|DINO|26.1|247.1|19.8|24244|14:59|\n|Lite DINO|20.2|160.0|16.0|42348|13:48|\n|Dynamic DINO|**23.4**|**146.6**|**22.5**|**22557**|**12:15**|\n||\n\n - On COCO val-set, MobileNet-V2, 12 epochs.\n|Model|AP|FLOPs (G)|FPS|GPU Memory (G)|\n|-|:-:|:-:|:-:|:-:|\n|DINO|25.9|172.6|24.0|15304|\n|Lite DINO|18.8|78.4|**36.2**|33950|\n|Dynamic DINO|**21.7**|**61.5**|36.0|**14082**|\n||\n\n - On VOC2007 val-set, ResNet-50, 12 epochs.\n|Model|AP|FLOPs (G)|FPS|\n|-|:-:|:-:|:-:|\n|DINO|65.7|241.6|15.5|\n|Lite DINO|38.1|151.0|**21.3**|\n|Dynamic DINO|**63.8**|**135.2**|21.1|\n||\n\n## C2: The usage of minival-set\n\n**Response**: Sorry for our unclear descriptions. All the results in our paper are evaluated on COCO val-set, which is strictly consistent with previous works (*Sparse detr, Lite detr, and Focus detr*). \n\n## C3: Experiemnts on LVIS dataset\n\n**Response**: We perform experiments on LVIS datasets to verify the generalizability and robustness of our approach.\n\nTab. T2. Performance of DINO and various efficient solutions on the LVIS-1.0 val-set.\n| Model | AP  | AP$_{\\mathrm{50}}$  | AP$_{\\mathrm{75}}$  | AP$_{\\mathrm{r}}$  | AP$_{\\mathrm{c}}$  | AP$_{\\mathrm{f}}$   | FLOPs (G) | FPS  | GPU Memory (G) |\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| DINO| 26.1 | 34.5 | 27.5 | 8.3 | 24.1 | 36.1 | 247.1 | 19.8 | 24244|\n| Sparse DINO| 22.9 | 32.0 | 24.2 | 8.4 | 21.3 | 30.9 | 151.7 | 21.2 | 22680|\n| Lite DINO| 20.2 | 28.0 | 21.4 | 3.0 | 17.5 | 30.8 | 160.0 | 16.0 | 42348|\n| Focus DINO| **23.7** | **32.9** | **25.2** | **10.2** | **21.7** | 31.9 | 168.2 | 20.4 | 22640|\n| Dynamic DINO| 23.4 | 31.8 | 25.0 | 7.7 | 20.8 | **33.4** | **146.6** | **22.5**| **22557**|\n||\n\nAs exhibited in **Tab. T3**, our Dynamic DINO lags Focus DINO slightly by 0.3% points, but outperforms it in inference speed by 2.1 FPS. To sum up, the proposed Dynamic token aggregation strategy significantly reduces the parameters of the baseline model (DINO), while also exhibiting a smaller performance loss compared to other efficient solutions. \n\n## C4: Writing, typos and figures\n\n**Response**: We sincerely appreciate your detailed comments on writing and figure clarity.\nWe will clarify the flow between Sections 3.2 and 3.3 to ensure a more intuitive progression of ideas. Moreover, the typos will be corrected and the font size in Figure 1 will be enlarged for better readability.\n\n## C5: Comparison between ours and other competitors in training iterations and memory imprint\n\n**Response**: Considering the training iterations of these methods are similar, we report the memory info of several setups (see **Tab. T2, T3, and T4** and **the responses to C1**). In comparison to other counterparts, our method shows best overall performance.\n\nTab. T3. Swin-Transformer on the COCO val-set.\n|Model|AP|FLOPs (G) | FPS  | GPU Memory (G) |\n|-|:-:|:-:|:-:|:-:|\n|DINO|51.5|252.3|14.0|27892|\n|Sparse DINO|49.6|137.0|18.0|**24696**|\n|Lite DINO|48.3|151.0|16.8|26371|\n|Focus DINO|**49.9**|156.9|15.3|31933|\n|Dynamic DINO|**49.9**|**149.4**|**18.2**|27764|\n||\n\nTab. T4. MobileNet-V2 on the COCO val-set.\n|Model|AP|FLOPs (G)|FPS|GPU Memory (G) |\n|-|:-:|:-:|:-:|:-:|\n|DINO|25.9|172.6|24.0|15304|\n|Sparse DINO|19.8|67.5|27.2|15810|\n|Lite DINO|18.8|78.4|**36.2**|33950|\n|Focus DINO|20.0|79.8|24.7|14141|\n|Dynamic DINO|**21.7**|**61.5**|36.0|**14082**|\n||\n\n## C6: Integration with BR Loss\n\n**Response**: The BR Loss could consitently facilitate the final performance of both DINO and Dy-DINO, see **Tab. T5**.\n\nTab. T5. Swin-Transformer on the COCO val-set.\n| Model|DINO|DINO+BR|Dy-DINO|Dy-DINO+BR|\n|-|:-:|:-:|:-:|:-:|\n|AP|51.5|52.2|49.9|50.3|\n||", "claims_and_evidence": "The main motivation behind the proposed method is motivated well and discussed in a detailed and objective manner. It also follows the established token merging literature [A, B, C] for computer vision.\n\nMain claims of the work could be listed as follows:\n\n**1.** Dynamic DETR performs at least on-par with the base models and other efficient DETR varieties.\n\n**2.** Dynamic DETR consistently outperforms the efficiency of the models compared to the base models and other efficient DETR varieties.\n\n**3.** Dynamic DETR framework finds _the_ sweet spot between the performance and efficiency.\n\n\nWith respect to these claims, the authors present:\n\n**1.** The authors provide experimental results on COCO2017 _minival-set_. In these results, it is evident that the Dynamic DETR is almost always ~1 AP behind the base model, while performing on-par with Lite DETR [D] and Focus-attention DETR [E] in both of the presented settings.\n\n**2.** The authors provide the FLOPs on COCO2017  _minival-set_. In these results, it is clear that the Dynamic DETR is more efficient with respect to FLOPs compared to base models, while being almost the same as Lite DETR [D] on D-DETR baseline and improving Lite DETR [D]  by 3% on the DINO baseline (albeit the FPS gain is almost the same as Lite DETR [D]).\n\n**3.** Based on the aforementioned two results, Dynamic DETR slightly compromises the performance while providing reasonable efficiency improvements compared to the base models.\n\nFollowing from these claims and presented evidence, it can be seen that Dynamic DETR is not an objectively better method compared to Lite DETR [D] (D-DETR efficiency results and DINO performance results on Table 1). In addition, while I acknowledge that the earlier works (e.g [A]) were also constrained to the FLOPs, the discussions in the work are limited solely with FLOPs, while the broader discussion on efficiency, and finding the sweet spot between efficiency and performance is a multi-faceted discussion, involving both number of training iterations and memory costs.\n\nBased on these discussions, the claims of the work are not strongly supported.\n\n\n\n[A] Bolya, Daniel, et al. \"Token Merging: Your ViT But Faster.\" The Eleventh International Conference on Learning Representations.\n\n[B] Bolya, Daniel, and Judy Hoffman. \"Token merging for fast stable diffusion.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[C] Yuan, Xin, Hongliang Fei, and Jinoo Baek. \"Efficient transformer adaptation with soft token merging.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[D] Li, Feng, et al. \"Lite detr: An interleaved multi-scale encoder for efficient detr.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[E] Zheng, Dehua, et al. \"Less is more: Focus attention for efficient detr.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.", "methods_and_evaluation_criteria": "- Basing the experiments on COCO2017 is valid as it is frankly the most established object detection benchmark. However, the rationale behind the usage of minival-set is not very clear.\n\n- In addition, object detectors (and thus DETRs) are utilized in broad range of domains, often involving long-tailed and challenging cases. Therefore, performing analyses/discussions on other established datasets involving much more classes and much denser annotations, such as LVIS [A] could be helpful for the work. From a token merging point-of-view, these cases could be more challenging given the dense nature of object annotations, though it would also be impressive Dynamic DETR works on it too.", "theoretical_claims": "The nature of the paper is mostly empirical without detailed theoretical claims. However, the design choices made in the paper are mostly motivated and explained by relevant examples and visualizations.", "experimental_designs_or_analyses": "Other than the aforementioned issue on the usage of minival-test, the rationale behind it and a short description of exactly what it corresponds to, the experiments seem sound.", "relation_to_broader_scientific_literature": "Object detection is one of the primary and most well-established computer vision tasks. Detection transformers are the trailblazing response of the detection community to the surge of transformers, though they are known to have various efficiency issues. Thus, the scope of the work is interesting to the broader audiences of detection and efficiency communities.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The idea proposed in the work is an extension of token merging for the detection literature. It contains various novel design choices to make this extension work in an efficient and reasonably performant manner. The writing is also mostly clear, although the flow of thought was a bit counter-intuitive for me between Sections 3.2 and 3.3, since the exact definitions of concepts in 3.2 are defined later.", "other_comments_or_suggestions": "- There is a minor typo on Pg.5: hyperparamt -> hyper parameter\n- Some figures, such as Figure 1 have very small font size. They are otherwise carefully designed and neat.", "questions_for_authors": "- Where do you think your work stands with respect to other efficiency concerns, such as the number of training iterations and memory imprint?\n- Do you think your work is complementary, synergic or orthogonal to other efficient DETR methods, such as [A] from ECCV 2024?\n\n[A] Yavuz, Feyza, et al. \"Bucketed Ranking-Based Losses for Efficient Training of Object Detectors.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t5PyPRJ6T0", "reviewer": "Reviewer_2gMK", "summary": "This paper introduces Dynamic DETR designed to enhance the computational efficiency of DETR-based methods. The study identifies the encoder as the primary computational bottleneck and proposes a dynamic token sparsification strategy to reduce redundant tokens, effectively lowering computational complexity while preserving detection performance.\n\nThe proposed method features two key modules:\n\n1. **Proximal Aggregation**: Merges tokens based on spatial adjacency to retain local structural details.\n2. **Holistic Aggregation**: Ranks token importance and aggregates less important tokens into their most similar important counterparts, ensuring efficient representation.\n\nCompared to the original DETR model, Dynamic DETR reduces computational cost by approximately 40% to 50% in terms of FLOPs, while only causing a minor 0.5% to 1% drop in AP.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your constructive comments. We respond to each comment as follows.\n\n## W1: A breakdown of token importance computation and matching strategy in terms of FLOPs or latency would provide better clarity\n\n**Response**: To quantify this overhead, we provide a detailed analysis of the FLOPs and Latency for these two components in **Tab. T1**. \n\nTab. T1. FLOPs and latency analysis of our method based on DINO, and the results are based on COCO val-set.\n|Model|AP| AP$_{\\mathrm{50}}$|AP$_{\\mathrm{75}}$|FLOPs (G)|FPS|Latency (s)|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|DINO| 50.9 | 68.9 | 55.3|244.5|14.4|0.069|\n|w/o Token Imp. (Random Selecting)|46.6|68.3|50.1|134.7|27.8|0.036|\n|w/o Mat. (Prox Agg.&Holi Selecting)|49.8|69.9|54.2|140.2|24.8|0.040|\n|Dynamic DINO|50.2|69.2|54.7|141.7|23.2|0.043|\n||\n\nWhen we discard the token importance in Eq. (10), namely randomly assigning tokens as important, the performance drops drastically. Meanwhile, turning the matching scheme off leads to a ~1% AP drop but improves FPS by 10.6. Moreover, both components contribute significantly to parameter reduction.\n\n## W2: The paper primarily evaluates the method on COCO2017, lacking experiments on other datasets such as ADE20K and Cityscapes\n\n**Response**: \nTo ensure fair comparisons, we follow the data setups of prior works (*Sparse detr, Lite detr, and Focus detr*), where COCO is primarily used for evaluation and analysis. ADE20K and Cityscapes are designed for segmentation tasks, making them less directly aligned with our study. Meanwhile, following the insightful advices from yours and previous reviewers, we conduct experiments on VOC and LVIS—two widely used object detection benchmarks beyond COCO.Note that all the models are with a ResNet-50 as the backbone and trained for 12 epochs. \n\nTab. T2. Performance of DINO and various efficient solutions on the VOC2007 val-set.\n| Model| mAP|FLOPs (G)|FPS|\n|-|:-:|:-:|:-:|\n| DINO|65.7| 241.6|15.5|\n| Sparse DINO|62.5|141.4|19.6|\n| Lite DINO |38.1|151.0|21.3|\n| Focus DINO|51.4| 153.6|20.2|\n| Dynamic DINO|**63.8**|**135.2**|**21.1**|\n||\n\nTab. T3. Performance of DINO and various efficient solutions on the LVIS-1.0 val-set.\n|Model|AP|AP$_{\\mathrm{50}}$|AP$_{\\mathrm{75}}$|AP$_{\\mathrm{r}}$|AP$_{\\mathrm{c}}$|AP$_{\\mathrm{f}}$|FLOPs (G)|FPS|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|DINO|26.1|34.5|27.5|8.3|24.1|36.1|247.1|19.8|\n|Sparse DINO| 22.9|32.0|24.2|8.4|21.3|30.9|151.7|21.2|\n|Lite DINO| 20.2 | 28.0|21.4|3.0|17.5|30.8|160.0|16.0|\n|Focus DINO| **23.7**| **32.9** | **25.2**|**10.2**|**21.7**|31.9|168.2|20.4|\n|Dynamic DINO| 23.4 | 31.8|25.0 |7.7|20.8|**33.4**|**146.6**|**22.5**| \n||\n\nAs exhibited in **Tab. T2** and **Tab. T3**, the results on VOC and LVIS datasets further showcase the superiority and generality of our dynamic strategy. Moreover, we plan to explore the potential of dynamic token sparsification in pixel-level dense prediction tasks in future.\n\n## W3: Given that Lite DETR achieves comparable performance, the advantages of Dynamic DETR in real-world applications should be further clarified\n\n**Response**: \nFirst of all, we are sorry for the incorrect parameter descrition about DINO and Dynamic DINO in the initial submission, while the corrected verison is as follows. Our anonymous code please refers to [here](https://anonymous.4open.science/r/Dynamic-DETR-4D7F)\n\nTab. T4. Corrected efficiency of DINO, Lite DINO, and Dynamic DINO on the COCO val-set.\n|Model|AP|AP$_{\\mathrm{50}}$|AP$_{\\mathrm{75}}$|FLOPs (G)|FPS|\n|-|:-:|:-:|:-:|:-:|:-:|\n|DINO|50.9|68.9|55.3|244.5|14.4|14.4|\n|Lite DINO|**50.4**|-|54.6|151.0|**23.2**|\n|Dynamic DINO|50.2|69.2|**54.7**|**141.7**|**23.2**|\n||\n\nFor COCO, our method slightly lead Lite DETR. However, on the LVIS and VOC datasets and when using MobileNet as the backbone, our method significantly outperforms Lite DETR in terms of AP, Params, and FPS. Noting that Lite DETR suffers from performance degradation under shorter training schedules, showing the efficiency of our dynamic strategy in reducing training costs.\n\nMoreover, to verify the potential of our dynamic token sparsification in real-time detection tasks, we investigate the performance of several efficient detectors when integrated a lightweight backbone MobileNet-V2, and the results are shown in **Tab. T5**. \n\nTab. T5. Performance of DINO and various efficient solutions with  MobileNet-V2 on the COCO val-set, where the output channels are set to 256 for convergence.\n|Model|AP|AP$_{\\mathrm{50}}$|AP$_{\\mathrm{75}}$|FLOPs (G)|FPS|\n|-|:-:|:-:|:-:|:-:|:-:|\n|DINO|25.9|38.7|27.5|172.6|24.0|\n|Sparse DINO|19.8|33.8|20.7|67.5|27.2|\n|Lite DINO|18.8|29.2|19.9|78.4|**36.2**|\n|Focus DINO|20.0|31.8|20.7|79.8|24.7|\n|Dynamic DINO|**21.7**|**33.1**|**23.1**|**61.5**|36.0|\n||\n\nDynamic DINO significantly boosts the inference performance from 24.0 to 36.0 while the accuracy also outperforms its counterparts by a large margin. This competitive result shows the potential adaptability of our method for practical scenarios.", "claims_and_evidence": "The authors claim that the encoder is the main computational bottleneck in DETR models. The paper provides FLOPs analysis (Figure 1) showing that the encoder contributes significantly to the total computational cost. \n\nIn addition, the authors claim that token importance dynamically changes across different encoder stages and support this claim through statistical analysis of token importance distribution at various levels.", "methods_and_evaluation_criteria": "Dynamic DETR incorporates Proximal Aggregation, Holistic Aggregation, and Center-distance Regularization to enhance token sparsification.  \n\n- Proximal Aggregation employs a window-based approach to determine proximity, ensuring that token merging preserves local structural integrity.  \n- Holistic Aggregation merges less important tokens into their most similar important tokens, effectively reducing redundancy while maintaining essential semantic information.  \n- Center-distance Regularization ensures that token representations remain statistically consistent before and after sparsification, preserving feature integrity and model stability.", "theoretical_claims": "Not applicable, the article does not have proofs for theoretical claims.", "experimental_designs_or_analyses": "The paper mainly conducts experiments based on COCO2017. Although multiple DETR variants are tested, there is a lack of verification of generalization capabilities on other datasets (such as ADE20K, Cityscapes, etc.).", "relation_to_broader_scientific_literature": "This study focuses on designing a dynamic token sparsification strategy to address the encoder computation bottleneck in DETR. Unlike previous lightweight approaches, it introduces a more adaptive and efficient mechanism for token selection.", "essential_references_not_discussed": "The paper provides a comprehensive discussion of previous related works.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. Compared to static sparsification methods, Dynamic DETR adopts a dynamic strategy, which more effectively balances computational cost and detection accuracy.\n2. It is applicable to multiple DETR variants, including Deformable DETR, DINO, and DAB-DETR, demonstrating its broad compatibility.\n\n**Weaknesses**\n\n1. While Dynamic DETR reduces the overall FLOPs, the token importance computation and matching strategy introduce extra operations. A breakdown of this overhead in terms of FLOPs or latency would provide better clarity. \n2. The paper primarily evaluates the method on COCO2017, lacking experiments on other datasets such as ADE20K and Cityscapes.\n3. Compared to Lite DETR (Li et al., 2023), the improvements in AP are relatively small, despite similar FLOPs reduction. Given that Lite DETR achieves comparable performance, the advantages of Dynamic DETR in real-world applications should be further clarified.", "other_comments_or_suggestions": "1. Adding the corresponding symbols from the equations in the paper to Figure 3 would improve clarity and make it easier for readers to follow the method.", "questions_for_authors": "please refer to the Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5gKHKpI24Q", "reviewer": "Reviewer_naQi", "summary": "This paper proposes Dynamic DETR to reduce token redundancy within the encoder for improving the efficiency of DETR-like object detectors. This problem has also been studied by previous work such as Sparse DETR and Focus-DETR. Compared to these existing efforts, this work proposes a finer-grained sparsification approach, using different retention ratios for different feature pyramid levels and designing different sparsification strategies for different levels. Another improvement is the introduction of an additional loss function to distill the feature representation of the original encoder. The proposed method was validated on multiple DETR variants and demonstrated advantages over established methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable suggestions, and next we respond to each comment as follows.\n\n## W1: There is some lack of clarity in the experimental setup.\n - *The paper does not indicate what device the FPS in Table 1 was measured on.*\n - *For which backbone is Figure 1(a) obtained?*\n\n**Response**: Sorry for our unclear descriptions about the experimental setup. \n - The FPS of all models in both the initial submission and the rebuttal phase is measured on a single RTX 3090, with the server load kept consistent.\n - The backbone for Figure 1(a) is ResNet-50.\n\n## W2: There are many writing issues.\n - *The figure captions are too long. It is recommended to move part of the content into the main text for better readability and clarity.*\n - *Figure 1(a) provides limited information. It would be more informative to include a comparison of FLOPs before and after applying the proposed method.*\n - *In Figure 1(b), both the horizontal (FLOPs) and vertical (FPS) axes represent efficiency. To enhance clarity and intuitiveness, it is recommended to replace one of the axes with AP.*\n\n**Response**: We sincerely appreciate your detailed comments on writing and figure clarity. \n - Regarding the figure captions, we will refine them by moving excessive details into the main text to improve readability.\n - For Figure 1(a), we acknowledge its limited informativeness and will incorporate a FLOPs comparison before and after applying the proposed method to intuitively showcase the effectiveness of our approach.\n - In Figure 1(b), we will revise the visualization to present Params vs. AP, providing a more balanced perspective on both efficiency and accuracy.\n\nThanks again for your insightful suggestions, and we  will incorporate these modifications to enhance the overall clarity and readability.\n\n## Q1: To what extent does the proposed method increase training costs? How does this compare to other counterparts?\n\n**Response**: Following your insightful suggestions as well as the recommendations from previous reviewers, we take DINO as the baseline detector and compare the training costs of our method and other competitors. Specific results are as follows.\n\nTab. T1. Training cost comparisons between Ours (Dynamic DINO)  and other efficient solutions with ResNet-50 on the LVIS val-set, where the memory are captured when batch size=2.\n| Model | AP  | AP$_{\\mathrm{50}}$  | AP$_{\\mathrm{75}}$  | AP$_{\\mathrm{r}}$  | AP$_{\\mathrm{c}}$  | AP$_{\\mathrm{f}}$   | FLOPs (G) | FPS  | GPU Memory (G) |\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| DINO| 26.1 | 34.5 | 27.5 | 8.3 | 24.1 | 36.1 | 247.1 | 19.8 | 24244|\n| Sparse DINO| 22.9 | 32.0 | 24.2 | 8.4 | 21.3 | 30.9 | 151.7 | 21.2 | 22680|\n| Lite DINO| 20.2 | 28.0 | 21.4 | 3.0 | 17.5 | 30.8 | 160.0 | 16.0 | 42348|\n| Focus DINO| **23.7** | **32.9** | **25.2** | **10.2** | **21.7** | 31.9 | 168.2 | 20.4 | 22640|\n| Dynamic DINO| 23.4 | 31.8 | 25.0 | 7.7 | 20.8 | **33.4** | **146.6** | **22.5**| **22557**|\n||\n\nTab. T2. Training cost comparisons between Ours (Dynamic DINO) and other efficient solutions with  Swin-Transformer on the COCO val-set, where the memory are captured when batch size=2.\n| Model | AP  | AP$_{\\mathrm{50}}$  | AP$_{\\mathrm{75}}$|FLOPs (G) | FPS  | GPU Memory (G) |\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|DINO|51.5|70.2|56.5|252.3|14.0|27892|\n|Sparse DINO|49.6|68.4|54.1|137.0|18.0|**24696**|\n|Lite DINO|48.3|66.1|52.8|151.0|16.8|26371|\n|Focus DINO|**49.9**|68.2|54.3|156.9|15.3|31933|\n|Dynamic DINO|**49.9**|68.8|54.3|**149.4**|**18.2**|27764|\n||\n\nTab. T3. Training cost comparisons between Ours (Dynamic DINO)  and other efficient solutions with  MobileNet-V2 on the COCO val-set, where the memory are captured when batch size=2.\n|Model|AP|AP$_{\\mathrm{50}}$|AP$_{\\mathrm{75}}$|FLOPs (G)|FPS|GPU Memory (G) |\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|DINO|25.9|38.7|27.5|172.6|24.0|15304|\n|Sparse DINO|19.8|33.8|20.7|67.5|27.2|15810|\n|Lite DINO|18.8|29.2|19.9|78.4|**36.2**|33950|\n|Focus DINO|20.0|31.8|20.7|79.8|24.7|14141|\n|Dynamic DINO|**21.7**|**33.1**|**23.1**|**61.5**|36.0|**14082**|\n||\n\nFrom the above results, it can be observed that the proposed Dynamic Token Sparsification strategy generally maintains a similar or even lower training cost compared to the baseline detector. In comparison to other counterparts, our method demonstrates clear advantages in both accuracy and speed.", "claims_and_evidence": "The claim that “the distribution of important tokens across different hierarchies” follows a certain pattern has been validated in only one sample (Figure 2). Is this phenomenon universal? It is better to define some quantitative indicator, which would increase the credibility of this claim.", "methods_and_evaluation_criteria": "Yes, the evaluation makes sense.", "theoretical_claims": "No, it's not a theoretical paper.", "experimental_designs_or_analyses": "The experimental designs are generally sound.", "relation_to_broader_scientific_literature": "No, not related to the broader scientific literature", "essential_references_not_discussed": "No, the existing related works have been well discussed", "other_strengths_and_weaknesses": "Strengths:\n1. The motivation makes sense. The proposed method uncovers the need to use different sparsification thresholds for different levels, which was neglected by previous works.\n2. The experiments are adequate. The proposed method was validated on multiple DETR variants such as D-DETR, DINO and DAB-DETR.\n\n\nWeaknesses: \n1. There is some lack of clarity in the experimental setup. \n- The paper does not indicate what device the FPS in Table 1 was measured on. \n- For which backbone is Figure 1(a) obtained?\n2. There are many writing issues: \n- The figure captions are too long. It is recommended to move part of the content into the main text for better readability and clarity. \n- Figure 1(a) provides limited information. It would be more informative to include a comparison of FLOPs before and after applying the proposed method.\n- In Figure 1(b), both the horizontal (FLOPs) and vertical (FPS) axes represent efficiency. To enhance clarity and intuitiveness, it is recommended to replace one of the axes with AP.", "other_comments_or_suggestions": "see the weaknesses part above", "questions_for_authors": "To what extent does the proposed method increase training costs? How does this compare to other counterparts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses the important problem of efficiency in Detection Transformers. While the initial reviews raised valid concerns regarding the scope of evaluation and the demonstrated advantage over existing methods, the authors provided an exceptionally thorough rebuttal with significant new experimental results. These results convincingly demonstrate the method's effectiveness and generalizability across multiple datasets (COCO, VOC, LVIS) and backbones (ResNet, Swin-T, MobileNet-V2), often showing favourable trade-offs compared to strong baselines like Lite DETR. The additional analyses on memory, training time, and latency further strengthen the paper. The reviewers were largely convinced by the rebuttal, leading to score increases. The proposed dynamic, multi-level token aggregation strategy combined with representation regularization offers a solid contribution to the field of efficient object detection. Assuming the authors incorporate the promised revisions regarding clarity and integrate the new results into the final paper, it is recommended for acceptance.
Decision: Accept (poster)