Enhancing Vision Transformers for Object Detection via Context-Aware Token Selection and Packing
Authors: Tianyi Zhang
Baoxin Li
Jae-sun Seo
Yu Cao
Venue: ICLR 2026 Poster (2026)
Keywords: vision transformer
object detection
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In recent years, the long-range attention mechanism of vision transformers has driven significant performance breakthroughs across various computer vision tasks. However, these advancements come at the cost of inefficiency and substantial computational expense, especially when dealing with sparse data. While sparse attention mechanisms have been introduced to mitigate these issues by pruning tokens involved in attention, they often lack context-awareness and intelligence, frequently limiting the number of selected tokens uniformly across different inputs. To address these challenges, we propose a novel algorithm: Select and Pack Attention (SPA). SPA dynamically selects informative tokens using a low-cost gating layer and packs these selected tokens into new batches, allowing for a variable number of tokens to be used in GPU batch training and inference. Through extensive experiments on diverse datasets and multiple computer vision tasks, our method demonstrates superior performance and efficiency, including a 0.5-2.7 AP improvement in object detection and a 10.9%-24.9% reduction in computation.

Published as a conference paper at ICLR 2026
ENHANCING VISION TRANSFORMERS FOR OBJECT DE- TECTION VIA CONTEXT- TOKEN SELECTION
AND PACKING
Tianyi Zhang
Baoxin Li
Department of Electrical and Computer Engineering University of Minnesota Minncapolis, MN 55455, USA (zhan91 67 @umn edu
Department of Computer Science University of Illinois Chicago Chicago, IL 60607, USA (baoxin] 1 @uicc.du
Jac-sun Seo Cornell Tech New York, NY 10044, USA [js3528 @ecorelll.e...o
Yu Cao
Department of Electrical and Computer Engineering University of Minnesota Minneapolis, MN 55455, USA {yucao) @umn.edu
ABSTRACT
In recent years, the long-range attention mechanism of vision transformers has driven significant performance breakthroughs across various computer vision tasks. However, these advancements come at the cost of inefficiency and substantial computational expense, especially when dealing with sparse data. While sparse attention mechanisms have been introduced to mitigate these issues by pruning tokens involved in attention, they often lack context-awareness and intelligence, frequently limiting the number of selected tokens uniformly across different inputs. To address these challenges, we propose a novel algorithm: Select and Pack Attention (SPA). SPA dynamically selects informative tokens using low-cost gating layer and packs these selected tokens into new batches, allowing for a variable number tokens be used in GPU batch training and inference. Through extensive experiments on diverse datasets and multiple computer vision tasks, our method demonstrates superior performance and efficiency, including 0.5-2.7 AP improvement in object detection and 10.9%-24.99% reduction in computation.
INTRODUCTION
Recent advancements in computer vision tasks such image classification, segmentation, and object detection have seen Vision Transformers (ViTs) surpass traditional convolutional approaches (Dosovitskiy al. 2220) Liu et al. [202]] Xia al.) 2022) et al.) (202)) due to their powerful self- attention mechanisms. ViTs are particularly effective at capturing long-range dependencies, enabling the learning of global features that are crucial for complex visual understanding. However, this strength comes with significant drawback: the computational overhead increases quadratically with the number of tokens (Liu et al. 2021 Hua et al. 2022), leading to excessive and often unnecessary computations among irrelevant tokens. This not only raises the computational burden but also risks degrading performance by incorporating extraneous, often redundant, information in typical computer vision tasks. As illustrated in Fig. [ the self-attention mechanism in ViTs inadvertently processes large amount superfluous data (1.c. background tokens), exacerbating computational inefficiency and potentially degrading task performance by introducing irrelevant information into the model's

---

Published as a conference paper at ICLR 2026
DynamicVIT
SparseViT
SPA (ours)
Unsupenwszd Solection
Unoupervised Selection
Snealer boton, restction and posking the nesimum
Supervited Selechion and Full in batch containers Train Batch
Train Batch
sbawdored
Train Batch
Acconately /eep unformttvciatso
Informative token for first image
Informative token for the second image
Baciground token
Attention Mask
Figure 1: Previous sparse attention methods either reduce computation only during the inference stage or require padding the length of selected tokens to the maximum within batch, which inevitably introduces background tokens. This leads to reduced efficiency and worse accuracy.
learning process. This issue is particularly severe when dealing with sparse data, such as in small object detection, where most pixels are not informative.
Numerous approaches have heen proposed to address this issue by performing self-attention only on the most informative tokens. However, these methods still encounter significant challenges both efficiency and performance.
Efficiency: The constraints of GPU batch training. where images within a batch often contain non-uniform numbers of informative tokens, pose challenges to parallelizing computation effectively. Some methods, such as SparseViT (Chen et al. 2023). address this by padding all effective tokens to match the maximum number in the batch, leading to inefficiencies, as illustrated in Fig. Other approaches, like Dynamic VIT (Rao et al. 2021) and EVIT (Liang al. 2022 reduce computation only during inference by discarding fixed number of tokens. However, these methods still attend to all tokens during training, employing attention mask focus informative tokens, which, along with the mask prediction module, can result in training costs that exceed those of a standard ViT. The Deformable Attention Transformer (DAT) (Xia et al. 2022), inspired by Deformable Convolutional Networks (DCN) (Dai et al. 2017). merely reduces the receptive field of query tokens while still computing all tokens, yielding minimal improvements efficiency.
Performance: Existing methods demonstrate effectiveness primarily in simpler tasks like image classification, where some degree of information loss is tolerable. However, their per- formance degrades in more complex tasks, such as object detection or instance segmentation, which demand richer semantic information. For example, DynamicSwin (Rao et al. 2003). Swin-based DynamicViT, struggles these scenarios due to inaccurate token selection, leading to significant information loss.
To address these challenges, we propose novel Select and Pack Attention (SPA) mechanism that dynamically selects varying numbers of informative tokens from input batches, supervised by selection labels, and packs them into new batches for parallelized training. Specifically, we introduce a linear gating layer to generate scores for token selection, supervised by multi-scale selection label derived from object labels (e.g., bounding boxes, instance segmentation labels). After selection, the chosen tokens placed into uniform-sized package containers to form new batches, illustrated in Fig. 1 For attention computation within each container, tokens attend only to those from the same original image, ignoring tokens from other images by using attention masks. Additionally, SPA can be effectively integrated with the window-based attention proposed by Swin Transformer (Liu et al. 2021), benefiting from the window shifting operation that captures information across windows. To prevent information loss across package containers, we shift the feature maps every two transformer blocks, ensuring that token pairs placed into containers vary, allowing the attention computation to encompass all tokens. Based on SPA, we propose backbone network, Select and Pack Transformer (SPT), featuring a hierarchical architecture to generate image representations at various scales for downstream computer vision tasks. To avoid mis-selection at the early stage which may cause serious information loss (Xia et al. 2022).. we leverage SPA from the third stage with the adapted image features. Ultimately, SPA addresses the efficiency issue by selecting only informative tokens and packing them into new batches, enabling efficient parallelized computation for both training and inference. Moreover, by leveraging selection label supervision, SPA improves
2

---

Published as a conference paper at ICLR 2026
performance in complex computer vision tasks, such as object detection. Comprehensive experiments on three kell- datasets demonstrate the efficacy of SPA across multiple computer vision tasks.
To summarize, our main contributions are follows:
We propose a novel sparse attention mechanism, Select and Pack Attention (SPA), to enhance both the efficiency and performance of Vision Transformers. For efficiency, SPA dynamically selects informative tokens from images in a batch using a linear gating layer and packs them together to enable efficient GPU batch training and inference. For performance, we introduce a multi-scale selection label to explicitly supervise token selection, thereby outperforming existing methods even in complex computer vision tasks.
By effectively integrating our SPA mechanism with Swin blocks, which use a window shifting trick capture information across packages, we propose a backbone network with a hierarchical structure called the Select and Pack Transformer (SPT). SPT can generate features at various scales, making it suitable for many computer vision tasks.
Through extensive experiments diverse datasets, with lower computation cost, SPT outperforms state-of-the-ane methods with a 0.5-2.7 AP improvement in object detection, a 1.3 AP gain in instance segmentation, and a 0.48 mAP increase in multi-label classification.
2 RELATED WORK
2.1
TRANSFORMER IN COMPUTER VISION
Given the remarkable success of transformers in natural language processing (NLP), this architectural
paradigm is progressively permeating diverse computer vision tasks (Vaswani et al. 2017
Bao et al.
2021
Touvron et al. 2021 He et al. 2022; Zhang et al. 2024 Konstantinidis et al. 2023
Yang et al.
2022
Kang et al. 2022 Ni et al. 2024a b Zhou et al. 2024 Fan et al. 2024 Fan & Tao
2024). For
instance, Vision Transformer (ViT) divides input images into 16 x 16 patches, which are subsequently
treated as tokens for the application of the attention mechanism (Dosovitskiy al.] (020). In
image segmentation, SAM (Kirillov et al. 2023) introduces a prompt-based algorithm, setting new benchmarks across state-of-the-an methods. For object detection, DETR conceptualizes it as direct set prediction problem and designs transformer-basee network (Carion et al. 2020). DINO advances self-supervised learning to propose a novel network rooted in the ViT architecture (Caron al.. 202I). For super-resolution, transformers demonstrate exceptional capability in capturing long-range dependencies for improved visual representations (Liang al., 2021
Zhang al. 2023).
2.2 EFFICIENT TRANSFORMERS
Despite their advantages in global feature extraction via self-attention across all tokens, Vision Transformers (ViTs) are hindered by significant computational overhead because the computation of attention weights scales quadratically with the number of tokens. To address this challenge, efficient
transformer variants (He et al. 2022 Child et al. 2019 Lingle 2023) have explored multiple forms of sparsity reduce attention cost. Swin Transtormer (Liu a 2021) introduces window-hased and shifted window-based self-attention mechanisms, significantly reducing computational demands within localized windows. Dynamic (Rao al.) 2202] l) proposes dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input, Sparse ViT (Chen al., 2023) optimizes computation by selecting tokens based on the la norm of window activations, prioritizing features with higher scores. DAT (Xia et al. (022) employs an offset network to refine the query token's 's receptive field, further enhancing computational efficiency. Fixed sparse patterns (Qiu et al., 2020: Zaheer et al. 2020) impose pre-defined attention masks to improve scalability for long sequences. Hybrid local local-glohal attention mechanisms (Chu et al.) [022 Dong et al. (022)) restrict most computation to local windows while adding sparse global mixing tokens. Spatial factorization approaches (Ho et al. (2019) decompose 2D attention into row/column operations to reduce complexity. Learned sparsity frameworks (Wei al. 2023) predict instance-dependent sparse attention patterns directly from features.
While effective, these approaches either (i) rely on heuristic diructuredrrver sparsity that does not adapt well to downstream detection objectives, or (ii) lack dense supervision to ensure retention of small-object regions. In contrast, SPA learns token importance under explicit multi-scale supervision
3

---

Published as a conference paper at ICLR 2026
from object-level labels. Moreover, packaging mechanism addresses a practical limitation of dynamic sparse attention (irregular sequence lengths) by restoring batchwise parallelism.
K2C
4:4C
32
BC
# x
Stage
Stage
Stage 3
Stage
Patch Embedding
Feature Embedding
Feature Embedding
Feature Embedding
Swin
Block
Swin Block
SPA
SPA
SPA
SPA
K.Ng
N.
LN
WSA/ 5-WSA
MLP
Swin
Multi-Scole Supervised Selection
Figure 2: Overall architecture of Select and Packing Transformer (SPT). The hierarchical structure can generate features with various scales common backbone networks. The SPA blocks in the last two stages can improve both efficiency and accuracy by disregarding uninformative tokens.
METHODOLOGY
3.1 OVERALL ARCHITECTURE OF SPT
As illustrated in Fig. SPT features a hierarchical structure composed of four stages. Each stage generates image representations of varying sizes, resulting in total of four different scales.
Specifically, suppose we consider a small x patch as single token, the input image a E "xwx3 (H and W are the imput image height and width), progressively embedded into representations T1 E R x xc r2 F x2C T3 R W. x4C r4 R xsC (C the embedding dimension of the first patch embedding layer) stage by stage. Each stage is structured around an embedding block for feature map downsampling, followed by N, transformer blocks tasked with feature learning (N; signifies the block count in the ith stage). Similar to Swin (Liu et al.) [021]] 2022), the embedding block of the first stage f'a employs convolution layer while the subsequent embedding block consists of a patch merging layer that concatenates features as groups of 2 x 2 patches and a linear layer for feature projection. For the transformer blocks, the first two stages fe,, for utilize standard Swin Transformer blocks, whereas the latter two stages feg, fe, incorporate Select and Pack Attention (SPA) block. This design decision is informed by observations from DAT (Xia et al. 2022). which noted that early-stage transformer block replacement diminishes accuracy due to the model's 's inability to efficiently distinguish positive tokens based on shallow features. SPA blocks in the second and third stages not only generate outputs for subsequent layers but also transfer the score map to the next stage for the multi-scale supervision 8o E R W and 81 E R x computing select loss. With the selection map 82 $ W x1 generated the last stage, there total of three different scales. The complete process is as follows:
T1 = fe, (x), (r)( = fe, (T2)
(1)
F3, So (r2, 80),
T4, 82 fo. (r3, 81),
where Tı, T2, T3, and r44 are output representations of four stages. fe. fø.. feg, and fe, denote the four stage models. And sp, $1 and $2 denote the predicted score map for selection from the last three stages, separately. fo, is the gating layer to generate scores for the output of stage 2.
In addition, given the higher efficiency of SPA blocks compared to Swin (Liu al.) 2021 included additional blocks at the third stage to enhance performance while maintaining a lower computation cost.
4

---

Published as a conference paper at ICLR 2026
3.2 SELECT AND PACK ATTENTION (SPA)
Inspired by the gated networks in Mixture of Experts (MoE) (Petersen et al. 2022] Huang et al.
2020, (hazzeer al., 2017 Aoki al. 2022 Chen al. 2022) and heterogeneous federated learning (Lin et al. 2021 Ye et al. 2023). which adeptly guide models in selecting appropriate computational paths and enhancing task-specific generalization, we design a Select and Pack (SnP) block. This block utilizes a linear gating layer to select informative tokens (Detailed in Section A and pack them into fixed-size package containers, generating new batches for GPU training or inference. While positive tokens undergo multi-head self-attention (MSA), negative tokens are directly passed to the feedforward network, as illustrated in Fig. Ba SPA not only enhances the efficiency of the self-attention mechanism but also improves performance by focusing exclusively informative tokens, effectively mitigating the potential for misleading context introduced by background tokens.
Posttive Tokens
Up-icale Seleec Score
MSA
Packed Attention
Gard Caanon
5mP
Select Loss
Negative Tokens
(a) SPA block.
(b) SnP block.
Figure 3: (a) SPA computes attention (i.e., MSA) only for informative tokens. (b) SnP block selects informative tokens under multi-scale supervision and packs selected tokens for batch training and inference. The packed tokens attend to only tokens from the same image.
Multi-Scale Supervised Selection. Although token selection can be implicitly guided by the final objective, our experiments reveal that the gating layer tends to assign large values to all tokens, leading the selection of too many tokens and reduced efficiency. To address this, we introduce selection label based on object labels, which directly indicate areas of interest, such as instance segmentation masks or object detection bounding boxes. For segmentation, a binary mask assigns a value of to all object pixels and O) otherwise. For object detection, an aggregated binary mask is formed by stacking all bounding boxes. However, a single-scale label overly restricts token selection, causing significant information loss and poor performance. To mitigate this, we reduce the Gumbel-Softmat function's ss threshold and integrate multi-scale select labels. In Fig. 33b each SPA block in SPT not only uses the selection scale matching the representation but also incorporates scores from up-scaled features, adjusted via max-pooling to match the correct feature size. This approach selects the maximum scores from two scales to include more informative tokens, thereby enhancing performance.
Specifically, given flattened input batch r E BxNxC (B, N and C' are the batch size, the length of each image representation and the number of channels, separately). the gate fe, assigns scores E R Bx each token. Then, we somentt-wise multiply the normalized scores by a sigmoid layer with the input representations to obtain the gated representations rg BBXNXC After that, we leverage the Gumbel-Softmax function (Jang et al. (0066) to separate positive tokens (i.e., informative tokens), Tp N,C (Np is the number of positive tokens from all images in the batch) as follows:
Max(fo, Sup):
(4)
rg Sigmoid(s) (8) r;
(5)
= Gumbel-Softtax(s @
(6)
where Sup. 8, Tg. Tp denotes the input representation, scores for up-scale features, scores for this scale, gated representation, and output positive tokens, separately. And ( wiem multiplication with boardcasting. fa, is the linear gating layer.
Token Packing. After the dynamic selection for each input image, the lengths of selected tokens vary. To avoid padding all tokens to the maximum length, which would introduce significant computational overhead, we pack the selected tokens into new batches. Inspired by (Dehghani et al. 2024), we set series of package containers with fixed length L and fill them with the selected tokens. After packing all selected tokens, if the total number of tokens is not a multiple of the packing length, we only pad the last package. This approach is significantly more efficient than padding the selected
5

---

Published as a conference paper at ICLR 2026
tokens for all images in the batch. Consequently, we obtain packed tokens, p BB'xX (B' is the new batch size of packed tokens), and the number of tokens (B' x L) much smaller than the original imput (B x N), especially for sparse data. And the attention computation depends on L, similar to the window size M of Swin, And we set I. to be M Specifically, for input representation batch E Bx the complexity of regular MSA, window-based self-attention (W-MSA), and SPA are as follows:
((MSA)= = B(4NC² + C2
(7)
(W-MMSA) = B(4NC + NC),
(8)
((SPA) BINC COICC+NCC N B'LLCC + 2L2C),
(9)
Compared to MSA, W-MSA is more efficient since the complexity is linear to the original token length N. However, our SPA is not only linear to N, the new batch size B' is also much smaller than B, resulting in higher efficiency. For the self-attention of the packed tokens, we employ an attention mask to ensure that all tokens attend only to tokens from the same image, as illustrated in Fig. 3b
Table 1:
SPT achieves superior object detection performance on COC02017 across three config- urations while requiring lower overall computation. FLOPs are reported for the training stage. Additionally, we include the performance gain over the best sparse attention baseline. *For all base models, we adopted Cascade Mask RCNN framework for fair comparison.
Methods
Attention
AP
APSo
OD Performance
AP3s
AP;
APw
AP,,
backhone
FL.OPs (G)
overall
PParams (M)
(imagek)
FPS
Swin-
Linetal 2021
Dense
46
68.1
50.3
31.2
49.2
60.1
96
267
48
50
Sparse
Spurse
12.4
DAT-T
Spanse
67.6
47.5
101
Symamk55a3-7
KJO ml. X02
Spanse
44.3
65.9
48.5
30.2
47.1
58.5
101
272
48
46
SPT-T (ours)
Spanse
47.1 (+2.7)
68.9
51.6
31.7
50.5
60.9
90 (-10.9%)
261
(-40%)
55
$4
Swin-S en
2021
Donse
48.5
70.2
53.5
33.4
52.1
63.3
188
359
69
32
Xineraa
Sponse
47.1
69.9
50.1
208
379
69
31
DynaniSSums5
K40
Spanse
47.2
68.1
52.7
32.6
49.8
363
SPT-8 (oums)
Spanse
(+.3
$5.2
33.9
54.3
64.1
171 (-10.9%)
(-5.8%)
76
33
Swin-B B ef
1021
Donse
31.9
70.5
56.4
35
55.2
67.4
332
982
145
11
DynimieSwms-
era 2023
Spanse
50.5
69.7
58.9
34.7
53.2
66.1
341
991
145
11
SPT-B (ours)
Spans
53.2
71.3
58.9
36
57.6
67.9
294 (-11.%)
944 (-3.9%)
153
12
3.3 Loss FUNCTION
The loss function of SPT comprises the loss for the target task and the selection loss. For the selection loss, we adopt binary cross cros--entrpyy and sum over all SPA blocks as follows,
Exelect
Σ (y s + (1 3) log(1 ([ss)))))))))))))))) +))
(10)
where 8 is the normalized score map through Sigmoid layer, and y denotes the ground truth label. The overall loss function of SPT is LSPT Lrask aLSselect where is hyperparameter.
4 EXPERIMENTAL RESULTS
4.1 DATA AND EXPERIMENTAL SETUP
To evaluate the effectiveness of our SPT Transformer in complex computer vision tasks, we conducted extensive experiments on object detection using the widely adopted COC02017 and BDD100K datasets. The COC02017 dataset includes objects from 80 categories, while BDDI00K comprises 10 categories. Beyond standard object detection, we further tested our method on the challenging task of early object detection, where objects are predominantly small, making both object detection and token selection significantly more difffiuut. This scenario underscores the importance of accurate token selection, as errors can result in the loss of entire objects, thereby highlighting the robustness of our approach. To simulate this scenario, we adopted the method proposed in (Zhang et al.) (024).. Specifically, we computed the abject-to-pixel ratios across entire images and selected samples from the BDD100K dataset with object ratios below 25%, creating the BDD-S dataset.
Additionally, to further showcase the robustness of our SPT Transformer, we extend our experiments to additional computer vision tasks, including instance segmentation COCO2017 and multi-label

---

Published as a conference paper at ICLR 2026
Table 2: SPT-based Mask RCNN achieves better object detection performance with less total computation on BDD100K for all three configurations. FLOPS is for training stage. We adopted the same image resolution COC02017.
Methods
Attention
APto
OD Performance
AP::
AF;
AFtr
APr.
FLOPs
#Porams (M)
(Imagees)
FPS
Swin-T Liu et 2021
Dense
22.4
34.6
21.6
7.9
20.1
46.8
267
48
50
Dynamiswmm- Kib of 2025
Sparse
22.0
33.1
23.7
9.2
19.7
44.7
101
272
48
46
SPT-T (ours)
Spunse
22,6 (+0,6)
33.1
24.6
8.8
18.5
47.7
84 (-16.8%)
255 (4.33%
55
58
Swin-S (Lin
2021
Dense
22.6
34,9
24.9
8.1
20.3
47.1
188
359
69
32
DynimicSames (ours)
Spane
45.4
32
Spane
22,9
25.2
(-19.3%)
(-10.2%)
Swin B al. 202
Dense
22.7
35.1
25.1
8.2
20.5
47.6
508
107
18
Dynamiewmm--
RID 2023
Sparse
22.4
33.6
24.6
8.5
20.2
46.3
341
517
107
18
SPT-B (ours)
Spunse
23.1 (+0.7)
34,3
25.5
8.2
20.6
48.5
256
(-24.9%)
432 (-16.4%)
115
20
Table 3: For early object detection on the more challenging BDD-S dataset, SPT also outperforms baseline models with 20.8-22.4% backbone computation cost reduction when compared to baselines.
Methods
Attention
APto
OD Performance
APar
backhone
FLOPs (G)
overall
#Parnus (M)
(Imagels
FPS
2021
Dense
2.7
DymariSSan- Roo
2025
Sparso
8.4
1.0
2.6
12.8
101
48
46
SPT-T (oars)
Sparse
5.6 (411.)
9.0
6.4
1.3
2.7
15.4
80 (-20.8%)
251 (-7.7%)
55
62
Swin [Lu et 2021
Dense
5.4
9.0
6.0
1.7
2.9
14.1
188
359
69
32
Dynameswar's
2023
Sparse
5.2
8.2
6.0
0.9
2.3
14.3
192
363
69
32
SPT-8 (ouns)
Sparse
(40.5)
29
(-22.4%)
(-11.8%
35
classification on PASCAL VOC 2012 dataset. For multi-label classification, we also select images with object pixel ratios smaller than 25% to show the effectiveness of SPT in challenging scenarios.
To evaluate object detection and instance segmentation, we utilize the Mask RCNN framework, and replace the backbone network with our SPT or other baselines. We adopt the default training settings, such as 36 max training epochs, batch size of In addition, we set the threshold of Humbel-Softmax to 0.01, and set the select loss weight a to 0.01. We set the window size of Swin block to 7, and the container length of SPA to 49. Experiments were performed on two Linux servers, each outfitted with dual NVIDIA L40S GPUs.
4.2 SPT FOR OBJECT DETECTION
In Table we present a comparison of SPT with other baselines for the tiny, small, and base configurations (i.e., SPT-T, SPT-S, and SPT-B) on the COC02017 dataset. For baselines, we focused on hierarchical backbones, vanilla ViT-based methods, such EViT (Liang et al. 2022). produce single-scale outputs and perform significantly worse on object detection tasks compared to hierarchical architectures like Swin Transformer (Liu al., 2021 (022), which provide feature pyramids. In addition to comparisons with sparse attention approaches, we included Swin results a dense attention reference. Under the same settings, including dense attention (i.c.. Swin). Compared to the best- west-performing sparse attention baseline, SPT delivers AP improvements ranging from 2.1 to 2.7. Similarly, Table [ we report the results on the BDD 100K dataset, where SPT surpasses all baselines and achieves state-of-the-a object detection performance, with an AP improvement of 0.6 0.7 over DynamicSwin (Rao et al., 2023). SPT-B attains the highest AP of 23.1.
approach outperforms all baselines
Early Object Detection Performance. For the performance of early object detection on the BDD-S dataset, we observed that the base model underperforms compared to the tiny and small versions in this scenario. Consequently, we included only the results of the tiny and small models in Table 3 Notably, SPT-T and SPT-S achieved performance improvements of 19.1 % and 9.6%, respectively. The superior performance in early OD, which involves much smaller objects, highlights the effectiveness of SPA in accurately selecting informative tokens, guided by multi-scale selection labels.
Efficiency Analysis. The GFLOPs reported in Table T Table 2, and Table 3 are computed over backbone, FPN and detection head with RGB input image the resolution of 1280 800 for training stage. For a clearer comparison, we evaluate the throughput (i.e., FPS) only over the backbone network machine with an NVIDIA L40S GPU, as including other components would result in
7

---

Published as a conference paper at ICLR 2026
Table 4: SPT also performs better for instance segmentation on COC02017.
Methods
AP
AP50
AP75
AP3
AP M
APL
DynamicSwin-T Rao ct al.
2023
38.8
61.8
41.7
20.1
40.8
57.7
SPT-T (ours)
39.3
62.4
42.1
20.4
41.5
58.1
DynamicSwin-S Rao et al. 2023
39.6
63.1
42.8
21.0
42.4
59.2
SPT-S (ours)
40.9
64.6
44.0
21.8
43.9
60.1
values that are too small. From these tables, we obtain the following foundings: 1) SPT consistently outperforms all baselines in terms of efficiency, achieving reduction in backbone computation cost by 10.9% to 24.9%. 2) The efficiency improvement is more pronounced when datasets contain smaller objects. For instance, the computation reduction of SPT-T on BDD-S 20.8%, compared to 16.8% the full BDDIOOK dataset. This aligns with expectations, object-containing tokens for computation. 3) To achieve better performance, SPA is applied only at the last two stages, with four additional blocks introduced at the third stage. However, due to downsampling at the third stage, the number of tokens is reduced by factor of 16x, leaving the early stages dominate computation costs. Despite SPA selecting only 22% of tokens for computation (discarding 78%), the overall efficiency improvement is limited to 16.8%.
our approach selects only
4.3 SPT FOR OTHER COMPUTER VISION TASKS
addition to the object detection task, we also evaluated our SPT on other tasks, including instance segmentation and multi-labe classification.
Instance Segmentation. As shown in Table both tiny and small versions of SPT outperform baseline models (Rao et al. 2023). SPT-S achieves more substantial improvement, increasing AP from 39.6 to 40.9 on COC02017.
Table The SPA blocks reduce the computation with a low number selected tokens (i.e., select ratio) and achieve better performance in multi-label classification on PASCAL VOC 2012.
Dataset
Methods
Mean Select Ratio %
mAP
DynamieSwin-T |Rao et al. 2023) SPT-T (ours)
44.12
PASCAL VOC
29.6
44.60
Performance on VOC
Computation cost on VOC
1.0
MSA
GT-SPA
1.0
20
MGA
GT-SPA
0.84
0.8
(a.u.
la.u
0.8
0.44
0.4
0.2
0.0
0.0
Swin
VIT
Swin
VIT
Figure 4: Under ground truth supervision, attending to only informative tokens can achieve better performance and efficiency.
Multi-Label Classification. For multi-label classification PASCAL VOC 2012, SPA improves per- formance to 44.6. outperforming DynamicSwin (Rao et al. 2023). as shown in Table Additionally, present the mean select ratios for SPA blocks. Overall, SPT reduces GFLOPs by 10.2%.
8

---

Published as a conference paper at ICLR 2026
Table 6: Starting replace the Swin blocks from the third stage performs the best, as early-stage selection leads to information loss. Seelect significantly reduces the number of selected tokens for computation, with lower select ratios. AP and FLOPs this table are evaluated with select.
Blocks
Select Ratios stages (w/o seleec)
Select Ratios stages (w/ Loetect)
Dataset
AP
AP
APTs
1,
Zna
3,a
4a
1at
2nd
Srd
4AA
2nd
Syd
4mh
X
x
x
X
X
X
36.5
X
X
25.02
21.9
32.7
24.2
BDD100K
X
85.02
52.41
23.21
25.01
22.6
33.1
24.6
X
X
99.24
82.15
71.45
20.18
22.56
25.0
20.5
31.3
22.3
83.49
99.12
88.64
79.86
13:42
20.78
22.36
25.0
29.4
20.6
Input
w/o & select
w/ L select
Input
w/o & select
w/ L select
Figure 5: We overlay the summation of the selection masks generated by all SPA blocks on the original image. Warm color denotes high frequency of selection while cold color means be pruned before the attention computation. With multi-scale supervision, the selection process is more accurate.
4.4 ABLATION STUDY
The Effect of Token Selection for Attention. To illustrate the effectiveness of informative token selection, we designed experiments where all informative tokens were selected based on ground truth selection. As illustrated in Fig. 4 for both plain ViT and window-based attention mechanisms (i.e., Swin), selecting tokens to disregard background information improves both accuracy and efficiency on PASCAL VOC 2012 dataset.
Specific Design for Selection. Even though we know that token selection works, a critical challenge how to correctly select these informative tokens without ground truth lahels. As discussed earlier, previous methods (e.g., Sparse commonly adopt uniform token selection, applying a fixed ratio for all images in each batch. However, the results in Table 7 demonstrate that our SPA with dynamic selection performs better. Additionally, Fig. provides visual comparisons to illustrate the effectiveness of our proposed multi-scale select label.
Table For uniform sparse attention, we adopt the top-50 technique as Sparse (Row 1). SPA block performs better. Leelect further improves the performance.
Dataset
SPA
Lxelect
Mean Select Ratio( %)
mAP
X
X
50
44.42
PASCAL VOC
59.77
44.49
29.60
44.60
Number of SPA blocks. Table 6 explore the optimal number of SPA blocks in SPT. The results match with the findings in Xia et al. (2022). Starting from the third stage yields the best performance. Early-stage selection leads to information loss, resulting in worse performance.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we analyze the current issues with sparse attention mechanisms and propose novel Select and Pack (SPA) mechanism to address these challenges for both efficiency and performance. SPA focuses attention computations solely informative tokens using supervised gating block in Vision Transformers and packs the selected tokens for parallelized GPU batch training and inference. Integrated into the Swin Transformer's s hierarchical architecture, SPA forms the efficient Select and Pack Transformer (SPT), which works as image backbone network for various computer vision tasks and generates multi-scale representations. Extensive experiments across three datasets and range of vision tasks validate the effectiveness of SPT.
Despite these advantages, SPA relies spatial supervision (e.g., bounding boxes masks) guide token selection; while this is naturally available in detection and instance segmentation, its benefits may diminish in tasks lacking structured spatial labels in highly dense-label settings. Exploring self-supervised or task-adaptive selection strategies represents a promising direction for future work.
REFERENCES
Raquel Aoki, Frederick Tung, and Gabriel L Oliveira. Heterogeneous multi-task learning with expert diversity. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 19(6): 3093-3102, 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint Stti::2106.0225555 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660 2021.
Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2061-2070, 2023.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding mixture of experts in deep learning. arXiv preprint arXiv:2208.08 02813, 2022.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in neural information processing systems, 34:9355-9344 99365,99366622 2021.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEFE international conference on computer vision, pp. 764-773, 2017.
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36, 2024.
Xiaoyi Dong, Jiammin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transforer:: A general vision transformer backbone with cross-shaped windows. In Proceedings the the IEEE/CVF conference on computer vision and pattern recognition, pp. 12124-121 2022.
10

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Xiaojing Fan and Chunliang Tao. Towards resilient and efficient Ilms: A comparative study of efficiency, performance, and adversarial robustness. arXiv preprint vrXiv:2408.04588 2024.
Xiaojing Fan, Chunliang Tao, and Jianyu Zhao. Advanced stock price prediction with xlstm-based
models: Improving long-term forecasting. Preprints, (2024082109) August 2024. doi: 10.20944/ sreprints02448.211999. URL https: /tts:://doo.ooo /ttt::///ooo..org/00.2222//////1
v1
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprint 1912. 2019.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International conference on machine learning, pp. 9099-9117. PMLR, 2022.
Tongwen Huang, Qingyun She, Zhiqiang Wang. and Junlin Zhang. Gatenet: gating-enhaneee deep
network for click-through rate prediction. arXiv preprint arXiv:2007.03519 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatiion with gumbel-softmax. arXiv preprint arXiv:1611.00 01 144, 2016
Yixiao Kang, Zhenglin Zhang, Meiqi Zhao, Xuanhui Yang, and Xubo Yang. Tie memories to e-souvenirs: Hybrid tangible ar souvenirs in the museum. In Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp. 1-3, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yer Lo, et al. Segment anything. arXiv preprint arXiv:2304.0266 02644333....43 2023.
Dimitrios Konstantinidis, Ilias Papastratis, Kosmas Dimitropoulos, and Petros Daras. Multi-manifold attention for vision transformers. IEEE Access, 2023.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Im- age restoration using swin transformer. In Proceedings of the IEEE/CVE international conference computer vision, pp. 1833-1844, 2021.
Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800, 2022.
Sen Lin, Li Yang. Zhezhi He, Deliang Fan, and Junshan Zhang. Metagater: Fast learning of conditional channel gated networks via federated meta-learning. In 2021 IEEE 18th International Conference Mobile Ad Hoc and Smart Systems (MASS), pp. 164-172. IEEE, 2021.
Lucas D Lingle. Transformer-vq: Linear-time transformers via vector quantization. arXiv preprint DrXiv:2309.16335 2023.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong. et al. Swin transformer v2: Scaling up capacity and resolution. Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 22009-12011 2022.
11

---

Published as a conference paper at ICLR 2026
Haowei Ni, Shuchen Meng, Xupeng Chen, Ziqing Zhao, Andi Chen, Panfeng Li, Shiyao Zhang, Qifu Yin, Yuanqing Wang, and Yuxi Chan. Harnessing carnings reports for stock predictions: A glora-enhanced Ilm approach. arXiv preprint arXiv:24088 4486:42408.06634, 2024a.
Haowei Ni, Shuchen Meng, Xieming Geng, Panfeng Li, Zhuoying Li, Xupeng Chen, Xiaotong Wang, and Shiyao Zhang. Time series modeling for heart rate prediction: From arima to transformers. arXiv preprini arXiv:2406. 22::::206..99999 2024b.
Felix Petersen, Christian Borgelt, Hilde Kuchne, and Oliver Deussen. Deep differentiable logic gate networks. Advances in Neural Information Processing Systems, 35:2006-2018, 2022.
Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self- attention for long document understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2555-2565, 2020.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937-13399, 2021.
Yongming Rao, Zuyan Liu, Wenliang Zhao, Jie Zhou, and Jiwen Lu. Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1509):18833-1897 2023.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 1701.0170,.,6588, 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Cong Wei, Brendan Duke, Ruowei Jiang, Parham Aarabi, Graham W Taylor, and Florian Shkurti. Sparsifiner: Learning sparse instance-dependee attention for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22680-22668 2023.
Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4794-4803, 2022.
Xuanhui Yang, Yixiao Kang, and Xubo Yang. Retargeting destinations of passive props for enhancing haptic feedback in virtual reality. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 618 619. IEEE, 2022.
Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogencous federated learning: State-of the-hhrrrtttt and research challenges. ACM Computing Surveys, 56(3):1 1-44, 2023.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Yang, al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297, 2020.
Tianyi Zhang, Kishore Kasichainula, Yaoxin Zhuo, Baoxin Li, Jae-sun Seo, and Yu Cao. Transformer- based selective super-resolution for efficient image refinement. arXiv preprint arXiv:2312.05803, 2023.
Tianyi Zhang, Kishore Kasichainula, Yaoxin Zhuo Baoxin Li, Jac-Sun Sco, and Yu Cao. Patch-based selection and refinement for early object detection. In Proceedings of the IEEE/CVF Winter Conference Applications Computer Vision, pp. 729-738, 2024.
12

---

Published as a conference paper at ICLR 2026
Yiming Zhou, Zixuan Zeng, Andi Chen, Xiaofan Zhou, Haowei Ni, Shiyao Zhang, Panfeng Li, Liangxi Liu, Mengyao Zheng, and Xupeng Chen. Evaluating modern approaches scene reconstruction: Nerf vs gaussian-base based methods. arXiv preprint arXiv:2408.0m 04268, 2024.
APPENDIX
A
GATE MECHANISM IN THE SELECT-AND-PACK MODULE
This section provides additional details on the gating mechanism used in the Select-and- (SnP) module, complementing the description in Section 3.2 Similar FLASH attention (Hua al.)
2022) which proposes a transformer with linear time complexity, utilizing Gated Linear Units (GLUs) Shazeer 2020p, the gate is lightweight, single-layer linear projection applied each token embedding to produce a scalar importance score. Formally, given token feature I;, the gate computes:
8; a(W + bg)
(11)
These scores s; represent token informativeness and are supervised using the multi-scale selection labels described in Section 3.2
To enable end-to-end training while sampling discrete tokens, we use the Gumbel-Softmax straight- through estimator. The sampling probability for each token is:
Gumbel-SSottaxx(s..77
(12)
where is the temperature parameter.
This allows gradients to flow through the sampling operation during training.
B
MORE RESULTS AND ANALYSIS
B.
SELECT Loss
We experimented with weighted cross-entropy emphasize the selection of positive tokens. This approach indeed increased the number of selected tokens. However, it also included more background tokens, which resulted worse performance. For example, when the weight was set to in our experiment on BDD-S, the selection ratio increased from 33% to 71 71,, but the final mAP dropped to 5.3.
B.2 SPT LIGHTWEIGHT VARIANTS
We actually offer two variants of the proposed SPT. The version shown in the main text (SPT-T/16) emphasizes higher accuracy and thus introduces slightly more transformer blocks, resulting in modest increase in parameter count.
To provide a fairer comparison in terms of model size and efficiency, we also implemented lightweight variant (SPT-T/12) that maintains similar parameter count to DynamicSwin-T while achieving better performance and higher FPS. Table 8 provides detailed comparison on the BDD dataset.
Table 8: SPT-T/16 achieves the best performance with slightly higher computational overhead compared to SPT-T/12; however, its FPS remains lower than that of DynamicSwin-T (Rao et al. 2023).
Model
AP
#Params
FPS
DynamicSwin-T (Rao et 2023)
22.0
48
46
SPT-T/12
22.3
48
62
SPT-T/16 (Main text)
22.6
55
58
13

---

Published as a conference paper at ICLR 2026
B.3 SDEEC-TOOKE RETENTION ANALYSIS
We analyzed the proportion of object-containing tokens retained by SPA after selection. An object- containing token is defined as any token whose receptive field overlaps with ground-truth object mask by 2 pixel.
Results show that 92.3% of ground-truth object tokens are retained, with the minor loss (7.7%) confined mostly to non-critical object boundaries. Furthermore, the density of object-containing tokens improves drastically from 11.2% to 49% after screening.
B.4 PERFORMANCE CERFORMANCE-EFFFIIEEEYY TRADE-OFF ANALYSIS
Unlike traditional sparse attention methods that require predefined pruning ratios, SPA learns an input-adaptive selection policy. This allows the model to retain more tokens for complex scenes and prune aggressively when the background is simple.
To validate this design, compare SPT with fixed pruning baselines. We enforee fixed percentage of tokens to be kept at Stages 3-4 and evaluate performance.
Fig. 6 shows that Dynamic SPA achieves the highest accuracy (22.6 AP) even while pruning more tokens (74%) on average than fixed 50% pruning. Fixed heavy pruning (80%) causes a noticeable accuracy drop. This demonstrates that input-adaptive token selection is crucial: scenes with dense traffic naturally require more tokens to be preserved, while sparse scenes allow aggressive pruning.
Performance (AF) Pruning Trade-off BDD100K
25
20%
22.4
22
.00%
Objct Detection
Pruning Rabo
SF1-1 Dynamic (0ptimal)
flsseinn
15
lokens Pruned (righer More Efficient!
Figure 6: Performance vs. Pruning Level BDD100K
C LIMITATIONS
Currently, the proposed method needs segmentation masks or bounding boxes to create select labels for supervision. Although our main target task is object detection, some vision tasks may lack this kind of labels. While some models can be used to generate these labels, like SAM (segment-anythinng) the generation process may introduce some errors.
D IMPACT STATEMENT
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
14

---

Published as a conference paper at ICLR 2026
E
SOURCE CODE
We provide the anonymized source code in a zip file.
15


Reviews:
[{"review_id": "bcBK9ToviB", "reviewer": "Reviewer_mkzg", "summary": "This paper proposes a new sparse attention mechanism, \"Select and Pack Attention\" (SPA), to make vision transformers more efficient, particularly for object detection. The core problem is that standard ViTs waste computation on uninformative background tokens.\nSPA tackles this in two steps:\nSelect: It uses a lightweight linear gating layer to dynamically choose informative tokens. Crucially, this gate is explicitly supervised during training using ground-truth object labels (like bounding boxes), which helps it learn to be more accurate than heuristic-based pruning methods.\nPack: To handle the irregular number of selected tokens per image, it packs all selected tokens from a batch into fixed-size \"package containers.\" This allows for efficient, parallel GPU processing without the massive padding overhead of other sparse methods.\nThe authors integrate SPA into a hierarchical backbone called the \"Select and Pack Transformer\" (SPT), which they apply only in the later stages to preserve early-stage features. They show that SPT improves both performance and computational efficiency compared to other state-of-the-art sparse attention models.", "questions": "See W1 and W5", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strength of this paper is the idea of using explicit supervision from downstream labels (bboxes/masks) to train the token selection gate. This is a simple and effective departure from most prior work, which relies on unsupervised heuristics (like token norms) or uniform pruning. This supervised approach is far more likely to preserve small, important objects, which is a common failure point for other sparse methods. The \"packing\" mechanism is also a smart engineering solution to the practical problem of running dynamic token selection in a batched GPU environment. This combination makes for a practical contribution to the field of efficient object detection.\n\nThe experimental validation is reasonably comprehensive. The authors don't just test on COCO; they also use BDD100K and create the BDD-S dataset to specifically test their hypothesis about small objects. The strong results on BDD-S are a convincing piece of evidence. The ablation studies in Tables 6 and 7 are also good, clearly justifying the design choice of applying SPA in later stages and showing the benefit of the selection loss.\n\nThe paper is exceptionally well-written and easy to follow. The problem is motivated clearly. Figure 1 provides a good visual intuition for the problem, and Figures 2 and 3 clearly lay out the proposed architecture and the core SPA block.", "weaknesses": "- The selection gate is trained using ground-truth object labels. This works great for fine-tuning on a detection or segmentation dataset, but it's unclear how this backbone could be pre-trained on a classification-only dataset like ImageNet. Does this method forgo pre-training entirely? Or does the gate have to be trained from scratch only during fine-tuning? This reliance on labels seems like a step back compared to other token selection methods. For example, EViT and DynamicViT learn to prune tokens dynamically without needing explicit object-level supervision. EViT, for instance, uses the attention to the class token as a guide, and DynamicViT trains its prediction module differentiably. \n\n- The method is conceptional similar to EViT but using ground truth labels to learn how to prune.\n\n- Missing Related Work: The related work section on efficient transformers (2.2) is missing some relevant research on sparse attention patterns. This paper would be much stronger if it positioned itself relative to: \n\n   - Fixed sparse patterns (e.g., \"Blockwise Self-Attention for Long Document Understanding, Big Bird: Transformers for Longer Sequences\"). \n\n   - Methods that combine local and global attention patterns (e.g., \"Twins: Revisiting the design of spatial attention in vision transformers\"). \n\n   - Axis-decoupled attention pattern (e.g., \"Axial attention in multidimensional transformers\"). \n\n   - Learned sparse attention pattern (e.g., \"Sparsifiner: learning sparse instance-dependent attention for efficient vision transformers\").\n\n- Modest Overall Speedup: \nWhile the backbone computation reduction of 10-25% is good, the actual wall-clock speedup (FPS) reported in the tables is modest. For example, in Table 1, SPT-T (54 FPS) is only slightly faster than the dense Swin-T (50 FPS). The authors themselves note that the early (dense) stages dominate the computation, which limits the total possible gains\n\n- Missing Performance vs. Pruning Trade-off Analysis: The experimental evaluation is the lack of a \"performance vs. efficiency\" trade-off curve. The authors present results for what seems to be a single, fixed operating point. But they didn't show how performance is affected by the level of pruning. An ablation study is missing: what happens to the accuracy as you change the selection ratio and apply selection on shallow layers? For example, how gracefully does performance degrade if the model is forced to prune 80% of tokens? How much does it improve if it only prunes 50%?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ME6rmyIIs", "reviewer": "Reviewer_Q89m", "summary": "This work proposes Select and Pack Attention (SPA) and build a computationally efficient Vision Transformer architecture based on SPA. Specifically, the authors introduce several linear layers to predict the importance scores of visual tokens. Through sampling based on the score, SPA can extract the informative tokens, thereby reducing the sequence length involved in the attention process. Experimental results demonstrate that the proposed method achieves advantages in both accuracy and speed.", "questions": "1.\tIn line 260, the authors mention “we set L to be M^2.” How is this value determined? As my understanding the length of the package containers should depend on the token sequence length corresponding to the scale of each stage, as well as the proportion of the predicted target relative to the entire image. Why is it related to the window size?\n2.\tWhy is the Gumbel-Softmax operation introduced? Once the importance scores are determined, what is the difference between sampling based on these scores and deterministically selecting high-importance tokens (e.g., those above a fixed threshold)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe motivation of this work is clear. The authors point out the limitations of previous sparse attention methods, including their uniform sampling strategies and training implementations. They further propose the importance-based sampling scheme and a token packing implementation strategy to effectively address these issues.\n2.\tThe presentation of experimental results is clear and well-organized. In each results table, the authors provide quantitative metrics including model performance, computational cost, and inference speed, which intuitively demonstrate the effectiveness of the proposed method. I also appreciate the experiment corresponding to Fig. 4, which employs GT-based selection to effectively decouple the effects of intermediate-layer GT supervision from the token selection process, thereby validating the effectiveness of the selection strategy itself.", "weaknesses": "1.\tThe main limitation lies in the restricted applicability, which is confined to the 2 tested tasks: object detection and instance segmentation. For tasks with sparser prediction targets, such as classification, captioning, and VQA, there is no available bounding box or segmentation mask to provide supervision for importance score prediction. Conversely, for tasks with denser prediction targets, such as SAM-style segmentation or panoptic segmentation, where targets may cover the entire image, the proposed token selection strategy would degenerate into uniform sampling across the whole image (as all tokens would have selection labels of “1” under the current framework). This greatly limits the general applicability of the proposed method.\n2.\tThe paper demonstrates that the proposed method achieves superior efficiency within the current overall model architecture. However, this improvement arises from multiple contributing factors, such as the hierarchical design, window attention, and the proposed SPA. A more critical aspect is the efficiency gain brought specifically by SPA itself. For instance, under the same model architecture, how much difference in computational cost and inference speed would there be if Stages 3 and 4 used standard global attention compared to SPA?\n3.\tNot a major weakness, but the paper has some formatting issues. Many figures and tables are placed one page away from the corresponding text description that references them, which causes inconvenience for readers. It is recommended to adjust the layout.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jggrzvpNeU", "reviewer": "Reviewer_mRG3", "summary": "To address the issue of numerous target-irrelevant redundant computations in attention calculation for object detection, this paper proposes a more efficient Vision Transformer (VIT). Different from traditional DynamicVIT and SparseVIT, which pad target-irrelevant tokens for a fixed batch size, the proposed SPT (Specific Token Preservation) only retains target-relevant tokens and packages them into the same batch for training. The proposed SPA (Sparse Attention Calculation) performs self-attention computation on the tokens screened and packaged by SnP (Selection and Packaging). Extensive comparisons with baselines on three datasets (COCO2017, BDD-S, and PASCAL VOC 2012) demonstrate that the proposed method achieves the", "questions": "Q1: If the model makes misjudgments in token selection, will more severe and irreversible errors occur? For instance, if an object is classified as background by the gate mechanism, does the model permanently lose the opportunity to learn the feature of it? Besides, Could you design an experiment to compare the proportion of tokens that actually contain objects before and after the screening process?​\nQ2: Will there be significant differences between different gate?​\nQ3: Will the multi-scale labels of small objects disappear as the image undergoes up sampling?​\nQ4: The improvement in computational efficiency is accompanied by limitations in the model's field of view, which may sacrifice the model's expressive ability. Would retaining an appropriate number of background tokens around the image lead to better performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Combines Swin Transformer (SwinT) to propose a more efficient attention computation method, and supervises token selection through multi-scale labels.\n2. Validates the proposed method on three sufficient datasets.", "weaknesses": "1. The overall architecture flowchart is slightly rough, especially in illustrating how multi-scale label selection is trained synchronously.\n2. The gate mechanism of SnP is not clearly defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1LE6oR36X7", "reviewer": "Reviewer_buQf", "summary": "In this paper authors analyzed the current issues with sparse attention mechanisms and propose a  SPA mechanism to address these challenges for both efficiency and performance. The SPA focuses attention computations solely on informative tokens using a supervised gatingblock in Vision Transformers. Experiments validated the effectiveness of SPT.", "questions": "1. In Tables 1 to 3, the comparison method is based on results published in 2021. Why not choose the most recent research?\n2. In the conclusion section, the analysis of the paper's shortcomings should be added.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The credible innovations of the paper are as follows:\nIntegrated into the Swin Transformer’s hierarchical architecture, SPA forms the efficient Select and Pack Transformer, which works as image backbone network for various computer vision tasks and generates multi-scale representations. Employ multi-scale selection labels for explicit supervision using object labels to enhance selection accuracy and ensure effectiveness in complex computer vision tasks. In terms of object detection, the method of this paper compared to the current SOTA , not only improves detection accuracy but also reduces computational costs.", "weaknesses": "In the conclusion section of the paper, the analysis of the paper's shortcomings should be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)