Exploring Token Pruning in Vision State Space Models
Authors: Zheng Zhan
Zhenglun Kong
Yifan Gong
Yushu Wu
Zichong Meng
Hangyu Zheng
Xuan Shen
Stratis Ioannidis
Wei Niu
Pu Zhao
Yanzhi Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: State Space Models
Token pruning
Efficiency
Interpretability
Primary area: machine_vision
TL;DR: We revisit the unique computational characteristics of SSMs and  designed a novel and general token pruning method specifically for SSM-based vision models.

Abstract: State Space Models (SSMs) have the advantage of keeping linear computational complexity compared to attention modules in transformers, and have been applied to vision tasks as a new type of powerful vision foundation model. Inspired by the observations that the final prediction in vision transformers (ViTs) is only based on a subset of most informative tokens, we take the novel step of enhancing the efficiency of SSM-based vision models through token-based pruning. However, direct applications of existing token pruning techniques designed for ViTs fail to deliver good performance, even with extensive fine-tuning. To address this issue, we revisit the unique computational characteristics of SSMs and discover that naive application disrupts the sequential token positions. This insight motivates us to design a novel and general token pruning method specifically for SSM-based vision models. We first introduce a pruning-aware hidden state alignment method to stabilize the neighborhood of remaining tokens for performance enhancement. Besides, based on our detailed analysis, we propose a token importance evaluation method adapted for SSM models, to guide the token pruning. With efficient implementation and practical acceleration methods, our method brings actual speedup. Extensive experiments demonstrate that our approach can achieve significant computation reduction with minimal impact on performance across different tasks. Notably, we achieve 81.7\% accuracy on ImageNet with a 41.6\% reduction in the FLOPs for pruned PlainMamba-L3. Furthermore, our work provides deeper insights into understanding the behavior of SSM-based vision models for future research.

Exploring Token Pruning in Vision State Space Models
Zheng Zhan',, ', Zhenglun ; Yifan Gong', Yushu Wu', Zichong Meng' Hangyu Zheng', Xuan Shen', Stratis loannidis', Wei Niu Pu Zhao', Yanzhi Wang' NNothhesttern University, "Harvard University, University of Georgia kong. zhe, yanz. wang) @nor mmorrheasterr. edu
Abstract
State Space Models (SSMs) have the advantage of keeping linear computational complexity compared to attention modules transformers, and have been applied to vision tasks as new type of powerful vision foundation model. Inspired by the observations that the final prediction in vision transformers (ViTs) only based a subset of most informative tokens, we take the novel step of enhancing the efficiency of SSM-based vision models through token-based pruning. However, direct applications of existing token pruning techniques designed for ViTs fail deliver good performance, even with extensive fine-tuning. To address this issue, revisit the unique computational characteristics of SSMs and discover that naive application disrupts the sequential token positions. This insight motivates us to design novel and general token pruning method specifically for SSM-based vision models. We first introduce a pruning-aware hidden state alignment method to stabi- lize the neighborhood of remaining tokens for performance enhancement, Besides, based on our detailed analysis, we propose token importance evaluation method adapted for SSM models, guide the token pruning. With efficient implementation and practical acceleration methods, our method brings actual speedup. Extensive experiments demonstrate that our approach can achieve significant computation reduction with minimal impact on performance across different tasks. Notably, we achieve 81.7% accuracy on ImageNet with a 41.6% reduction the FLOPs for pruned PlainMamba-L3. L3. Furthermore, our work provides deeper insights into understanding the behavior of SSM-based vision models for future research"
Introduction
Recent years have witnessed the rapid evolvement of the computer vision field in the of deep learning. Significant research efforts have been devoted to designing effective and efficient archi- tectures of deep neural networks (DNNs) for visual tasks. Convolution Neural Networks (CNNs)
are two representative categories of backbone networks. Though ViTs exhibit superior modeling capabilities with the incorporation of the self-attention mechanism 6 32), the complexity of self-attention grows quadratically as the input size increases. Inspired by the great potential of State Space Models (SSMs) for long sequence modeling with linear complexity in natural language processing (NLP) tasks [9] 24
22
12
23
30 and Vision Transformers (ViTs) 6 22
31 43]
33 44 the
latest backbone network designs for visual tasks [10 leverage SSM-based blocks. Particularly, VMamba (2 reduces the complexity of attention computation with the selective scan mechanism presented in the S6 model and matches the performance with existing foundation models.
Like the existing research efforts promoting the efficiency of CNNs and ViTs, the exploration of the SSM efficiency is desirable facilitate real-time applications. While weight pruning is the prevalent
"qqual contributions
Code available at htttpps///gtcaab.co//////ggg/ee ViM
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

technique for CNNs B4 13 to be an effective way to enhance the efficiency of ViTs due to the independent patch processing design. Given that the SSM-based blocks also process input by dividing it into patches like ViTs, the existing token pruning techniques [8] for ViT' can be applied as a straightforward approach to boost the SSM efficiencyy However, as shown in Figure 2 although enjoying certain benefits of faster inference with less tokens, this naive token pruning application for SSMs suffers from significant accuracy drops. Even after extensive fine-tuning efforts, its accuracy is still not able to recover from the token pruning with non-margina gaps compared with the original accuracy. This indicates that the direct application of token pruning designed for ViTs permanently harms the performance of SSM- vision models.
38 7 8 35 40
token pruning 127 26 39 28 16 41 proves
Given this observation, we conduct a thorough analysis of the computation patterns in SSM-based blocks, aiming to find the root cause and provide a foundation for efficient token pruning design in SSMs. Unlike ViTs whose attention mechanism computes the correlation between each pair of patches, SSM-based blocks follow traversal paths and thus the paths are sensitive to their ad- jacent patches. The direct application of token pruning techniques from ViT disrupts the patch bocations/neighbortoodd in SSM-based blocks, thus incurring massive accuracy drops.
Based on our analysis, the question naturally arises whether we can keep the sequential property of tokens/batches in SS.M-based vision models while pruning tokens to accelerate the forward computation. A successful solution not only improves the computational efficiency, but also provides more insights into the interpretability of SSM scan/token for future research. We take the first novel step towards this direction by proposing general token pruning method for -SM--asedd vision models. Specifically, we propose a token importance evaluation method adapted for SSM models to guide the token pruning process based on comprehensive analysis of SSM-based models. More importantly, to address the root cause of the above significant accuracy drop, we introduce a pruning-aware hidden state alignment method to reform the scan mechanism in SSMs for pruned and remaining tokens, thus stabilizing the neighborhood of remaining tokens and enhancing performance. Following the token pruning designs, we explore the efficient implementation and practical acceleration methods. With our tailored design, the computations can be significantly reduced with high accuracy performance. Notably, achieve 81.7% accuracy ImageNet for token pruned PlainMamba--33, with 41.4% FLOPs reduction. We summarize our contributions as follows:
After observing the incapability of directly applying token- baseed pruning techniques from ViTs for vision SSMs, we conduct a comprchensive analysis of SSM-based blocks to identify the failure reason, as well as provide more insights for the SSM scan mechanism in vision tasks, shedding lights on future research on SSM-based vision models.
Based on our analysis, we propose general token pruning method for SSM-based vision models, incorporating an adapted token importance evaluation to determine the pruned tokens, pruning-awe aware hidden state alignment method to reform the SSM scan mechanism for pruned and remaining tokens, and practical implementation for efficient inference.
We take the first step towards accelerating vision SSM models with token-based pruning. Our extensive and comprehensive experiments for image classification and object detection demonstrate the effectiveness of our proposed method for vision SSMs.
2 Related Work
were first proposed tackle long sequence modeling in the NL. community. The design has the strength to model complex systems by focusing on how the input, output, and state variables evolve over time. Recent progress has demonstrated that the variants of SSMs can be applied to visual tasks as an alternative to CNNs and ViTs with promising results. S4ND 25 is the first work that applies the state space mechanism
State Space Models. SSMs 9
24 3K
visual tasks and shows the potential to achieve competitive performance with ViTs (6) The design expands the S4 model [10] and normalizes the parameters into a diagonal structure. But it fails to efficiently capture image information in an deputddepnndent manner. ViM 46) proposes a novel vision backbone with bidirectional Mamba. Based on that, PlainMamba 37 invents a continuous 2D scanning to
enhance spatial continuity by ensuring adjacency of tokens in the scanning sequence. VMambaa [2] introduce Cross-Sean Module (CSM) to enable ID selective sean, matching the performance with
existing foundation models including ResNet [12, VIT (6), Swin (22), and ConvNext [23]. The

---

great accomplishments demonstrate the potential of vision SSMs as an emerging fantastic foundation model family.
Token Pruning. Token pruning is an effective strategy to enhance computational efficiency by reducing the number of processed tokens patches. It enables significant acceleration without requiring additional weights or specialized hardware, aiming to selectively retain the most informative tokens and sparsify the sequence. It is also vital for dense prediction tasks where sequence sizes are extensive. Several innovative approaches have been developed for vision transformers. For example, EVTT 18 uses the attentiveness of the [CLS] token with respect to other tokens identify the most important tokens. DynamicViis VIT [27] and SPVIT [15] add layers that employ the Gumbel- Softmax trick selectively prune less informative tokens. IA-REDA [26] drops redundant tokens with a multi-head interpreter. PS-ViT (T2T) [39 discard useless patches in a top-down paradigm.
PATCHMERGER 28 uses spatial attention to generate a small set of tokens adaptive the input.
ToMe 2 measures dot product similarity between token keys to determine redundancy and prune accordingly. However, the dynamics of information flow between tokens and the learning mechanisms in models like Mamba [9] remain largely unexplored. Unlike ViTs that reply attention features, the absence of attention layers in Mamba makes current pruning methods ineffective. Furthermore, the inclusion of the SSM module prevents the effective use of existing token pruning methods [42]
3
Preliminary and Motivation
3.1 State Space Models
State Space Models (SSMs) are sequential models that map input sequence zzt) 12 to an output sequence y(t E IR_ through a hidden state h(t) E R as follows,
h' (t(t) Ah(t) + Bx(t),
(I)
y(t) = Ch((),
where L denotes the length of the sequence, N denotes the number of representation dimensions, E NxN is the evolution matrix, B E NR and C E RLXN are the projection matrices.
The Mamba model [9] represents discrete version of the continuous system for SSMs and incor- porates timescale parameter A to facilitate the transformation of continuous parameters with the zero-order hold (ZOH) as follows,
= exp(AA),
(2)
(AA) (exp(AA) -)- AB.
After obtaining the discretized A and B, the discretization of Equation can be rewritten follows,
h, Ah,-1 + Bag,
8) Ch..
Finally, the Mamba model computes the output through global convolution as follows,
K = (CB, CAB. CA--BB)
(4)
= x K,
where y denotes the output sequence, L denotes the length of the input sequence x, and K e R.
denotes a structured convolutional kernel.
3.2 Failure of Applying VIT Token Pruning for ViMs
(Observation) After applying token pruning method to an SSM-based vision model, the Zero-shot performance will drop significantly. Moreover, this process will permanently harm the model's 's performance, even after extensive fine-tuning.
Epic failure of traditional token pruning for vision SSMs. To explore the token sparsity in vision SSMs, we first prune tokens SSM-based models with the 'must-try' baseline, which directly applies the token pruning techniques designed for ViTs. Specifically, prune the tokens using
3

---

6
9
13
15
16
17
19
22
10
13
6
7
48
15
25
16
17
19
19
VIT pruning
pruning
Condensed token matrix
Actual after token pruning
Figure 1: Illustration of the cross-scan in ViM models before and after token pruning.
selector metrics of EVIT 18 on both transformer-basee VIT-S [6] and SSM-based VIM-S models [46]. Given input tokens for one layer, with token pruning, K tokens remain while the other N K tokens are pruned. The remaining tokens are relabeled as (x, } K-1 and their hidden states obtained following Equation 3 In this way, the number of the tokens and the corresponding hidden states are reduced, condensing token matrix to save computation costs, as shown in Figure || After pruning tokens based the token selector, and fine-tuning tuning accuracy. The results are shown in Figure As observed, the direct application of token pruning from ViTs is not capable of delivering satistying performance ViMs. Specifically, direct token pruning suffers from substantial zero-shot accuracy degradation on ViM-$ (with over 68% accuracy drop compared with the original accuracy), despite its success for VIT-S with merely 1.4% accuracy drop. Furthermore, even after extensive fine-tuning for the pruned model, its accuracy restored still with 5.7% accuracy gap compared with the original ViM-S model, while it can boost the accuracy to a competitive level on ViT-S after fine-tuning. The significant performance degradation in ViM-S demonstrates that the direct application of token pruning hurts the underlying computations in SSM-based blocks, with permanent negative effects which can hardly be restored after fine-tuning.
evaluate the performance in terms of zero-shot
Computation patterns in vision SSM. Observing the great success for ViT-S and epic failure for ViM-S with the motivated revisit
model
Zero-shot
Fine mun-ttniing
80
79.9
78.5
79.4
80
74.8
same method, the unique computation characteristics of SSMs and rethink the token pruning strat- egy in ViMs. To figure out the reason of failure, we look into the token computation patterns in SSM-based blocks. Given the input data, SSM-based blocks first unfold image patches/tokens into sequences along traversal paths (i.e., cross-scan scan, as shown in Figure I with ViM scan), process each token sequence using a separate computa- tion block in parallel, and subsequently reshape and merge the resultant sequences to form the output map (i.e., cross-merge). The traversal paths facilitate the integration of information from all image pixels in various directions with linear complexity, enhancing the model's understanding.
Top-1 Accuracy
Acc.
Acc.
Drop
Dropi
5.7%
>68%;
(e) Token pruning on VIT-S
(b) Token pruning on VIM-S
Figure 2: Accuracy comparison for token pruning on transformer-basee ViT-S and SSM-based ViM-S.
Reason for the failure. However, the unique traversing along the sequence paths in ViM makes each token sensitive to its neighboring tokens. This is not problem for ViT as the quadratic design of the attention mechanism calculates the correlation between the target token and all other tokens introducing in the image, eliminating the sensitivity to adjacent tokens. As shown in Figure [ token pruning strategy within an SSM-based block disrupts the original token positions in the SSM scan. Consequently, tokens that were not previously adjacent become neighbors during the scan distorted scan functionality and a significant accuracy
different directions or paths, leading degradation. Especially considering that the tokens are actually image patches in visual tasks with semantic information, disrupting their positions during the sean brings great difficulties understand their relationship and the overall semantics.

---

In response to the limitations of directly applying existing token pruning methods designed for ViTs, we aim to address the following question:
(Question) Can we prune tokens in SSM-based vision models to accelerate their forward computation without disrupting the original sequential token positions in different directions during the scan?
Methodology
To address the above Question, we propose a general token pruning method tailored for SSM- based vision models. Specifically, we propose a pruning-awar hidden state alignment method to stabilize the neighborhood of remaining tokens during the scan, addressing the distorted scan functionality in traditional token pruning and thus enhancing accuracy performance. Furthermore, based on our detailed analysis of SSM-based vision models, we propose token importance evaluation method adapted for SSM models, to guide the token pruning. Moreover, discuss the efficient implementation and practical acceleration methods for bkenn-runned SSM-based vision models.
4.1 Pruning-Aware Hidden State Alignment
To maintain the sequential property of SSM tokens during the scan and tackle the Question, we propose the following novel pruning-aware hidden state alignment technique to align sequential positions or neighhourhood of tokens before and after token pruning during the scan, thus maintaining the model performance under token pruning. For SSM-based vision models, the input token sequence for the 1th layer is denoted as Ti BxXXXD where B, N, and D are the batch size, token number, and hidden state dimension, respectively. The tokens in one batch of the sequence can unfolded as tokens are kept while the other N K tokens are removed from the input token sequence. We adopt different strategies to align the hidden states of remained tokens and pruned tokens during the scan as detailed below.
{xy ) N-11 with N tokens total. After applying token pruning (Section 4.2
Alignment of hidden states for remaining tokens. We denote the of the remaining token indices {qN 1 with K elements and 9s qi if s < t. Formally, the pruning the scan corresponding to the remained tokens can be represented as
-wwin aware hidden states during
B go:
A
00 BIg
+ BX
(5)
A
1(--1)
BIg. + - Brg, + A (K-1) " Bz +
Brqu
K terms/tokcns
As shown in Equation (5). the hidden states of remained tokens depend on its current token and all previous remaining tokens. The pruned tokens are not effective in the hidden states.
Aligument of hidden states for pruned tokens. As observed in Figure I andd [ if one token is pruned, removing position during the scan disrupts the neighbourhood of its adjacent tokens, leading to significant zero-shot accuracy drop which can hardly be compensated even after extensive fine-tuning. To mitigate this problem, our pruning-awe position gap from pruned tokens during the scan to stabilize the neighbourhood of all remaining tokens. Specifically, to make the problem tractable, for two adjacent remaining tokens Vig and T if 1++1 4: > 1, meaning there are tokens pruned between 2ig and Ty 11 we denote the number of pruned tokens between Tq:: and Fg as K, (K; 1) and their indices can be represented as (g: + K₁ < 9(++1)- To highlight the difference between 6++1 and + 1, we use round brackets in the expression (e.g., ((-1)) and (g:) + 1) without changing their meanings. Thus, the hidden states for the pruned tokens between two remaining adjacent tokens can be represented follows.
hidden state alignment maintains the
{q: + We have 9: < (q:) +1
5

---

1/4 A th
gi go
+ A
51
Br +... + B
W M(a+11 A (4)+1-(00) (16)
gin A (1:)+1((1) Ba + ... + A A
(6)
(1)) (1))K;-00 (10)
Bz.. + + (+)+k;-(o) B +... ... A K Bzq
Ma))K.
W Mquen ROLD
BB
((41)) 91 BIg + + A 9i41y - B.ng. + Brg
For pruned tokens with indices smaller than qo- their hidden states are set to zero. For pruned tokens with indices larger than QK-1, their hidden states can still be obtained following Equation (6) As shown in Equation (6). if a token pruned, we do not simply remove its corresponding hidden state during the scan as it leads to substantial accuracy degradation shown in Figure | and 2 Instead, its hidden state in the scan can be obtained by using the previous state with one step forward, i.e.,
Al(a.) where the token *(g+++1 is pruned. In this way, the hidden states corresponding to pruned tokens are aligned with that of the original unpruned tokens maintain
M(a)11 A (o.) + (x)(+
the sequential positions of the original tokens without disrupting their neighbours.
Comparison with traditional token pruning. As discussed in Section 3.2 in traditional VIT token pruning, the remaining tokens are relabeled ) (disrupting their neighbours due removal of pruned indices) and their hidden states are obtained following Equation (3). Different from ViT token pruning, we still keep the original indices of all tokens (including remaining and pruned tokens) to record their original sequential positions and neighbourhood. During the scan, the hidden states of pruning tokens becomes completely zero in VIT token pruning, which different from our adapted scan mechanism in Equation 6 to keep copy from its previous unpruned neighbour.
4.2 Token Pruning based on Importance Evaluation
In SSM-based vision models such as ViM, for the 1th layer, the input token sequence Ti-1 E BxND is first projected to X' E BxxxxD and then goes through bidirectional SSMs for data- dependent global visual context modeling. It processes X' from the forward and backward scan via:
ym < SSM Bm, (X,),, for m E {forward, backward),
(7)
where Yo. E RR Nx the output of SSM. Then ym gated to obtain Vforaard and Hbackuard- The token sequence output of the tth layer can be obtained follows:
T Linear" (ytorvard + Voackvard) + Ti-1-
(8)
Therefore, the output of SSM can directly reflect the token importance. The Mamba architecture, with its nigh-dimensional channel space, allows for a finer-granule ener-granularity analysis of attention across numerous channels. Unlike Transformers that produce single attention matrix per head, Mamba models exploit their extensive channel capacity for a more detailed attention distribution, enhancing the model's ability to discern subtle features and interactions among tokens. Thus, aggregate the clipped values across all channels for each token to evaluate token importance as follows,
D' E max (0, (aa)
D'
where Hed denotes the ghh feature map in the feature dimension with size D'. We use S as the token importance metric guide the pruning process, ensuring that only the most contextually relevant tokens are retained, thereby optimizing computational resources. Given the sparsity requirement for the token pruning, we sort and prune the corresponding less important tokens. To make a comprehensive study, we compare the performance with other token importance metrics, including the & norm, & norm, as well as unclipped values without the max operation. We find that using clipped values in Equation (9) the token importance metric can constantly yield better results.

---

4.3 Efficient Implementation and Practical Acceleration
Efficient implementation for the SSM scan. Based on the pruning- -ware hidden state alignment technique discussed in Section we propose the pruning- aware hidden state al alignment kernel for practical acceleration. It utilizes a position map to guide the SSM operator, ensuring the correctness of computations. The position map is the token pruning indicator based on { in Section which inherits from token importance evaluation and records the location of remained tokens and pruned tokens. The pruning-awa -ware hidden state alignment kernel takes the pruned dense sequences and the position map as its inputs. During the scan, it switches between the token remaining pattern and the token pruned pattern based on the remaining/pruning state of the token indexed by the position map. The token remaining pattern in the kernel follows the computations in Equation ( Similarly, following Equation (6. the token pruned pattern still updates the hidden states but ignores computations related to the current token. The kernel switches to another pattern if it detects a corresponding change in token pruning state. A pseudo-code for our pruning-ave aware hidden state alignment is demonstrated in Appendix A Thus, the pruning aware hidden state alignment kernel can effectively accelerate the SSM scan under token pruning.
Practical acceleration for the whole model. Note that the SSM scan only takes up around 10~20% computations in the whole model. With less tokens, other parts in the model can be accelcrated directly due to less computations from pruned tokens, leading significant inference speedup performance as demonstrated in our experiments.
5 Experiment Results
We conduct comprehensive experiments on ImageNet-IK[A) COCO 2017 [20] and ADE20K [45] datasets. All experiments are conducted on 4 NVIDIA V100s. "-EViT" means apply EViT token pruning method. "- "-prune" means apply our token pruning method. We report average results of multiple runs for all experimental sections, and different runs do not vary much. For ViM-T, we prune after the 10th and 20th layers. For ViM-S, we prune after the 5th, 10th, 15th, and 20th layers. For PlainMamba-LI, we prune after the 5th and 10th layers. For PlainMamba-L2, we prune after the 5th, 10th and I 5th layers. For PlainMamba-L33, we prune after the 5th, 11th 17th, and 23th layers.
5.1 Image Classification on ImageNet--1K
Settings. We finetune both the ViM and PlainMamba for 30 epochs on the ImageNet- IK dataset. The top-1 accuracy on the validation set reported. For ViM, we set a patch extraction stride of 8 while keeping the patch size unchanged, constant learning rate of 10 10-0, and a weight decay of 10-5, For Plmin, we use a warm-up period of 5 epochs. The weight decay is set to 1e-8, the base learning rate to 2e-5, the warm-up learning rate to 2e-8, and the minimum learning rate to 2e-7.
The comparison results of our token pruning models against benchmark backbone models on ImageNet- IK are summarized in [a One advantage of our method is that it is general and can be applied to a wide range of SSM-based vision model architectures to reduce computation complexity with minor loss of performance. We evaluate our method on five base models including ViM-T, ViM-S, Plain.amma--LI. PlainMamba-LL2. and PlainMamba-L33. We report the top-1 accuracy and FLOPs. Compared to directly applying the EVIT method on vision state space models, using our pruning- hidden state alignment and token importance metric constantly outperforms EVTT across various models of different scales. Specifically, On ViM, our method surpasses EVIT by 3.8% on ViM-T and 4.0% on ViM-S. On PlainMamba, our method exceed EViT by 2.4% on PlainMamba-LL1, 2.7% on Plain-amba--12, and 2.8% on PlainMamba--L3.
Results.
5.2 Object Detection and Instance Segmentation
Settings. Following previous works, we conduct experiments for object detection and instance segmentation on the COCO 2017 dataset. The COCO 2017 dataset contains 118K images for training. 5K images for validating, and 20K images for testing. We use both the two-stage Mask R-CNN [T] and the single-stage RetinaNet [9] For both models, we report the results of both 1x schedule. to compute multi-scale features to fit the FPN network
Following (37] we use ViTAdapter structure.
7

---

Table 1: Classification results of different models ImageNet- IK. We compare the proposed token pruning method with existing methods under comparable GFLOPs.
Method
Img. Size
Params (M)
FLOPs(G)
Top-1 Acc. (%)
ViT-Base
3842
86
55.40
77.9
ViT-Large
384
307
190.70
76.5
Deny-Tiny
224
1.30
72.2
Deil-Small
224
22
4.60
79.8
DeiT-Base
221
86
17.50
81.8
ViM-T
224
1.50
76.1
ViM-S
224
26
5.10
80.5
ViM-T-EViT
224
1.28 (-14.3%)
71.3
ViM-S-EViT
224
26
3.57 (-30.0%)
74.8
ViM-T-ToP
2212
7
1.29 (-14.0%)
75.1
ViM-S-ToP
224
26
3.60 (-29.4%)
78.8
Plain-amba-LL1
224
3.0
77.9
PlainMamba-LL2
224
25
8.1
81.6
Plain-amba-L33
224
50
14.4
82.3
PlainMamba-LL--Viit
224
7
2.44 (-18.7%)
75.0
PlainMammba-I2-EEViT
221
6.22 (-23.2%)
78.3
Plain-Mamba-LL--EVii
224
50
8.35 (-42.0%)
78.9
PlannMamba-LL--ToP
221
2.46 (-18.0%)
77.4
PlainMamba-L2-ToP
224
25
6.27 (-22.6%)
81.0
Pluin-Mamaa-L3--TTPP
2212
50
8.44 (-41.4%)
81.7
Table 2: Results on COCO object detection and instance segmentation.
Backhone
Apb
A A
AP 75
Apm
A pm
AP
76
PVT-Small
40.4
62.9
43.8
37.8
60.1
40.3
PVT-Medium
42.0
64.4
45.6
39.0
61.6
42.1
PVT-Large
42.9
65.0
46.6
39.5
61.9
42.5
Swin-Tiny
42.7
65.2
46.8
39.3
62.2
42.2
Swin-Small
44.8
66.6
48.9
40.9
63.2
44.2
PlainMamba-LL
64.8
47.9
39.1
61.6
41.9
PlainMamba-L.2
46.0
66.9
50.1
40.6
63.8
43.6
PlainMamba-L3
46.8
68.0
51.1
41.2
64.7
43.9
PlainMamba-LL--EViT
41.9
62.8
45.7
37.2
60.1
40.2
PlainMamba-LL--EViI
43.7
64.2
47.6
38.3
62.2
41.9
PlainMamba-LL3-EViiT
44.2
66.4
49.7
39.5
62.8
42.7
PlanMMamba-LI-ToP
43.7
64.6
47.4
38.9
61.3
41.5
PlainMamba-L22-TTP
45.5
66.2
49.9
40.1
63.3
42.7
PlainMamba-L.3-ToP 46.5
67.7
50.8
40.6
64.1
43.4
Results. We used our pruned PlainMamba models as the backbone and compared them with existing token pruning methods and dense backbones. As shown in Table 2 our token pruning method maintains similar performance to dense models (less than 0.5% When compared to existing token pruning methods, specifically for PlainMamba-LLI our pruning method outperforms EViT- based pruning by average of 1.59% across all six precision metrics. For PlainMamba-L2,, our method surpasses EVIT by an average of 1.63% across all six precision metrics. Additionally, for PlainMamba--33, our method exceeds EVIT by an average of 1.30% across all six precision metrics.
5.3 Semantic Segmentation on ADE20K
Settings. We conduct experiments semantic segmentation on the ADE20K dataset 45 ADE20K contains 150 fine-graines semantic categories, with 20K, 2K, and 3K images for training, validation.

---

and testing, respectively. We choose UperNet [36] as our base framework. We train all models for 160 iterations with batch size and set the default training image size 512x512.
Results. We show the results of semantic segmentation on ADE20K in Table The results indicate that our method also works well for the semantic segmentation task by greatly reducing the computation costs while maintaining satisfying performance. For instance, our token pruned PlainMamba-LI reaches a mloU of 44.1% 1%, which the same the unpruned PlainMamba-La Our ShinMamma--L-pruma has a mloU of 48.6%, which is bet- ter than current state-of-te tf-th--rrt model architectures including Local Vim-S and VMamba-T.
Table Semantic Segmentation.
Method
mloU/%
ViM-T
41.0
ViM-S
44.9
LocalVim-TT
43.4
LocalVim-S
46.4
PlainMamba--.I
44.1
Plain-amba--L2
46.8
PlainMamba-LL3
49.1
5.4
Ablation & Analysis
PlainMaamba-L--EViT
42.2
PlainMManbba-L2-EViii
44.1
5.4.1 Token Importance Metric Analysis
PlainMaambs-L3--VVii
46.3
PlainMaamsa-LI-ToP
44.1
In Table we study on the impact of different token importance metrics, focusing pruning-awan hidden state alignment. We test on two models: ViM-$ and PlainMamba--L33 For the ViM-S model, both - and 22-mo methods achieve an accuracy of 78.6%, while the method without clipping (w/o Clip) results in a lower accuracy of 77.4%. The proposed clipping method (Clip) achieves the highest accuracy 78.8%. For the L3 model, similar trends are observed: the ---oorm and 62-nrmm methods yield accuracies of 81.6% and 81.5%, respectively. The non-clipping approach results decrease in accuracy to 80.5%, whereas the clipping method provides better enhancement, achieving 81.7%. These results suggest that the clipping mechanism in token importance metrics offers a consistent improvement in model accuracy, particularly the context of pruning-ane aware hidden state alignment. It can potentially mitigate the adverse effects of extreme token importance values.
PlainMamma-LL2-TPP
46.5
Llain-Mamb--L--TPP
48.6
5.4.2 Quantitative Evaluation of pruning-aware hidden state alignment.
Table 5: Comparison of w/o and w/ our alignment (both using Eq. 9
token importance metric).
Model
Method
FLOPs
Top-1 1 Acc. (%)
Throughput
Dense
5.10G
80.5
1
ViM-S
Prune w/o alignment
3.57G
75.4
1.30x
Prune w/ our alignment
3.60G
78.8
1.27
Dense
14.40G
82.3
1x
PlainMamba-Le
Prune w/o our alignment
8.35G
79.3
1.4xx
Prune w/ our alignment
8.44G
81.7
1.43x
In Table compare the performance of different prun- ing methods across two models: ViM-S and PlainMamba- L3. Our pruning method without the alignment process reduces the FLOPs to 3.57G but also lowers the accuracy 75.4%, resulting in improved throughput of 1.30 In contrast, adding the align matrix achieves a much higher accuracy 78.8%. with similar throughput of 1.27 For the PlainMamba--3 model, our pruning method with- the alignment reduces FLOPs 8.35G but decreases accuracy to 79.3%, while increasing throughput to 1.47 Equipping the alignment process improves accuracy 81.7% and achieves throughput of 1.43x. These results demonstrate that the proposed pruning method with the alignment process can effectively balance computational efficiency and model accuracy, outperforming the baseline pruning approach.
Table 4: Ablation study of token impor- tance metric with pruning-awe aware hidden state alignment.
Model
Method
Accuracy (%)
f--norm fz-norm
78.6
78.6
ViM-S
w/o Clip
77.4
Clip (ours)
78.8
f--norm &g-norm
81.6
81.5
L3
w/o Clip
80.5
Clip (ours)
81.7

---

Prune w/o our align.
Prune w/ our align.
Prune w/o our align.
Prune w/ our align.
Original
Dense
Original
Dense
Figure 3:
Visual representation on ImageNet-IK. We present the original images, attention visualiza- tions from ViM-S, and zero-shot results of w/o and w/ our alignment method after the final layer.
5.5 Visualization and Interpretability
To further interpret token pruning in SSM-based vision models and understand the pruning-aware hidden state alignment behavior of our approach, present attention visualizations based on zero- shot results in Figure 3 Our pruning-aware hidden state alignment effectively aligns the hidden states of pruned tokens the SSM scan, maintaining similar visual representations and atfention regions as the dense model. In contrast, pruning without our alignment shows significantly different attention regions, which could explain the huge accuracy drop. This demonstrates the effectiveness of our proposed pruning-awand aware hidden state alignment. The visualization tool is adopted from [[].
Figure 4: visualizations of locations of pruned token. We use the output after the final layer to visualize this reduction results.
We further visualize the token reduction results of our method within Fig. 4
We show the input images along with their sparsification results. The masked regions represent the tokens that have been pruned. Our method can gradually drop less informative tokens during forward pass and preserve the tokens that contain representative regions with adaptive pruned region for each image.
Conclusion and Limitation
In this paper, we take the first step toward accelerating vision SSM models with token-based pruning. We analyze SSM-based blocks understand the failure of direct token pruning and propose general token pruning method for SSM-based vision models. This method includes an adapted token importance evaluation, pruning-aware hidden state alignment, and practical implementations for efficient inference. Our extensive experiments confirm the effectiveness of our method and provide deeper insights into the SSM scan mechanism, guiding future research on SSM-based vision models, Though our method is general, the efficiency is limited by baseline model architecture design.
Acknowledgement
This work is supported by National Science Foundation CNS-2312158, and also CCF-2428 108, OAC-2403090. We would like to express our sincere gratitude to the reviewers for their invaluable feedback and constructive comments to improve the paper.
10

---

References
[1] Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models, 2024.
[2] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023.
[3] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.085.2 085304, 2022.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255,20 2009.
[5] Peiyan Dong, Mengshu Sun, Alec Lu, Yanyue Xie, Kenneth Liu, Zhenglun Kong, Xin Meng, Zhengang Li, Xue Lin, Zhenman Fang, et al. Heatvit: Hardware-efficient adaptive token pruning for vision transformers. 2023 IEEE International Symposium on Hligh-Performance Computer Architecture (HPCA), pages 42-4555 IEEE, 2023.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint M.Ki:20000.11929 2020.
[7] Yifan Gong, Geng Yuan, Zheng Zhan, Wei Niu, Zhengang Li, Pu Zhao, Yuxuan Cai, Sijia Liu, Bin Ren, Xue Lin, al. Automatic mapping of the best-suited dnn pruning schemes for real-time mobile acceleration. ACM Transactions on Design Automation of Electronic Systems (TODAES), 27(5):1-28 2022.
[8] Yifan Gong, Zheng Zhan, Pu Zhao, Yushu Wu, Chao Wu, Caiwen Ding, Weiwen Jiang, Minghai Qin, and Yanzhi Wang. All-in-one: A highly representative dun pruning framework for edge devices with dynamic power management. In Proceedings of the 41st IEEE/ACM International Conference Computer-Aided Design, pages 1-9, 2022.
[9] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint XrXiv: /2.002222.07222 2023.
[10] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2 2021.
[11] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] Yang He, Guoliang Kang. Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. International Joint Conference on Artificial Intelligence (IJCAI), 2018
[14] Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[15] Zhenglun Kong, Peiyan Dong. Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Bin Ren, Minghai Qin. Hao Tang, and Yanzhi Wang, Spvit: Enabling faster vision transformers via soft token pruning. ECCV, 2022.
[16] Zhenglun Kong, Haoyu Ma, Geng Yuan, Mengshu Sun, Yanyue Xie, Peiyan Dong, Xin Meng, Xuan Shen, Hao Tang. Minghai Qin, et al. Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 8360-8368. 2023.

---

[17] Tuanhui Li. Baoyuan Wu, Yujiu Yang. Yanbo Fan, Yong Zhang, and Wei Liu. Compressing convolutional neural networks via factorized convolutional filters. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[18] Youwei Liang, Chongjian GE, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. EVit: Expediting vision transformers via token reorganizations. In International Conference Learning Representations, 2022.
[19] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2988, 2017.
[20] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedinngs, Purt V 13, pages 740-755. Springer, 2014.
[21] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang. Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024.
[22] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning. Yue Cao, Zheng Zhang. Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12009-12019, 2022.
[23] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976-11 1986, 2022.
[24] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.13944 2022.
[25] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846-2888 2022.
[26] Bowen Pan, Rameswar Panda, Yifan Jiang. Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia-red?; Interpretability-nne awarre redundancy reduction for vision transformers. Advances in Neural Information Processing Systems, 34:24898-24911 2021.
[27] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937-13949 2021.
[28] Cedric Renggli, André Susano Pinto, Neil Houlsby, Basil Mustafa, Joan Puigcerver, and Carlos Riquelme. Learning to merge tokens vision transformers. arXiv preprint arXiv:2202.12015, 2022.
[29] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
[30] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning, pages 6105-6114 14. PMLR, 2019.
[31] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pages 10347-10357. PMLR, 2021.
Massa, Alexandre Sablayrolles, and
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
12

---

[33] Jue Wang. Wentao Zhu. Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6387-6397,
2023.
[34] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems (NeurIPS), pages 2074-2082, 2016.
[35] Yushu Wu, Yifan Gong, Pu Zhao, Yanyu Li, Zheng Zhan, Wei Niu, Hao Tang, Minghai Qin, Bin Ren, and Yanzhi Wang. Compiler-aware neural architecture search for on-mobile real-time super-resolution. In European Conference on Computer Vision, pages 92-111. Springer, 2022.
[36] Tete Xiao, Yingcheng Liu,, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 2018.
[37] Chenhongyi Yang. Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot J Crowley. Plainmamba: Improving non-hierarchical mamba in visual recognition. arXiv preprint 17X1v:2403.17695, 2024.
[38] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[39] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, pages 558 088-567,22 2021.
[40] Zheng Zhan, Yifan Gong, Pu Zhao, Geng Yuan, Wei Niu, Yushu Wu, Tianyun Zhang, Malith Jayaweera, David Kacli, Bin Ren, et al. Achieving on-mobile real-time super-resolution with neural architecture and pruning search. In Proceedings of the IEEE/CVF international conference computer vision, pages 4821-4831, 2021.
[41] Zheng Zhan, Zhenglun Kong. Yifan Gong, Yushu Wu, Zichong Meng, Hangyu Zheng. Xuan Shen, Stratis loannidis, Wei Niu, Pu Zhao, and Yanzhi Wang. Exploring token pruning in vision state space models. arXiv preprint arXiv:2409. 18962, 2024.
[42] Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang, Yifan Gong, Xuan Shen, Xue Lin, Pu Zhao, and Yanzhi Wang. Rethinking token reduction for state space models. arXiv preprint wrXiv:2410.14725 2024.
[43] Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang, Qi Dai, Qixiang Ye, and Qi Tian. Hivit: A simpler and more efficient design of hierarchical vision transformer. In The Eleventh International Conference on Learning Representations, 2022.
[44] Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, and Xue Lin. Pruning foundation models for high accuracy without retraining. arXiv preprint 27X55.2410.15567, 2024.
[45] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[46] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprini arXiv:2401.09411 2024
13

---

Appendix
A Pseudo-code Example
Algorithm 1: PRUNING-AWARE HIDDEN STATE ALIGNMENT
#example code of pruning aware hidden state alignment
def aruninggawwre__ha (state, position_map, x, dt, A, B, C, y-ptr): dA exp dt);
if position_map:
#remained token computation E
dB B dt; state state dA dB x; y-ptr &sum(state C);
else:
#pruned token computation as Eq.6
x.ptr++; dA;
return state
This is pseudo-code example of our pruning-aware hidden state alignment for demonstration.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We explain method and summarize the contribution in introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation included conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: The datasets and models we used is open- and we have provide our code in the footnote of page one.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https //ips..cc/pulli//uuui guwde//ddee/eee Coc//ooodeeesiseeeeeeeeee hwtpibbmsssiioooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details necessary to understand the results
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report average results of multiple runs in our experimental section. Our paper does not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We explain the computation resources in experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Research is conducted in the paper conform with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper is not highly related to societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks. Our work does not release a new model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: CC-BY 4.0, and referenced the works that we used to implement our code.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provided our code in the footnote of page
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "rSgqEiE1R7", "reviewer": "Reviewer_6myn", "summary": "This paper presents a token pruning method for vision state space models. The goal of this paper is to expand the token pruning methods for ViTs to recent SSM-based vision backbones. The authors observed that token pruning will change the computational characteristics of SSMs and lead to significant accuracy drop. To solve this problem, a hidden state alignment method is designed to explicitly adjust the hidden states of SSMs after token pruning. Experiments are conducted on on ImageNet, COCO and ADE20k to show the effectiveness of the method.", "questions": "- How to determine the pruning protocol mentioned in line 262-265? It seems that the models are not uniformly partitioned. So it is better to show more details behind the partition strategy used in the experiments. \n\n- How about the results with different pruning ratios? The paper only demonstrate one pruning ratio for each model and the ratios are different for different models. \n\nOverall, I find there are some interesting observations and results presented in the paper. But the experimental studies look a bit weak and the results are not that impressive/useful. Therefore, I would rate this paper as \"Borderline reject\".", "limitations": "The limitations have been discussed in Section 6.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well organized. The analyses in Section 3.2 clearly show the problem that the paper want to solve and help the readers understand the background easily. \n\n- The proposed alignment method and importance metric look reasonable, which are new since they are designed for  SSM-based models.", "weaknesses": "- There are still noticeable accuracy drop even after finetuning.  Although there is no previous work on token pruning in SSM-based vision models, many methods are proposed for ViT token pruning and achieved nearly no accuracy drop when reducing around 20% FLOPs. But I find the proposed method will lead to around 0.5 accuracy drop. The results are not that impressive.\n\n- The actual speed-up is not reported. One key advantage of ViT token pruning compared to traditional channel pruning is hardware friend structures after pruning to easily lead to actual speed-up. If the proposed method can also achieve similar results, the method will be much more useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the feedback from the reviewer. We address the raised questions as below.\n\n---\n\n**W1. Accuracy drop after finetuning.**\n\nWe appreciate the detailed feedback and would like to address the concern regarding the observed accuracy drop. Our work represents a novel advancement in enhancing the efficiency of SSM-based vision models through tailored token pruning techniques. Unlike ViTs, where existing token pruning methods have been extensively researched and optimized, SSM-based models present unique challenges. As demonstrated in Figure 2 and Table 1 of our main paper, **traditional token pruning methods for ViTs do not perform well on SSMs**.\n\nWe would like to highlight that, in comparison to existing designs, our method consistently delivers better accuracy across various SSM-based vision models and scales. For example, our approach surpasses the state-of-the-art (SOTA) token pruning technique for ViTs, EViT, by 3.8% on ViM-T, 4.0% on ViM-S, 2.4% on PlainMamba-L1, 2.7% on PlainMamba-L2, and 2.8% on PlainMamba-L3. This **significant performance improvement** (**over 2.4% across different models**) has been **recognized** and acknowledged by other reviewers (Reviewer M2Lk, TrFe, g8vG). Our method demonstrates superior performance compared to the ViT-based approaches, highlighting the **need for SSM-specific** pruning techniques.\n\nRegarding the 0.5% accuracy drop compared to dense computations on SSM-based vision models, one possible explanation is the inherent difference in computational complexity. SSM-scans achieve linear complexity, which poses greater challenges in attaining high computation reductions compared to the quadratic complexity of the attention mechanism in ViTs, which tends to have more redundant computations. This makes direct comparisons of FLOPs reductions between SSM and ViT models challenging.\n\nOn the other hand, the ViM has relatively small and efficient architecture. Smaller models generally have less redundancy, making it more challenging to achieve significant FLOPs reductions without impacting accuracy.\nOur experiments on the larger PlainMamba-L3 model can show a 40% FLOPs reduction with only a 0.6% accuracy drop. This result is better than the performance reported for the transformer model EViT-DeiT-B, which achieved a 35% FLOPs reduction with a 0.5% accuracy degradation (as reported in Table 8 of the EViT paper).\n\n---\n**W2. Speedup performance.**\n\nWe agree that actual speed-up is an important metric to demonstrate the effectiveness of token pruning schemes. Thus, we would like to clarify that we also include the speedup performance of our methods in Table 4 in the main paper. For instance, on PlainMamba-L3, our method with 8.44G FLOPs achieves 1.43$\\times$ throughput accelerations than the dense method. Here is the table in our paper,\n\n|Model| Method | FLOPs(G) | Top-1 Acc. (%) | Throughput |\n|-|-|-|-|-|\n| ViM-S |Dense| 5.10| 80.5 | 1×|\n| ViM-S |Prune w/o our alignment|3.57| 75.4| 1.30×|\n| ViM-S |Prune w/ our alignment| 3.60|78.8 | 1.27×|\n| PlainMamba-L3 |Dense| 14.40| 82.3 | 1×|\n| PlainMamba-L3 |Prune w/o our alignment|8.35| 79.3| 1.47×|\n| PlainMamba-L3 |Prune w/ our alignment| 8.44| 81.7 | 1.43×|\n\n---\n**Q1. How to determine the pruning protocol mentioned in line 262-265.** \n\nThank the reviewer for raising this valuable question. We will provide more details for comprehensive experimental results. Our partitioning strategy was determined through zero-shot experiments. This allowed us to identify the most effective pruning locations for different model architectures.\n\nThe varying number of layers in different models necessitated a flexible approach to partitioning. We found that a uniform partitioning strategy was suboptimal due to these architectural differences.\nExisting works on ViT pruning also have different partitions. For example, Table 3 in the DynamicViT [1] paper and Section 5 of SPViT [2] paper demonstrate that different scales of ViT models use different pruning locations.\n\nWhile the exact pruning locations vary between models, we maintained a consistent approach by **ensuring fixed intervals** and **same pruning rate** between pruning stages for each model. This also aligns with methodologies in existing works.\nIn the paper, we constantly set the pruning ratio to 0.8 for each location. The FLOPs reduction difference is due to the difference in model size related to the pruning location of layer index as stated in Line 262-265.\n\nWe also include more detailed information about Pruning ratio and pruned layers in Global rebuttal A1.\n\n[1] Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks. TPAMI\n\n[2] SPViT: Enabling Faster Vision Transformers via Soft Token Pruning, ECCV 2022\n\n---\n**Q2. Results with different pruning ratios.**\n\nIn the paper, we constantly set the pruning ratio to 0.8 for each location. The FLOPs reduction difference is due to the difference in model size related to the pruning location of layer index as stated in Line 262-265. We managed to finish evaluating more results with different pruning ratios on ViM-T and ViM-S, we will include more results in the revision.  \n\nTable A. Classification results of different ratios on ImageNet-1K.  \n|Methods | prune ratio | location | FLOPs(G) |  Top-1 Acc. (%)|\n|-|-|-|-|-|   \n| ViM-T | 0.9| [10,20]  |1.38 | 75.6  |\n| ViM-T | 0.8| [10,20]  |1.29 |  75.1|\n| ViM-T | 0.7| [10,20]  |1.21 |  74.5 |\n| ViM-S | 0.9 | [5,10,15,20]  | 4.21 | 79.4 |\n| ViM-S | 0.8 | [5,10,15,20]  |  3.60 | 78.8|\n| ViM-S | 0.7| [5,10,15,20]  | 2.90| 78.1 |\n\n---\nOverall, we take the **first step** toward accelerating vision SSM models with token-based pruning. We conduct a comprehensive analysis of SSM-based blocks to identify the **failure reason**, as well as provide more **insights** for the SSM scan mechanism in vision tasks, shedding lights on future research on SSM-based vision models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ooXLEdM359", "reviewer": "Reviewer_kiGk", "summary": "This paper explores the token pruning in vision state space models. It gives the observations that utilizing the token pruning techniques designed for ViTs leads to significant performance drop in SSMs. The main reason is that naive application disrupts the sequential token positions. To solve this, this paper introduces a pruning-aware hidden state alignment method and a token importance evaluation method to guide the token pruning while maintain better performance. Extensive experiments on ImageNet classification and object detection demonstrates the effectiveness of the proposed methods.", "questions": "The observation of this paper is interesting. However, I have major concerns about the novelty of the designed methods. I hope the author can solve my questions in the weaknesses part.", "limitations": "Limitations have been discussed in this paper. The efficiency is still limited by the baseline model architecture design.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths of the paper are:\n\n1、\tThis paper gives insights about the failure of traditional token-pruning method adopted in ViTs on SSMs. This observation can better guide the token-pruning in SSMs in the future research.\n\n2、\tExtensive ablation studies on all relevant components. Apart from the quantitative evaluations, its visual results are also clear. \n\n3、\tThe paper is well written and easy to read and follow.", "weaknesses": "This paper gives interesting observations about the token-pruning in vision state space models. However, I have some concerns about the analysis and the token-pruning methods designed for SSMs.\n\n1、\tFig. 1 illustrates the token computation difference between ViT and ViM. This introduces the large performance gap when adopting the traditional token-pruning techniques. Is there any more quantitative analysis result here to prove the impact of token computation patterns on performance of SSMs, such as randomly exchanging the order of adjacent tokens.\n\n2、\tWhat inspires the design of token importance in Eq(9). I can’t understand the reason of this design. Why adopting value 0 as the clipped threshold? Why does a larger S correspond to the most contextually relevant tokens? The result in Table 3 doesn’t indicate much superiority of the algorithm. In the traditional token-pruning method, the importance of a token should be represented by its’ impact on the loss value.\n\n3、\tThis paper also introduces the efficient implementation for the SSM scan. The FLOPs and Params are provided to demonstrates the computations reduction. I think we care more about the actual running latency. This information is necessary for us to evaluate the practicality of the methods. The improvement of Throughput in table 4 is not as obvious as that of FLOPs. What caused this?\n\n4、\tSome detailed information is missing. Such as what proportions are reduced in different layers? For the SSM model, does the ratio of token reduction in different layers need to follow any principles? I think this can make the article better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the feedback from the reviewer. We address the raised questions as below.\n\n---\n**W1. More quantitative analysis result here to prove the impact of token computation patterns on performance of SSMs.**\n\nWe would like to thank the reviewer for this valuable suggestion, we conduct additional experiments to randomly shuffle the token positions. For each pruning location of ViM (not all layers), we use the shuffle() function from the random package to randomly exchange the order of tokens. We observe the following performance. \nOn ViM-T, the accuracy dropped to 26.35% for zero shot, and 69.2%(-6.9) after finetune. \nOn ViM-S, the accuracy dropped to 25.69% for zero shot, and 74.1% (-6.4) after finetune. \nThe results indicate that the random token positions lead to much worse performance. \n\n---\n**W2. Reason behind our design of token importance.**\n\nCurrently, SSM is a novel model architecture and to our best knowledge, there are **no previous works** studying the importance of tokens in SSM models, especially for vision SSM models. Therefore, we try to tackle this problem by leveraging previous experience from Transformers. In Transformers, the softmax operation ensures that importance scores are always **positive**. We aimed to maintain a similar property in our approach. That’s why we choose general metrics like $\\ell_1$ norm, $\\ell_2$ norm and clip at 0 that could provide positive importance score values. \n\nMamba utilizes implicit attention within its layers. It processes information through SSM layers, allowing tokens to interact and influence each other. By the time tokens reach the output of the SSM layers, they have undergone multiple rounds of implicit interactions and transformations.  Mamba 2[1], has shown that the SSM block is **equivalent** to a form of linear attention. Therefore, the output contains the cumulative effect of these interactions, reflecting how each token has contributed to and been influenced by the overall context of the input. We then aggregate the clipped values across all channels of the output as our token importance score, as discussed in Line 228-231.\n\nFurthermore, the choice of 0 as the clipping threshold serves a role similar to **ReLU** activation: \n1. It introduces non-linearity, which can help in capturing more complex relationships. \n2. It improves the stability of backpropagation by preventing negative gradients. \n3. It could introduce sparsity which we think could benefit the token pruning fine-tuning process. Our experimental results also show that the choice of 0 as the clipping threshold could yield fairly good results.\n\nIn Figure 3 of our paper, we provide a visualization of the attention patterns derived from the output, which helps to illustrate the implicit attention mechanism in mamba.\n\nMoreover, deriving token importance directly from the model itself without additional token selection layers or algorithms can be beneficial for specific hardware optimization. \n\n---\n**Q1. Questions about throughput in table 4.**\n\nThank the reviewer for pointing this out, the FLOPs of ViM-S was a typo. We have double checked that the accuracy and throughput results in our paper are correct, sorry for the confusion. The \"1.21G\" typo corresponds to the FLOPS of ViM-T (ratio=0.7),  which has been presented in table of Global rebuttal **A1**. We have reported all the results for better clarity. The updated table is shown below:\n\n|Model| Method | FLOPs(G) |  Top-1 Acc. (%)| Throughput |\n|-|-|-|-|-|\n| ViM-S |Dense|\t5.10| 80.5 | 1×|\n| ViM-S |Prune w/o our alignment|3.57| 75.4|  1.30×|\n| ViM-S |Prune w/ our alignment| 3.60|78.8 | 1.27×|\n\nNote that the purpose of this table is to do an ablation study on the effectiveness of the alignment approach, using our token importance pruning method. \nFrom the results, we can see that our method could improve the throughput by 1.30× with around 30% FLOPS reduction.\n\n---\n**Q2. Detailed information about pruning ratio and pruned layers.**\n\nThank the reviewer for pointing out this missing part.\nThe pruning location is determined through zero-shot experiments. This allowed us to identify the most effective pruning locations for different model architectures.\n\nThe varying number of layers in different models necessitated a flexible approach to partitioning. We found that a uniform partitioning strategy was suboptimal due to these architectural differences.\n\nWe would like to clarify that existing works on ViT pruning also have different partitions. For example, Table 3 in DynamicViT paper (TPAMI version) and Section 5 of SPViT paper demonstrate that different scales of ViT models use different pruning locations.\nWhile the exact pruning locations vary between models, we maintained a **consistent** approach by ensuring **fixed intervals** between pruning stages for each model. This also aligns with methodologies in existing works.\n\nRegarding pruning ratio, we set the same pruning ratio (e.g. 0.8) for each location, with a progress pruning manner, pruning an additional $r$ tokens for each location. We also add additional results of different ratios (e.g. 0.7, 0.9) in **A1**.\n\nWe managed to finish evaluating more results with different pruning ratios on ViM-T and ViM-S, we will include more results in the revision.  \n\nTable A. Classification results of different ratios on ImageNet-1K.  \n|Methods | prune ratio | location | FLOPs(G) |  Top-1 Acc. (%)|\n|-|-|-|-|-|   \n| ViM-T | 0.9| [10,20]  |1.38 | 75.6  |\n| ViM-T | 0.8| [10,20]  |1.29 |  75.1|\n| ViM-T | 0.7| [10,20]  |1.21 |  74.5 |\n| ViM-S | 0.9 | [5,10,15,20]  | 4.21 | 79.4 |\n| ViM-S | 0.8 | [5,10,15,20]  |  3.60 | 78.8|\n| ViM-S | 0.7| [5,10,15,20]  | 2.90| 78.1 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L4OCcIgFE9", "reviewer": "Reviewer_M2Lk", "summary": "This paper introduces a token pruning method for vision SSMs to improve computational efficiency while maintaining performance. The authors identify that direct application of existing token pruning techniques designed for ViTs fails in SSMs due to disruption of sequential token positions. To address this, they propose a pruning-aware hidden state alignment method to stabilize the neighborhood of remaining tokens.", "questions": "- In Eq.6, how do you handle the accumulation of errors when computing hidden states for multiple consecutive pruned tokens? Does this lead to any instability in longer sequences?\n- How sensitive is the performance to the number of pruned tokens? Is there an upper limit to K_i before performance degrades significantly?\n- Is there any specific analysis about the pruning rate?", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is well-motivated, with a clear analysis of why existing token pruning techniques fail for SSMs.\n- Comprehensive experiments demonstrate efficiency gains across multiple tasks.\n- The structure and presentation of the paper are clear and well-organized.", "weaknesses": "See the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for recognizing the strengths of our papers and providing valuable feedback. We are happy to address the raised questions as below.\n\n---\n**Q1. How to handle the consecutive pruned tokens and longer sequences?**\n\nThank the reviewer for raising this valuable question. Here is a more detailed explanation about how our method handles the accumulation of errors when computing hidden states for multiple consecutive pruned tokens. Our method adopts $h^\\prime_{(q_{i})+1}= \\mathbf{\\overline{A}} h^\\prime_{(q_{i})} + \\mathbf{\\overline{B}} x_{(q_{i})+1} = \\mathbf{\\overline{A}} h^\\prime_{(q_{i})}$ where the token $x_{(q_{i})+1}$ is pruned, which leads to Equation (6).  As shown in Equation 6, if a token is pruned, we do not simply remove its corresponding hidden state during the scan as it leads to substantial accuracy degradation. Instead, its hidden state in the scan can be obtained by using the previous state with one step forward. In this way, the hidden states in SSM corresponding to pruned tokens are aligned with that of the original unpruned tokens to maintain the sequential positions of the original tokens without disrupting their neighbors. Based on the visualization results of Figure A1 in appendix, our method can prune consecutive tokens without additional designs while maintaining good performance.\n\nIn our paper, we provide Table 2 and Table A1 to show longer sequence results on object detection with PlainMamba. Where the input size is 1280×800 and 512×2048 respectively. Demonstrating our stability on longer sequences.\n\n---\n**Q2. How sensitive is the performance to the number of pruned tokens?**\n\nWe observe that the performance drops faster after accuracy reaching below 70%. Setting this 70% accuracy as a threshold, when input length is 197:\n\nFor ViM-T, there are 60 remaining  tokens after our token pruning, with around 31% FLOPs reduction.\nFor ViM-S, there are 36 remaining  tokens, which is around 56% reduction FLOPs.\nWe can observe that the larger the model, the more the tokens can be pruned. Also, the upper limit is also closely related to the pruning locations, thus  a better pruning location may  lead to more tokens to be pruned.\nWe will include this analysis in the revision.\n\n---\n**Q3. Is there any specific analysis about the pruning rate?**\n\nRegarding pruning ratio, we set the same pruning ratio (e.g. 0.8) for each location, with a progress pruning manner to prune additional tokens following the pruning ratio for each location. We discuss the pruning locations in Line 263-265. \nThe FLOPs reduction can vary due to different pruning locations associated with the number of layers.  \n\nWe managed to finish evaluating more results with different pruning ratios on ViM-T and ViM-S, we will include more results in the revision.  \n\nTable A. Classification results of different ratios on ImageNet-1K.  \n|Methods | prune ratio | location | FLOPs(G) |  Top-1 Acc. (%)|\n|-|-|-|-|-|   \n| ViM-T | 0.9| [10,20]  |1.38 | 75.6  |\n| ViM-T | 0.8| [10,20]  |1.29 |  75.1|\n| ViM-T | 0.7| [10,20]  |1.21 |  74.5 |\n| ViM-S | 0.9 | [5,10,15,20]  | 4.21 | 79.4 |\n| ViM-S | 0.8 | [5,10,15,20]  |  3.60 | 78.8|\n| ViM-S | 0.7| [5,10,15,20]  | 2.90| 78.1 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9xdtHyTKMe", "reviewer": "Reviewer_TrFe", "summary": "This manuscript aims to propose an effective pruning method for SSMs to achieve great trade-off of computation overhead and accuracy. The authors find that directly applying pruning strategies designed for transformer structure would greatly impair the performance of SSMs and give related analysis, that is SSMs take traversal path to consturct the interaction between different tokens and simple pruning would disrupt this connection. To address this challenge, the author first propose a strategy to build the alignment between the remaining tokens with the pruned tokens by keeping the indices and part calculation process of the pruned tokens. Based on the output of SSM layers, the author also propose a token selection strategy.", "questions": "* I have an additional explanation about why directly applying pruning strategy of EViT greatly impair the performance of SSMs. Because transformer takes non-local operation, all the tokens have knowledge of the overall context, so that removing some tokens would not cause huge information loss. However, for SSMs, the traversal interaction path makes them greatly rely on the continuous context and token pruning would lead to huge information loss. If space available, I am happy to discuss with the authors about that.", "limitations": "The authors have discussed the potential limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The authors give a clear presentation of the motivation and the proposed method. The structure of the manuscript is well-organized. \n* The motivation for designing specific pruning strategy for SSMs that take traversal interaction path is reasonable. The experiments of directly applying pruning method of EViT makes sense.\n* Experiment results show that the proposed method reduces comparable FLOPs with directly applying EViT pruning strategy yet achieves better performance.", "weaknesses": "* The authors only compare the proposed method with EViT. However, it seems that EViT is proposed in 2022. The authors should consider comparing with more recent pruning methods.\n* Lack of clear explanation and analysis of why the proposed pruning-aware hidden state alignment strategy is helpful. From Eq.6 we can see that the hidden state of h_{q+1} do not utilize any information of the pruned tokens, while the indices of evolution matrix A are changed. This part is confusing and I hope the author could give more descriptions about why only changing the indices brings such remakable improvements.\n* The proposed Token Pruning based on Importance Evaluation does not make sense. Why the output of SSM can directly reflect the token importance? Just because it shares the same length with input? I recommend the authors could give move explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for recognizing the strengths of our papers and providing valuable feedback. We are happy to address the raised questions as below.\n\n---\n**W1. More comprehensive comparison.**\n\nWe agree that a more comprehensive comparison would help the audience better understand the contribution of our work. We would also like to point out that EViT is still considered a strong baseline for token pruning methods. As for other token reduction methods, there have been recent advancements on Token merging as well as pruning + merge. Therefore, we implemented ToMe[1] as well as LTMP [2]  for our SSM-based models to provide a comprehensive comparison with state-of-the-art techniques. As demonstrated in the following table, our method can outperform all baselines with non-marginal improvements. \n\n|Methods | FLOPs(G) |  Top-1 Acc. (%)|\n|-|-|-|\n| ViM-T  |1.50| 76.1 |\n| ViM-T-ToMe |\t1.28|71.6|\n| ViM-T-LTMP |\t1.29|72.2|\n| ViM-T-EViT | 1.28|71.3|\n| ViM-T-Ours | 1.29 |75.1|\n\n[1] Token Merging: Your ViT But Faster, ICLR 2023\n\n[2] Learned Thresholds Token Merging and Pruning for Vision Transformers, TMLR 2023\n\n---\n**W2. Proposed pruning-aware hidden state alignment strategy explanation.**\n\nAs discussed in Section 3.2, for ViT, the hidden state of the pruned token is removed, i.e., $ h^\\prime_{(q_{i})+1} = 0$ if the token $x_{(q_{i})+1}$ is pruned, which leads to substantial accuracy drop  as this strategy dropping hidden states  within an SSM block disrupts the original token positions in the SSM scan.  Consequently, tokens that were not previously adjacent become neighbors during the scan in different directions or paths, leading to a distorted scan functionality and a significant accuracy degradation.  Especially considering that the tokens are actually image patches in visual tasks with semantic information, disrupting their positions during the scan brings great difficulties to understand their relationship and the overall semantics.   \n\nTo address this issue, our method adopts $ h^\\prime_{(q_{i})+1}  = \\mathbf{\\overline{A}} h^\\prime_{(q_{i})} + \\mathbf{\\overline{B}} x_{(q_{i})+1} = \\mathbf{\\overline{A}} h^\\prime_{(q_{i})}$ where the token $x_{(q_{i})+1}$ is pruned. It leads to Equation (6). As shown in Equation (6), if a token is pruned, we do not simply remove its corresponding hidden state during the scan as it leads to substantial accuracy degradation. Instead, its hidden state in the scan can be obtained by using the previous state with one step forward, i.e., $ h^\\prime_{(q_{i})+1}  = \\mathbf{\\overline{A}} h^\\prime_{(q_{i})}$. In this way, the hidden states in SSM corresponding to pruned tokens are aligned with that of the original unpruned tokens to maintain the sequential positions of the original tokens without disrupting their neighbours.  For the question “our method does not utilize any information of the pruned tokens”, our method is aware of the pruned tokens and can maintain the sequential information of the hidden states even if the tokens are pruned.\n\nWe would like to clarify that in SSM models, the SSM scan keep doing a sequential computation by multiplying the previous hidden state with evolution matrix A and add the next hidden to the current state, Therefore, for the concern that **SSM did not change the indices of evolution matrix A**, the changing number on the head of evolution matrix A is the **exponent** of A. We maintain the position information by maintaining the sequential positions of the original tokens without disrupting their sequential relation of evolution matrix A.\n\n---\n**W3. Explanation of our design of Importance Evaluation.**\n\nPlease refer to Global rebuttal **A2** for more detailed explanation.\n\nMamba utilizes implicit attention within its layers. It processes information through SSM layers, allowing tokens to interact and influence each other. By the time tokens reach the output of the SSM layers, they have undergone multiple rounds of implicit interactions and transformations. Mamba 2[1], has shown that the SSM block is equivalent to a form of linear attention. Therefore, the output contains the cumulative effect of these interactions, reflecting how each token has contributed to and been influenced by the overall context of the input. We then aggregate the clipped values across all channels of the output as our token importance score, as discussed in Line 228-231.\n\nIn Figure 3 of our paper, we provide a visualization of the attention patterns derived from the output, which helps to illustrate the implicit attention mechanism in mamba.\nMoreover, deriving token importance directly from the model itself without additional token selection layers or algorithms can be beneficial for specific hardware optimization. \n\n[1] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality.\n\n---\n**Q1. Insight discussion.**\n\nWe would like to first thank the reviewer for such an insightful discussion. In “Reason for the failure” discussion of section 3.2 of our paper, we try to find out the reason why token pruning can work on ViT while failing on ViM. An overall explanation could be that the non-local operation of transformers is due to the quadratic operation of self-attention’s token wise multiplication. This enables the model to gain comprehensive knowledge of the overall context.\n\nMoreover, our explanation on SSM aligns with the reviewer's thought on vision state space models that the SSM relies on sequential information (continuous context), directly removing the tokens could lead to huge information loss.\n\nOur results in Table 1 show that plainmamba could achieve larger FLOPs reduction (-41.4%) while maintaining good performance (-0.6%). This could be because plainmamba chose a continuous 2D scanning which scans from 4 directions, providing more overall context and robustness compared with 1D scanning in ViM, which lead to higher token pruning ratios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zre8xw6ig9", "reviewer": "Reviewer_g8vG", "summary": "The paper  introduces a pruning-aware hidden state alignment method, stabilizing token neighborhoods and maintaining model performance during token pruning. It also proposes an adapted token importance evaluation method tailored for SSM-based models, effectively guiding the pruning process. Extensive experiments demonstrate the method's efficacy, achieving significant computation reduction with minimal performance impact. Notably, the approach achieves 81.7% accuracy on ImageNet with a 41.6% reduction in FLOPs for pruned PlainMamba-L3, highlighting its practical acceleration benefits and effectiveness in maintaining high accuracy in vision tasks.", "questions": "Can the proposed method generalize to the hierarchical variants like VMamba?", "limitations": "The authors adequately addressed the limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- the proposed method ensures the stability of token neighborhoods, maintaining model performance even after pruning.\n\n- The importance evaluation is tailored specifically for SSM-based models, this evaluation method effectively guides the token pruning process.\n\n- The approach achieves significant computation reduction with minimal impact on performance, exemplified by 81.7% accuracy on ImageNet with a 41.6% reduction in FLOPs for pruned PlainMamba-L3.\n\n- The paper provides extensive visualization to further validate the effectiveness", "weaknesses": "- The proposed method is limited to plain, non-hierarchical SSM-based models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for recognizing the strengths of our papers and providing valuable feedback. We are happy to address the raised questions as below.\n\n---\n**W1. The proposed method is limited to plain, non-hierarchical SSM-based models.**\n\nWe would like to kindly point out that ViM has proved to be the first fast and strong baseline for vision state space models and its design has been widely adopted by later vision state space models. That’s why we choose the ViM and its variants as our base models.\n\nMoreover, all the token reduction methods face similar difficulties when adapting to hierarchical model architecture due to their conflicts with the nature of hierarchical model architecture. However, with analysis and specific designs, the efficiency can still be achieved on hierarchical model architecture with our method. We discuss more design details in the following Response to **Q1**  to show how we can tackle this problem with our method. \n\n---\n**Q1. Can the proposed method generalize to the hierarchical variants like VMamba?**\n\nThanks for pointing this out, following the settings from previous token pruning work on Swin[1], we apply the sparsification operations at **stage 3** of Vmamba which contributes most of the complexity.  This is because the Layer number inside each stage of  VMamba-T is [2,2,8,2]  and  VMamba-S is [2,2,15,2], with major computations in Stage 3.\n\n|Methods | prune ratio at stage 3 | FLOPs(G) |  Top-1 Acc. (%)|\n|-|-|-|-|\n| VMamba-T| - |4.9|82.6|\n| VMamba-T-prune| 0.8| 4.0|82.1 |\n| VMamba-S | - |8.7|83.6|\n| VMamba-S-prune| 0.8|6.1 |83.2 |\n\nThe toke pruning process cannot transfer across stages due to the downsampling process between stages. Therefore, we need to perform padding to restore the sequence at the end of the stage. But significant speedup inside stages can be achieved.\n\n[1] Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks. TPAMI", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude to the reviewers for their positive comments and constructive feedback on the paper. We sincerely appreciate the reviewers for acknowledging **our motivation** is clear and reasonable (Reviewer TrFe), with interesting observations that give insights to better guide the token-pruning in SSMs in the future research (Reviewer 6myn, kiGk), **our method** is new, resonable (Reviewer 6myn), well-motivated (Reviewer M2Lk), and effective with good performance (Reviewer g8vG), **our experiments and results** are extensive, comprehensive, and impressive with both quantitive and visual evaluations (Reviewer kiGk, M2Lk, TrFe, g8vG),  and **our paper** is well-organized, well-written, and easy to read and follow (Reviewer 6myn, kiGk, M2Lk, TrFe).

---

**A1. Detailed information about Pruning ratio and pruned layers.** 

In the paper, we constantly set the pruning ratio to 0.8 for each location. The flops reduction difference is due to the difference in model size related to the pruning location of layer index as stated in Line 262-265. We will include more results in the revision. Here are the results about pruning ratio and pruned layers,

Table A. Classification results of different ratios on ImageNet-1K.  
|Methods | prune ratio | location | FLOPs(G) |  Top-1 Acc. (%)|
|-|-|-|-|-|   
| ViM-T | 0.9| [10,20]  |1.38 | 75.6  |
| ViM-T | 0.8| [10,20]  |1.29 |  75.1|
| ViM-T | 0.7| [10,20]  |1.21 |  74.5 |
| ViM-S | 0.9 | [5,10,15,20]  | 4.21 | 79.4 |
| ViM-S | 0.8 | [5,10,15,20]  |  3.60 | 78.8|
| ViM-S | 0.7| [5,10,15,20]  | 2.90| 78.1 |


**A2. Explanation of our design of Importance Evaluation.**

Currently, SSM is a novel model architecture and to our best knowledge, there are **no previous works** studying the importance of tokens in SSM models, especially for vision SSM models. Therefore, we try to tackle this problem by leveraging previous experience from Transformers. In Transformers, the softmax operation ensures that importance scores are always **positive**. We aimed to maintain a similar property in our approach. That’s why we choose general metrics like $\ell_1$ norm, $\ell_2$ norm and clip at 0 that could provide positive importance score values. 

Mamba utilizes implicit attention within its layers. It processes information through SSM layers, allowing tokens to interact and influence each other. By the time tokens reach the output of the SSM layers, they have undergone multiple rounds of implicit interactions and transformations.  Mamba 2[1], has shown that the SSM block is **equivalent** to a form of linear attention. Therefore, the output contains the cumulative effect of these interactions, reflecting how each token has contributed to and been influenced by the overall context of the input. We then aggregate the clipped values across all channels of the output as our token importance score, as discussed in Line 228-231.

Furthermore, the choice of 0 as the clipping threshold serves a role similar to **ReLU** activation: 
1. It introduces non-linearity, which can help in capturing more complex relationships. 
2. It improves the stability of backpropagation by preventing negative gradients. 
3. It could introduce sparsity which we think could benefit the token pruning fine-tuning process. Our experimental results also show that the choice of 0 as the clipping threshold could yield fairly good results.

In Figure 3 of our paper, we provide a visualization of the attention patterns derived from the output, which helps to illustrate the implicit attention mechanism in mamba.

Moreover, deriving token importance directly from the model itself without additional token selection layers or algorithms can be beneficial for specific hardware optimization. 

[1] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality,

Meta-review: This paper proposes a novel technique to prune State-State Models (SSMs) tokens for vision tasks based on insights specific to SSMs. Experiments are conducted on ImageNet-1K, COCO 2017, and ADE20K datasets.

Upon the rebuttal and the authors-reviewers discussion, the reviews were slightly diverging, and three were borderline: 1x borderline reject, 2x borderline accept, and 1x weak accept.

The AC summarizes these main aspects:

(1) The paper is clear.

(2) The proposed model compression by token pruning for SSMs-based vision models is novel. This virtue mainly stems from considering the recent hot trend of SSMs. Although token pruning is well known, the paper provides insights into applying the technique to SSMs, which are informative and not trivial.

(3) The performance improvement is satisfactory. The majority of reviewers and the AC reckon the compromise between computation reduction and accuracy is analyzed thoroughly, demonstrating the proposed technique's quality. Two reviewers have demanded comparisons against more baselines other than the comparison with EViT in the submitted paper. The authors could not provide a comparison with AViT [cvpr'22] and STViT [cvpr'23], which one reviewer suggested, but they provided a comparison with ToMe (ICLR'23) and LTMP (TMLR'23). One of the reviewers deems this is sufficient, and the AC seconds this statement but encourages the authors to complete the performance evaluation and consider those baselines if they can manage before the camera-ready submission deadline.

Overall, the AC stands with the favorable opinion of most reviewers and recommends this paper for acceptance.
Decision: Accept (poster)