Pruning-Robust Mamba with Asymmetric Multi-Scale Scanning Paths
Authors: Jindi Lv
Yuhao Zhou
Mingjia Shi
Zhiyuan Liang
Panpan Zhang
Xiaojiang Peng
Wangbo Zhao
Zheng Zhu
Jiancheng Lv
Qing Ye
Kai Wang
Venue: NeurIPS 2025 poster ()
Keywords: State Space Model
Mamba
Token Reduction
Efficient AI
Primary area: deep_learning

Abstract: Mamba has proven efficient for long-sequence modeling in vision tasks. However, when token reduction techniques are applied to improve efficiency, Mamba-based models exhibit drastic performance degradation compared to Vision Transformers (ViTs).  This decline is potentially attributed to Mamba's chain-like scanning mechanism, which we hypothesize not only induces cascading losses in token connectivity but also limits the diversity of spatial receptive fields.
In this paper, we propose Asymmetric Multi-scale Vision Mamba (AMVim), a novel architecture designed to enhance pruning robustness. AMVim employs a dual-path structure, integrating a window-aware scanning mechanism into one path while retaining sequential scanning in the other. This asymmetry design promotes token connection diversity and enables multi-scale information flow, reinforcing spatial awareness.
Empirical results demonstrate that AMVim achieves state-of-the-art pruning robustness. During token reduction, AMVim-T achieves a substantial 34\% improvement in training-free accuracy with identical model sizes and FLOPs. Meanwhile, AMVim-S exhibits only a 1.5\% accuracy drop, performing comparably to ViT. Notably, AMVim also delivers superior performance during pruning-free settings, further validating its architectural advantages.

Pruning-Robust Mamba with Asymmetric Multi-Scale Scanning Paths
Jindi Lv
Yuhao Zhou 18
Mingjia Shi
Zhiyuan Liang?
Panpan Zhang
Xiaojiang Peng?
Wangbo Zhao
Zheng Zhu*
Jiancheng Lv
Qing Ye
Kai Wang +GigaAl
Sichuan University
University of Virginia
National University of Singapore
100
VIT (Transformer)
Vim (SSM)
base model
90
merge
70
multi-scale
Accuracy DN
80
75.5
75,2
70
50
break 2
break
boost (symJ
60
1 2
40
asym.
40
38.7
VITT
Vim-T
Connection Survival Rate (%)
(a) Accuracy of token merg-
(b) An illustrative example showing the impact ing: VIT-T (Transformer) of token connection disruption in VIT (Trans- Vim-T (SSM).
(c) Correlation between token connection survival rate and accuracy.
former) versus Vim (SSM).
Figure I: Comparative analysis of token reduction. In Figure (b), blue solid lines denote direct token connections, whereas black dotted lines signify potential connections.
Abstract
Mamba has proven efficient for long-sequence modeling in vision tasks. However, when token reduction techniques are applied to improve efficiency, Mamba-base models exhibit drastic performance degradation compared to Vision Transform- ers (ViTs). This decline is potentially attributed to Mamba's 's chain-like scanning mechanism, which we hypothesize not only induces cascading losses in token connectivity but also limits the diversity of spatial receptive fields. In this paper, we propose Asymmetric Multi-scale Vision Mamba (AMVim), a novel architecture designed to enhance pruning robustness. AMVim employs dual-path structure, integrating a window-aware scanning mechanism into one path while retaining sequential scanning in the other. This asymmetry design promotes token con- nection diversity and enables multi-scale information flow, reinforcing spatial awareness. Empirical results demonstrate that AMVim achieves state-of-the-us pruning robustness. During token reduction, AMVim-T achieves a substantial 34% improvement training-free free accuracy with identical model sizes and FLOPs. Meanwhile, AMVim-S exhibits only 1.5% accuracy drop, performing comparably ViT. Notably, AMVim also delivers superior performance during pruning-fred -ree settings, further validating its architectural advantages.
Introduction
In recent years, the Mamba architecture, built upon state space models (SSMs), has emerged transformative paradigm for efficiently modeling long-range dependencies in vision tasks [|
By introducing an innovative chain-like scanning mechanism, Mamba 66 & 10 II] successfully reduces the computational complexity from the quadratic demands of Transformers 12 linear scale. Alongside these advancements, token reduction techniques (eg., pruning [5
to
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and merging [8 [9] for Mamba have garnered increasing attention as promising avenues toward
further optimization.
Nevertheless, Mamba exhibits significantly greater performance degradation during token reduction compared to Transformers (eg., ViT [3], shown in Figure Ta This discrepancy arises from Transformers using self-attention to establish fully connected token relationships, while Mamba processes tokens sequentially along chain-like scanning paths. This chain-based structure makes Mamba susceptible to cascading information loss during token reduction. For clarity, an illustrative example is provided in Figure lb
Recent Mamba variants [20] 21 22]. such as Vim [6], have attempted to mitigate this limitation through dual-path scanning strategies that combine forward and reverse sequential paths. While these symmetric designs enhance sequence modeling capabilities and improve baseline performance, they remain ineffective for token reduction. The inherent symmetry of dual--path scanning confines token relationships to the same chain-like structure, failing to address the systemic vulnerability to large-scale connection disruption during pruning.
Inspired by these observations, we hypothesize that minimizing connection disruption during token reduction can mitigate performance drop. To validate this, we introduce asymmetric scanning into dual-path Mamba. As illustrated in Figure Ic asymmetric scanning paths reduce accuracy degradation from 38% (with symmetric paths) to 21% at the same pruning ratio. This suggests that diversifying chain-like dependencies effectively mitigate pruning- induuee performance decline.
To further elucidate this phenomenon, we quantify the token connection survival rate across dif- ferent dual-path strategies during token reduction. Figure reveals a strong positive correlation between connection survival rates and accuracy, with asymmetric paths exhibiting superior robust- ness. This confirms our hypothesis; enhancing token connection survival rates via asymmetric path diversification is pivotal for improving Mamba's pruning resilience.
In this work, we propose AMVim, a novel Asymmetric Multi-scale Vision Mamba for pruning ro- bustness. To enhance space information diversity, we integrate a window- scanning mechanism into one path. By adopting a different scanning direction within windows compared to the main path, we construct multi-leve asymmetric paths. This multi- multi-dimensional information flow enables each token perceive neighborhood information from multiple perspectives. Furthermore, the integration of window-based scanning with the main path creates multi-level complementary design, allowing for interactions between global context and local dependencies.
Empirically, as shown in Figure Icl our method results in just a 3% accuracy drop during token reduction (with the blue dotted line representing the baseline accuracy of 76.1 1%) on ImageNet- achieving a 34% improvement over Vim. This highlights that the multi-scale scanning mechanism enhances the spatial awareness of SSMs, significantly reducing token sensitivity local variations.
We highlight the main contributions of this paper below:
We hypothesize token connection survival rate is critical factor in performance degradation and propose asymmetric scanning paths to effectively mitigate this issue.
We design multi-scale asymmetric scanning mechanism that balances global and local spatial information while preserving the benefits of asymmetric paths.
Our method achieves state-of-the-art pruning resilience, outperforming Vim-T by 34% on ImageNet- K under identical parameters and FLOPs.
2
Related Work
2.1
State Space Models
SSMs
27 were initially proposed in the NLP community to model long-range dependencies text. Recently, SSM variants have emerged as effective alternatives to ViTs [3
23
24
25
26
12 28 29
30, reducing computational complexity in visual tasks from quadratic to linear time. S4ND 31 is the first work to apply SSMs to visual tasks, extended the S4 (32 model by normalizing
the parameters to diagonal structure. However, this approach struggled to capture image information
an input-dependent manner. In response, Vim G was proposed, introducing bidirectional scanning
to enhance spatial awareness in vision tasks. Building upon this, PlainMamba (33) introduced

---

continuous 2D scanning, which improves spatial continuity by maintaining adjacency among tokens within the scanning sequence. Moreover. VMamba 7 proposed an SS2D scanning mechanism that enables comprehensive scanning across four distinct paths.
35 have identified limitations in Mamba's chain-like scanning structure, particularly in capturing local spatial dependencies. To address this, Shi al. (35] introduced multi-scale 2D scanning technique based on VMamba, combining original and downsampled feature maps to alleviate the long-range forgetting issue. Similarly, LocalMambal (34] proposed window-band based scanning mechanism that dynamically selects search paths at each layer to capture local dependencies. In contrast, we introduce multi-scale asymmetric scanning mechanism. By improving both direct token connection complementarity and multi-scale information synergy, our method enhances the spatial perception capability of SSM.
Despite significant advancements, recent studies 24
2.2
Token Reduction
Token reduction aims to enhance computational efficiency by dynamically removing consolidating redundant tokens during inference. These methods are typically classified into token pruning [15
19
16
and token merging [18
36
37
38
39] Token pruning identifies and eliminates low-
employs the Gumbel-Softmax strategy to prune less informative tokens, while EViT [99 relies on the attentiveness of the [CLS] token determine key tokens. In contrast, token merging combines semantically similar tokens, as demonstrated by ToMe 18] training-free free approach that merges tokens via bipartite matching. However, the underlying architectural differences between ViTs and Mambas present unique challenges when applying these techniques to Mambas. First, most token pruning methods are designed for ViTs and rely on self self-attention scores, which Mamba lacks, making direct transfer infeasible. Second, token merging in Mamba leads to significant performance degradation due its chain-like scanning mechanism, which enforces rigid sequential dependencies between tokens.
importance tokens. For example, Dynamic VTT 15
Recent work has sought address these challenges. For instance, Zhan al. 40 proposed pruning-awar awware hidden state alignment method to selectively skip tokens in Mamba. Their follow- up work [4I] introduced hybrid metries combining token importance and similarity for pruning. These approaches focus on designing specialized pruning strategies for Mamba. In contrast, our approach aims to strengthen the intrinsic robustness of Mamba's architecture for token reduction. By mitigating chain dependency vulnerabilities, our method enables seamless integration of existing token merging techniques while further improving performance, offering significant advantage for practical deployment.
3 Method
3.1 Preliminaries
SSMs map a 1D input sequence æxt) E R to output sequence y(t E IR through an implicit latent state h(t) E RRW, governed by linear ordinary differential equations (ODEs):
h'(t) Ah(t) + Bx((),
y(t)
-Ch(t).
(1)
projects inputs to the state space, and Ce E R Ix maps the state to outputs. A defining feature of SSMs their chain-like scamning path. which processes inputs sequentially to achieve linear computational complexity. This sequential dependency can be formalized in the discretized recurrence:
where A E NNN governs state transitions, B E Nx1
hr Ah,- Bzi,
= Ch+-
(2)
where A A CAA and B (AA) - -)--BBBB AB are derived from the continuous-timg parameters via zero-order hold (ZOH) discretization. The chain-like structure ensures that each token axt) interacts directly with its immediate predecessor z(t 1), propagating information sequentially through the state h().
Mamba enhances SSMs with input-dependent selectivity (S6), dynamically adjusting parameters B, C, A based on (f)):
B, = (inearp(z.))
C, = Lzrearg(zz), 4, = Softplus(Linear (xx))
(3)
3

---

multi-scale scan
encoder
cross-merge
Xesa
Hlrr
5
Comro
SSM
16
16
16
input patches
:6
6
Convte
SBM
I0
16
output patches
Figure 3: Illustration of the Window- based Multi-Scale State Space (WMS3) block. Input patches are processed along two scanning paths at different scales (multi-scale scan), with each sequence independently encoded (encoder). The outputs are then restored to sequential order and merged to form a 2D feature map as the final result (cross-merge).
While this improves flexibility, the underlying chain-like scanning path remains core component. Importantly, this chain structure design introduces critical limitation: the model relies on local neighbor dependencies for information propagation, which amplifies sensitivity to token removal.
3.2 Multi-Scale 2D Selective Scanning
The sequential scanning operation in S6 works well for time-series data in NLP tasks but struggles with non-causal visual data, which inherently non-sequee sequennaltlll and spa- tially complex. To address this, Vim [6] introduces bidi- rectional symmetric scanning paths (Figure 2a) to enhance spatial context. Although this design improves spatial con- text modeling, fails to resolve the performance collapse during token reduction.
We attribute this limitation to the chain-like scanning mechanism underlying Mamba, which enforces a rigid neighbor dependency as depicted Equation [ mit- igate this issue, we propose simple yet effective solution: asymmetric scanning paths (Figure 2b). By diversifying the scanning directions, asymmetric paths enhance the complementarity of token connections, as shown in Figure
(a) Vim
(b) Asym.
(c) Ours
Figure Comparison dual-path scan- ning strategies. The solid dot represents the starting position.
Ic leading to significant boost in pruning robustness. However, achieving full token connection complementarity through scanning paths alone remains challenging under computational constraints.
Recent studies [34] (35] emphasize the critical role of local spatial awareness in enhancing Mamba's s robustness. Inspired by this insight, we propose a window-base multi-scale selective scanning mechanism. As shown in Figure[2c this design integrates window-aware scanning strategy into one path while preserving the global scanning directions of Vim. Specifically, the tokens are partitioned into non-overlapping windows, where tokens are scanned vertically within windows and horizontally globally. This orthogonal design ensures asymmetric dependencies, minimizing redundant token connections while maximizing connection complementarity. Moreover, by harmonizing global structures and local textures, it enables more comprehensive spatial understanding.
3.3 Overall Model Architecture
In this work, we extend Vim 66] by introducing the Window-based Multi-Scale State Space (WMS3) block. As shown in Figure 3, WMS3 operates through three sequential stages: multi-scale scanning, encoding, and cross- Given input token sequence X E R Lxx where is the sequence length and D is the feature dimension, WMS3 first reorders along two distinct traversal paths: the sequential path and the window-aware path.
Fach reordered sequence is then independently processed by a dedicated encoder block, which integrates ID convolutional layer and S6 module. Formally, for sequence X, along path

---

seeuential, window]; J, the encoder block computes:
Hp , = S6(Conv ID(X,))
(4)
where Conv (-) enhances local feature interactions, and S6(.) models long-range dependencies via selective state transitions.
Finally, the outputs from both paths are restored to their original spatial order and merged via cross-merge operation:
Linear( H sequential H Window):
where Linear(-) denotes a linear projection layer. Beyond the WMS3 block, our architecture retains the core design of Vim [6], including the placement of the [CLS] token at the sequence center. This multi-scale design expands spatial perception granularity without additional computational overhead, significantly improving robustness to token reduction while maintaining efficiency.
4 Experiment
4.1 Datasets and Settings
We evaluate AMVim on the ImageNet-IK dataset, which includes 1,000 object classes, 1.28 million training images, and 50,000 validation images. Images are augmented and resized to x24 224 for evaluation. This study focuses on the ImageNet-IK classification task, and we report top-1 validation accuracy. All experiments are conducted with 4 x NVIDIA L40S GPUs.
The window size of AMVim is set 3x3. AMVim is fine-tuned for 150 epochs with AdamW optimization, initialized using the publicly available weights of Vim [6]. A batch size of 128 is used with two-step gradient accumulation, resulting in an effective total batch size of 1,024. Additional training details are listed in Table Appendix.
During token reduction, employ the ToMe technique 18] by default. To ensure fair comparison, token merging is applied to the even-indexed blocks, covering a total of 12 layers. In each layer, [5, 8, 11, 14] tokens are pruned, corresponding to reduction ratios of [0.17, 0.27, 0.36, 0.46).
4.2 Pruning Robustness Analysis
Comparison with SOTA methods.
To validate the advancement of the proposed ar- chitecture, we compare AMVim with two state-
Table Performance comparison with token prun- ing methods designed for Mamba on ImageNet- IK classification. AMVim achieves the highest top-1 accuracy across different model scales while main- taining comparable FLOPs.
of-the-art token pruning techniques specifically
designed for Mamba: Token Recognition [19] and Hidden State Alignment [40]. For fair comparison, these two methods are evaluated on
Vim [6] with a dual-path structure. Additionally, we adhered to their fine-tuning protocols [40) during token reduction and reported the final fine-tuned accuracy.
reduction ratio
top-1 acc(
FL.PPs (G)
method
Tiny
Small
Tiay
Small
Vim (baseline)
0.00
76.1
80.5
1.45
5.08
Token Recognition
1v
0.17
71.3
74.8
1.28
3.57
Alignment
75.1
78.8
1.29
The results, presented in Table I show that AMVim consistently achieves the highest ac- curacy across various model scales while maintaining comparable FLOPP. Specifically, AMVim-T outperforms Token Recognition by 3.7% in accuracy, while AMVim-S achieves a 4.7% improvement. Furthermore, AMVimmS exhibits only 1% accuracy drop while reducing FLOPs by one-quarter, significantly surpassing both Token Recognition and Hidden State Alignment. These results highlight that instead of designing specialized token pruning methods for Mamba, addressing its architectural fragility offers a more promising solution.
AMVim: ToMe (finetune)
0.17
75.3
79.5
1.27
3.60
Robustness across various reduction ratios. Figure 4a compares the top-1 accuracy of our method with recent state-of-the-art vision Mamba models across various token reduction ratios. reduction ratio increases, all methods exhibit a predictable decline in performance due to increased information loss. Our method consistently demonstrates superior robustness, outperforming Vim 6 by approximately 30% across all reduction ratios. LocalVim [34] shows improved robustness over
5

---

Accuracy (56
76.2
76.3
72.8
60.4
70.3
66.7
61.3
Throughput (img/l)
2631
2962
3291
3063
3279
3417
76.1
2751
50.9
2497
2453
43.3
1549
Vim
LocalVim
38.7
41.6
Ours
37.6
36.2
34.2
955
.-
Vim
LocalVim
1066
Ours
1220
0.17
0.27
0.36
0.46
0.17
0.27
0.36
0.46
Reduction
Reduction ratio
(a) Accuracy.
(b) Throughput.
Figure 4: Comparison with state-of-the-art vision Mamba methods in terms of top-1 accuracy and throughput across various token reduction ratios. Our method demonstrates superior robustness against Vim and LocalVim, while achieving comparable throughput with Vim and twice that of Local Vim. This highlights that our method strikes good balance between efficiency and robustness.
Vim by incorporating local modeling. However, it still lags significantly behind our method, especially as the reduction ratio increases. Notably, even nearly threefold higher reduction ratio (0.46 vs. 0.17), our method still achieves higher accuracy than both Vim and LocalVim.
Figure 4b presents the computational throughput across methods. Our approach achieves throughput comparable to Vim, with a marginal deficit of 200 img/s attributable to directional change operations. In contrast, it delivers approximately 2x higher throughput than LocalVim, which incurs heavier computational overhead from per-layer directional changes. These results collectively underscore our method's optimal trade-off between performance robustness and computational efficiency.
Robustness on various pruning methods.
To further validate the pruning robustness of AMVim, we introduce random token pruning, method that randomly discards tokens without relying on any pruning criteria (e.g., similarity or importance). Table compares the top-1 accuracy of Vim and AMVim under both token merging and random pruning conditions. The results demonstrate that AMVim consistently outperforms Vim by approximately 30% across all scenarios, irrespective of the pruning method employed. This significant performance gap highlights the exceptional robustness AMVim''s 's architecture, which consistently delivers strong performance under any pruning method, solidifying its design superiority over Vim.
Notably, AMVim exhibits inferior performance under random token pruning compared to token merging. This observation expected, as token merging leverages similarity as a guiding metric, whereas random pruning lacks such guidance. In contrast, at reduction rates of 0.17 and 0.27, random pruning yields better performance than token merging on Vim. This counterintuitive result suggests
Table 2: Random pruning vs. Token merging: top-1 accuracy (%) under various token reduc- tion techniques. A denotes the performance difference between Vim and AMVim. AMVim achieves roughly 30% higher accuracy than Vim across both token merging and token pruning methods.
Table 3: Performance comparison between Vim and AMVim on ImageNet- IK. "ToMe" indicates token reduction applied via ToMe [8 for each method, with training-free accuracy reported. A represents the performance gap between Vim and AMVim. AMVim consistently outperforms Vim across both pruned and non-pruned settings.
operation
reduction
Vim-T
AMVim-T
method
image
#param
FLOPS
top-1 acc.
ratio
(%)
(%)
(%)
size
(G)
(%)
0.17
42.7
72.4
29.71
Vim-T
224
1.5
76,
0.27
45.9
69.5
23.61
Vim-S
224
80.5
pruning
0.36
36.6
65.2
28.61
ToMe-Vim-TT
224
1.3
38.7
0.46
22.5
58.6
32.11
ToMe-Vim-S
224
78.4
0.17
38.7
72.8
34.11
AMVim-T
224
1.5
76.3
0.21
28.71
AMVim-S
224
80.7
merging
0.36
37.6
66.7
29.11
ToMe-AMVin-I
224
1.3
72.8
34.11
27.11
ToMo-AMVim-S
224
79.2
0.81

---

(a) D1
(b) D3
(c) D5
(d) D2
(e) D4
(f) D6
Figure 5: Illustration the scanning directions commonly used in vision Mamba. The solid dot represents the starting position.
that Vim's 's performance degradation not inherently tied to the pruning method but rather stems from its architectural vulnerability, rendering the guiding metric ineffective. These findings collectively highlight the resilience of AMVim's architecture while exposing fundamental limitations in Vim 's.
4.3 Image Classification
Table compares the performance of Vim and AMVim on the ImageNet- 1 classification task with an image size of 224x224, The results show that AMVim consistently achieves higher top-1 accuracy than Vim across various model scales while maintaining the same parameter count and FLOPs. This improvement can be attributed to AMVim's 's enhanced spatial awareness capability, which effectively achieves the learning of both local dependency and global context.
With token reduction applied (reduction ratio 0.16), AMVim- exhibits a training- -fre accuracy drop of only 3.8%, while A.VimmS shows an even smaller drop of 1.5%, approaching the performance of VIT. Compared Vim, AMVim-T delivers a substantial 34.1% improvement pruned accuracy. and AMVim-S shows 0.8% increase. These results collectively demonstrate that AMVim not only delivers superior robustness token reduction but also enhances model expressive capability through its innovative design. AMVim strikes a balance between performance and stability, emerging as powerful solution for vision tasks.
4.4 Semantic Segmentation
We evaluate AMVim on the downstream se- mantic segmentation task using the ADE20K dataset [42, with results summarized in Ta- ble 4 When integrated into the UperNet framework 43 AMVim consistently outper- forms Vim across both full- prlciieeeiioon and token- reduced settings. In the non-pruned setting, AMVim achieves improvements of +0.2% and +0.1 1 % mIoU over Vim-Ti and Vim-S, respec- tively, demonstrating enhanced representational capacity due to its multi-scale scanning mecha- nism.
Table 4: Semantic segmentation performance on ADE20K [42) val set with UperNet. "ToMe" de- notes token merging applied via ToMe [8] training-free free manner. A indicates the improvement of AMVim over the Vim baseline. AMVim consis- tently outperforms Vim, and when combined with ToMe, achieves significantly higher mloU under heavy pruning.
image
#param
val mloU
A
method
backbane
(M)
(%)
(%)
UperNet
Vim-Ii
512
13
40.0
Under ToMe-based token reduction 18 the performance advantage of AMVim becomes even more pronounced, yielding mloU gains of +5.9% and +6.7% over the corresponding pruned Vim models. This significant improve- ment under aggressive token merging highlights the robustness and generalization capability of AMVim in dense prediction tasks. These results confirm that AMVim not only improves accu- racy in standard evaluation but also exhibits stronger generalization under token merging, making it more effective and reliable backbone for efficient downstream vision tasks.
UperNet
Vim-S
512
46
43.3
UperNet
AMVim-TI
512
13
40.2
0.21
UperNet
AMVin-S
46
43,4
0.11
UperNet
ToMe-Vin-Ti
512
13
21.1
UperNet
ToMo-Vim-S
5122
46
22.0
UperNet
ToMc-AMVin-II
512?
13
27.0
5.91
UperNet
TeMc-AMVim-S
512"
28.7
6.71
4.5 Ablation Study
We conducted extensive ablation studies to validate the effectiveness each component AMm.m. The scanning directions involved in these experiments are shown in Figure This study specifically
7

---

Table 5: Comparison of Symmetric and Asym- metric Paths. A denotes the performance gap be- tween Vim (i.e., DI-D2) and other dual-patt con- figurations. Asymmetric paths demonstrate su- perior pruning robustness than symmetric ones.
Table 6: Ablation study on the impact of the
window- aware scanning mechanism on top- ac- curacy (%) during token reduction. Integrating the window-aware scanning mechanism signifi- cantly enhances resilience to pruning.
puth
path2
reduction
FLOPs
A
path1
path2
reduction Eañio
(G)
top-1 acc.
(%)
(%)
(5)
(%)
1.45
76.1
0
1.45
76.1
0.17
1.27
38.7
0
0.27
1.13
41.6
0.27
1.13
41.6
0.36
1.00
37.6
0
D1
D2
0.46
0.86
0.36
1.00
37.6
1.45
76.4
0.31
0.17
1.27
66.5
27.81
1.45
76.0
0.14
0.27
23.51
0.36
1.00
61.2
23.67
17.21
D1
D2_D2
0.86
56.9
22.71
0.27
1.13
54.5
12.97
0.31
0.36
1.00
48.6
0.17
1.27
69.0
30.31
DI
D4
0.46
44.9
10.71
0.27
1.13
67.0
25.41
0.36
1,00
63.6
26.07
1.45
74.9
1.21
D2 D2_d
0.46
0.86
59.3
25.11
0.17
1.27
45.1
6.5t
1.45
76.3
0.21
0.27
1.13
43.5
1.91
0.17
1.27
72.8
34.11
0.27
1.13
70.3
28.71
D1
rand
0.36
1.00
39.3
0.36
29.11
0.46
0.86
34.1
0.14
DI
D2
0.46
0.86
61.3
27.11
focuses on dual-path configurations. For clarity, the hyphen symbol "." represents the connection between the two paths, with the ends indicating the directions of the main paths. The underscore symbol " " denotes direction and the right end reflects the scanning direction within the window. Here, D1-D2 represents Vim (6. while D1-D2_D4 corresponds to AMVimm. Additionally, the pruning-free performance discussed in this study is defined the performance at reduction ratio of O.
window-aware scanning mechanism, where the left end indicates main path
Effect of asymmetry. Table T compares the performance symmetric (D1-D2) and asymmetric paths (D1-D4 and D1 While D1-D4 exhibits marginal 0.1 % decrease in pruning-free free accuracy compared to D1-D2, achieves consistent improvements of 10% across all reduction ratios, with notable 17.2% gain at reduction ratio of 0.17.
To further validate the importance of asymmetry, we introduce random scanning direction in the second path (D1-rand). This configuration yields the lowest pruning-free performance, as the random path struggles capture token relationships and impedes the training process. Despite its reduced pruning robustness relative to D1-D4, DI -rad still significantly outperforms the symmetric D1-D2 These results conclusively demonstrate that asymmetric path designs are essential for enhancing the pruning resilience of Mamba-based architectures.
Effect of window-aware scanning. Table presents the ablation study results for the window-awan scanning mechanism, with global scanning directions aligned with Vim (D1-D2). For simplicity, this study focuses applying the window-awer scanning mechanism to the second path, as similar trends are observed in the first path (see Table in the Appendix).
Accuracy (%)
Accuracy (%)
72.8
66.7
66.3
45.5
41.9
48.6
38.7
r=0
r=0.17
37.6
r=0
000.6
14x14
Window size
Window size
(a) reduction ratio 0.17
(b) reduction ratio 0.36
Figure 6: Ablation study on the impact of window size on top-1 accuracy (%) during token reduction. The symbol 'r' represents the reduction ratio. Across different reduction ratios, moderate window sizes (i.e., 3x3 3 and 5x5) consistently demonstrate superior pruning resistance.

---

Table 7: Ablation study on the impact of multi- scale paths on top-1 accuracy (%) during token reduction. Red text in brackets indicates win- dow sizes. The single-path window-aware design demonstrates superior robustness than the dual- path mechanism, despite both retaining asym- metric multi-scale properties.
Table Ablation study on the impact of win- dow scanning direction on top-1 accuracy (%) during token reduction. The symbol A quanti- fies performance differences relative to Vim (i.e., D1 D1-D2). Distinct scanning directions between the window and global path ensure superior re-
sistance to pruning.
puth1
path2
reduction
FLOPs
top-1 nec
A
patb1
path2
reduction
ratio
FLOPs
(G)
top-1 acc.
(%)
(%)
A
ralio
(G)
(%)
(9)
1.45
76.1
0.17
1.27
38.7
0
0.17
1.27
38.7
0
0.27
1.13
41.6
0
0.27
1.13
41.6
0.36
1,00
37.6
0
0.46
0.86
34.2
0
D1
D22
0.36
1.00
37.6
0
1.45
76.4
0.31
0.46
0.86
34.2
0
0.17
1.27
66.5
27.8-
1:45
75.2
0.81
0.27
1.13
65.1
23.5
0.36
23.6"
0.17
1.27
69.9
31.30
DI
D2.
0.46
0.86
$6.9
22.77
0.27
25.07
U
1.45
76.2
U.IT
DI_ D__D)))2_222
0.36
1.00
62.8
25.31
0.46
0.86
58.1
23.9
0.27
1.13
71.0
29.47
0.36
1,00
67.6
30.1
1.45
76.3
0.21
D1
D2 D6
0.46
0.86
62.3
28.1"
0.17
1.27
72.8
34.17
U
1.45
76.3
0.21
0.27
28.77
0.17
1.27
72.8
34.1"
0.27
1.13
70.3
28.7"
D1
D2_D4
0.36
1.00
66.7
29.17
0.36
1.00
66.7
29.
0.46
0.86
61.3
27.17
DI D2_D4
0.46
0.86
27.1
The results reveal that introducing a same-direction window-awan aware scanning mechanism (D1-D2 D2)
already significantly enhances pruning robustness, underscoring the importance of local dependency modeling in improving architectural robustness. When the main path and window scanning directions further diversified (D1-D2_DA) performance improves by additional 5% compared D1- D2_D2, reinforcing the effectiveness of asymmetric design in boosting pruning resilience. Notably, the pruning-free performance increases by 0.3% for D1-D2 D2 and 0.2% for D1-D2 D4, highlighting the mechanism's 's ability to enhance model representation through local dependency capture.
When the window scanning direction is randomized (D1-D2, DII-D_rrnd), its pruning robustness lies between DI-D2_D2 and D1-D2_D4 underscoring the irreplaceability of asymmetry and the importance of semantic relationship modeling. Additionally, the pruning-free frre performance of D1-D2 rand decreases by only 0.3%, demonstrating that the window mechanism effectively mitigates randomness by coordinating global and local information flows.
Effect of window size.
Window size is critical hyperparameter introduced in this study. To assess its impact on performance, we conducted extensive ablation studies, as shown in Figure In these experiments, the global path remains aligned with Vim (DI-D2), while the second path incorporates the window-aware scanning mechanism. Notably, window size of 1x1 corresponds the D1-D2 configuration, whereas window size of 14x 14 corresponds to the D1-D4 configuration.
The results reveal that the window size has relatively minor effect pruning-free performance, with optimal performance achieved at 3 x However, token reduction performance exhibits significant fluctuations the window size varies, with this trend remaining consistent across all reduction ratios (r=0.17 in Figure 6a and r= 0.36 in Figure (6b). The optimal pruning robustness is observed at intermediate window sizes (3 (3x3 and 5x5), whereas excessively small or large windows all lead to performance collapse.
This behavior is expected, as the window-aware mechanism is specifically designed to capture local dependencies. However, larger window sizes exacerbate Mamba's 's long-range forgetting issue 35355 reducing its effectiveness. Furthermore, the poor robustness of the 1x1 and 14x 14 window sizes results from their deviation from the multi-scale path design. These findings demonstrate that careful window size selection is essential for harmonizing local dependencies with global context, key factor in maximizing the efficacy of the multi-scale architecture.
Effect of multi-scale path. The proposed design integrates window-awen aware scanning into path

---

while retaining sequential scanning in the other, forming asymmetric multi-scale paths. This raises natural question: can a dual-path window-aware mechanism achieve comparable performance by varying window sizes?
As shown in Table 7 the dual-path window-aware configuration (D1_D5(5)-D22 (4(3))))))2(33 underper- forms the single-path window-aware design (D1-D2_Do) D4). despite both maintaining asymmetric multi-scale properties. This performance gap arises from the homogeneous scanning mechanisms in dual window-aware paths. While varying window sizes introduce scale diversity, the lack of receptive field heterogeneity limits their ability to capture hierarchical spatial dependencies. In contrast, the single-pant window-awan aware design achieves superior robustness through heterogeneous scanning mechanisms, balancing local granularity and global continuity.
Effect of scan direction. Table g summarizes ablation studies on the scanning direction within windows, with the global scanning direction fixed align with Vim (D1-D2)). The results show that when the window and main path share the same scanning direction (D1-D2_DD) (DD__D)), accuracy drops by approximately 7% compared heterogeneous scanning directions (D1-D2_ D6 and DI-D2_De Although DI-D2 maintains an asymmetric path design, its diversity in token connections remains significantly lower than that of the heterogeneous configurations, which primarily accounts for its inferior performance.
Additionally, we observe that as long as the scanning direction within the window differs from the main path, the performance remains largely comparable (i.c., D1-D2 D6 vs. DI-D2 DI-D2_D4). While D1-D2_ D6 exhibits slightly better pruning robustness, D1-ID2_DD D4 offers marginal pruning-free performance improvement of 0. 0.1%. this study, we adopt D1-D2 D4 as our default architecture, but will release weights for both configurations collectively demonstrate that pruning robustness is ensured by heterogeneous scanning directions between the window and main path, regardless of the specific direction chosen.
allow flexibility in choice. These findings
5 Conclusion
In this work, we propose AMVim, a vision Mamba architecture designed for pruning robustness. By incorporating window-aware scanning mechanism into one of paths, AMVim enables asymmetric multi-scale scanning. Extensive experiments have verified the effectiveness and high robustness of AMVim, laying foundation for future research on the stability of SSM-based models. Limitations. Due to resource constraints, we have not yet fully explored the scalability of AMVim, such validating its efficacy multi-task scenarios integrating it with cross-modal frameworks. Despite these limitations, our experiments conclusively demonstrate AMVim's superiority pruming robustness and spatial modeling, and we plan to investigate its broader applicability in future work.
Acknowledgments and Disclosure of Funding
This work was supported by National Major Scientific Instrument and Equipment Development Project of National Natural Science Foundation of China under Grant 62427820; in part by Na- tional Natural Science Foundation of China under Grant 62306199 198 and in part by Natural Science Foundation of Sichuan Province under Grant 2024NSFSC1468.
10

---

References
[1] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: simple baseline for image restoration with state-space model. In European conference on computer vision, pages 222-241. Springer, 2024.
[2] Tianxiang Chen, Zi Ye, Zhentao Tan, Tao Gong, Yue Wu, Qi Chu, Bin Liu, Nenghai Yu, and Jieping Ye. Mim-istd: Mamba-in-man mamba for efficient infrared small target detection. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[3] Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv preprins arXiv:24022 10739, 2024.
[4] Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Video- mamba: State space model for efficient video understanding. In European Conference Computer Vision. pages 237-255. Springer, 2024.
[5] Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Bjor Ommer. Zigma: Zigzag mamba diffusion model. arXiv e-prints, pages arXiv-2403, 2024.
[6] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint 2rXiv:2401.09411 2024.
[7] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang. Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model 2024. arXiv preprint arXiv:2401.10166 2024.
[8] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances neural information processing systems, 34:572-555 585, 2021.
[9] Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolu- tional models great on long sequence modeling? arXiv preprint arXiv:2210.09298. 2022.
[10] Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference Machine Learning, pages 26670-26698. PMLR, 2023.
[11] Xuanhua He, Ke Cao, Jie Zhang, Keyu Yan, Yingying Wang, Rui Li, Chengjun Xie, Dan- feng Hong, and Man Zhou. Pan-mamba: Effective pan-sharpening with state space model. Information Fusion, 115:102779, 2025.
[12] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[13] Alexey Dosovitskiy. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 2020.
[14] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training efficeffffienttt image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[15] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 1393719919399, 2021.
[16] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEF/CVE conference computer vision and pattern recognition, pages 10809-10818, 2022.
11

---

[17] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu. Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency- aware soft token pruning. In European conference on computer vision, pages 620-640. Springer, 2022.
[18] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
[19] Youwei Liang, Chongjian Ge. Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.09 07800, 2022.
[20] Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. In European Conference on Computer Vision, pages 75-92. Springer, 2024.
[21] Xiaohuan Pei, Tao Huang, and Chang Xu. Efficientvmamba: Atrous selective scan for light weight visual mamba. arXiv preprint arXiv:2403.09977, 2024.
[22] Badri N Patro and Vijay S Agneeswaran. Simba: Simplified mamba-based architecture for vision and multivariate time series. arXiv preprint arXiv:2403.15361 2024.
[23] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 2rXhv:2312.07722 2023.
[24] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206. 13947, 2022.
[25] Jue Wang, Wentao Zhu, Pichao Wang. Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6387-6397, 2023.
[26] Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212 /2 /4052, 2022.
[27] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208 220.:29938. 2022.
[28] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational visual media, 8(3):415 2022.
[29] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2 18886. 2021.
[30] Jindi Lv, Yuhao Zhou, Yuxin Tian, Qing Ye, Wentao Feng, and Jiancheng Lv. Hypernas: Enhancing architecture representation for nas predictor via hypernetwork. arXiv preprint arXiv:2509.18151, 2025.
[31] Eric Nguyen, Karan Goel, Albert Gu. Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846-2288 22611 2022.
[32] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.0033 2021.
[33] Chenhongyi Yang, Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot J Crowley. Plainmamba: Improving non-hierarchical mamba in visual recognition. arXiv preprint arXiv:2403. 17695, 2024.
[34] Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. arXiv preprint 4033032403.09338 2024.
12

---

[35] Yuheng Shi, Minjing Dong, and Chang Xu. Multi-scale vmamba: Hierarchy in hierarchy visual state space model. arXiv preprint arXiv:2405.14174, 2024.
[36] Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 17164-17174, 2023.
[37] Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and 2021.
Oncel Tuzel. Token pooling in vision transformers. arXiv preprint
[38] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 18134-18144, 2022.
[39] Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can learned tokens do for images and videos? arXiv preprint 1XXv:2106611297, 2021.
[40] Zheng Zhan, Zhenglun Kong. Yifan Gong, Yushu Wu, Zichong Meng, Hangyu Zheng, Xuan Shen, Stratis loannidis, Wei Niu, Pu Zhao, et al. Exploring token pruning in vision state space models. arXiv preprint arXiv:2409 18962, 2024.
[41] Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang. Yifan Gong, Xuan Shen, Xue Lin, Pu Zhao, and Yanzhi Wang. Rethinking token reduction for state space models. arXiv preprint arXiv:2410.14725. 2024.
[42] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Antonio Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. In IJCV, 2019.
[43] Tete Xiao, Yingcheng Liu, Bolei Zhou, Masayoshi Tomizuka, and Fisher Yu. Unified perceptual parsing for scene understanding. ECCV, 2018
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately capture the paper's contributions and scope, with claims supported by results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses its limitations in the conclusion, providing transparency about the boundaries of the work and guiding future research directions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: For each theoretical result, the paper provides the full set of assumptions and a complete and correct proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The method pipeline and experimental details are presented along with corre- sponding reproducible credentials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: data sources used in this study are clearly cited in the paper, and the code will be uploaded in zipped format.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/ccpppblligggddeeeeeeeeeeee Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Comprehensive training and testing details are provided in the Appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The results include the standard deviation calculated from multiple random
runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information about computational resources in the experi- mental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research presented in this paper complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper addresses both the potential positive contributions and possible negative societal impacts of the research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The original owners of the data and models used in the paper properly credited, and the licenses and terms of use are clearly stated and fully respected.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced in the paper are thoroughly documented and made available along with the existing ones.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

A
Implement Details
AMVim fine-tuned using the pretrained weights from Vim. The detailed training configurations are presented in Table D
All other settings are aligned with those of Vim.
Table 9: Training settings for AMVim
Table 10: Comparison results with window-
ImageNet- IK.
aware scanning mechanism applied on various paths. The window-awar mechanism on the
finetune config optimizer
AMVim-T
AMVim-S
AdamW
AdamW
second path exhibits stronger pruning resistance
base learning rale
4e-5
le.5
than on the first path.
minimal leamiig rate
lo-5
5o-6
patl
path2
red.
FLOPs(GG
top-1
(%)
weight decay
1,45
76.1
D
optimizer momentum
31,9920,9,09999
$1,92=00.,0.999
0.17
1.27
38.7
0.27
1.13
41.6
barch size
1024
1024
0.36
1.00
37.6
training epachs
DI
D2
leamning rate schodule
cosine docay
cosine docay
76.4
0.17
1.27
72.3
33.61
warmup epochs
1.13
70,2
28.61
warmup learning
1e.5
0.36
1.00
65.4
27.81
warmup schodule
lincar
lincar
DI DI_3
D2
0.46
0.86
56.8
22,61
drop path
0.3
1.45
76.3
0.21
0.17
1,27
72.8
34.11
mixup
0.8
0.27
1.13
70.3
28.71
cutmix
0.36
1.00
66.7
EMA
Nome
None
DI
D2_D4
0.46
0.86
61.3
27.11
B More Experiments
Impact of window-aware on various paths. To further validate the superiority of our design, applied the window-aware scanning mechanism the first path (DI_D3-1)2) for comparison. As shown in Table I0] both configurations exhibit comparable performance at reduction ratios of 0.17 and 0.27. However, as the reduction ratio increases, the DI-D2 D4 configuration significantly outperforms DI DI_ D3-D2. This indicates that while the window-aware mechanism benefits either path, its integration into the second path yields optimal performance, particularly under aggressive pruning scenarios. Impact of the global scanning direction. In this work, we align the global scanning direction with Vim (i.e., D1-D2). To investigate the impact of global scanning direction. we conducted experiments by changing the global scanning direction of the second path from D2 to D4. shown Table [T] when the intra-window scanning direction set to D2 (D1-D4_D2) pruning robustness improves but remains inferior to configurations with (DI-D4_DG This performance gap likely stems from the significant overlap in token connections and receptive fields between the DI and D2 directions, limiting their complementary information flow.
direction D6
Furthermore, while D1-D4 D6 achieves notable improvements in pruning robustness, it still under- performs our design with global direction D2 (D1-D2 D4) by approximately 2% in accuracy. These results demonstrate that the choice of global scanning direction is critical when combined with the window-aware mechanism.
Impact of sequential scanning path. To further investigate the role of the sequential scanning branch, we implemented the window-aware scanning mechanism on both paths. As shown Table regardless of the intra-window scanning direction, these configurations achieve comparable pruning robustness. While they significantly outperform Vim (which relies solely on sequential scanning) due to the stability introduced by the window-awar mechanism in capturing local dependencies, they exhibit performance degradation pruning-fiee settings compared to Vim. This underscores the importance of sequential scanning for preserving base performance.
The proposed architecture combines sequential scanning with window-awart aware scanning, inheriting the strengths of both strategies. This combination achieves optimal balance between pruning robustness and model performance.
21

---

Table 11: Ablation study on the impact of global scanning directions. A denotes the performance gap relative to Vim (i.e., D1-D2). When inte- grated with the window-aware scanning mech- anism, the global scanning direction D2 outper-
Table 12:
Ablation study on the impact of the sequential scanning path. A represents the performance gap relative Vim (i.e., D1- D2). While the dual-ppth window-and awwre design demonstrates superior pruning robustness com- pared to Vim, it suffers from performance degra- dation in pruning- free settings.
forms D4.
path1
path2 red. ratio
FLOPs(G)
top-1 acc(S)
(1%)
1.45
76.1
path1
path2
FLOPs/G)
top-1
((%)
0.17
1.27
38.7
0
1.45
76.1
0.17
1.27
38.7
D1
D2
0.36
1.00
37.6
0
0.27
1.13
41.6
0.46
0.86
34.2
0.36
1.00
37.6
DI
D2
0
1.45
76.2
0.11
0.46
0,86
34,2
0.17
1.27
59.7
21.11
1
75.8
0.31
17.61
1.27
69.8
31,21
0.36
1.00
54.6
17.01
0.27
1.13
67.1
25.51
D1
D4_D2
0.36
1.00
63.2
25.61
0.46
0.86
51.1
17.01
DI_DI D2_D2
0.46
0.86
58.3
24.21
0
1.45
76.0
0.14
1.45
75.7
0.4.
0.17
1.27
70.9
32.31
26.91
D4_D6 D6
0.36
1.00
64.0
26.51
DI_33 D2
1.00
61.8
24.31
0.46
0.86
58.7
24.51
0.46
0.86
56.3
22.11
0
1.45
76.3
0.21
1,45
75.4
0.6.
0.17
1.27
72.8
34.11
0.17
1.27
70.6
31.91
0.27
28.71
0.27
1.13
67.5
25.91
D2_D4
0.36
1.00
66.7
29.11
DI DLD3 D2
0.46
27.11
22


Reviews:
[{"review_id": "g4Gxc7ztgx", "reviewer": "Reviewer_FyAa", "summary": "This paper proposes a novel architecture as a variant of Vim, named Asymmetric Multi-scale Vision Mamba (AMVim). The core insight is that the proposed asymmetric scanning mechanism outperforms the symmetric scanning strategy used in Vim. In addition, a window-aware scanning mechanism is introduced to further enhance model performance. The proposed method is evaluated on the ImageNet-1K dataset, along with extensive ablation studies.", "questions": "As outlined in the Weaknesses section above.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper is well-organized and clearly written.\n2. It proposes a novel pruning-robust architecture, AMVim, featuring an innovative asymmetric and window-aware scanning mechanism.\n3. Comprehensive ablation studies are conducted to demonstrate the effectiveness of each component.\n---\n**Weaknesses**\n1. **Inappropriate comparison setting in experiments.** In Table 1, it is not fair to compare the performance of AMVim with token-pruned results of Vim. A more appropriate comparison would involve evaluating AMVim against various Vim variants without pruning to establish its baseline performance. For pruning robustness, it would be more convincing to compare both Vim and AMVim under the same pruning strategies—such as Token Recognition and Hidden State Alignment. Including such results would strengthen the empirical support.\n2. **Inconsistency in experimental results.** In Table 1, the top-1 accuracy of AMViM-Tiny is reported as 75.3, while in Table 3, the same model is shown to achieve 76.3. Since these refer to the same architecture, this discrepancy should be clarified.", "was_revised": true, "final_justification": "All my concerns have been resolved. Therefore, I have decided to maintain my decision of borderline accept.", "rebuttal": "We sincerely thank Reviewer FyAa for the valuable comments and suggestions regarding the presentation of Table 1. Below, we provide our detailed response and corresponding modifications.\n> **Q1:** Inappropriate comparison setting in experiments. In Table 1, it is not fair to compare the performance of AMVim with token-pruned results of Vim. A more appropriate comparison would involve evaluating AMVim against various Vim variants without pruning to establish its baseline performance. For pruning robustness, it would be more convincing to compare both Vim and AMVim under the same pruning strategies—such as Token Recognition and Hidden State Alignment. Including such results would strengthen the empirical support.\n\n**A1:** Thanks for pointing out this confusion. We clarify the setting and motivation behind Table 1 as follows:\n\n**Setting:** The Top-1 accuracies reported in Table 1 correspond to AMVim with **ToMe** under a **reduction ratio of 0.17**, and are obtained after **fine-tuning**. These results are derived follow the Hidden State Alignment setting, as mentioned at the end of the first paragraph in Section 4.2. In contrast, all other results presented in the tables and figures (outside of Table 1) are based on training-free evaluations.\nTo avoid confusion, we will revise Table 1 in the final version for greater clarity. The updated table is shown below:\n\n\n|        method         | reduction ratio | top1-acc. (%) | top1-acc. (%) | FLOPs (G) | FLOPs (G) |\n|:---------------------:|:---------------:|:-------------:|:-------------:|:---------:|:---------:|\n|                       |                 |   **tiny**    |   **tiny**    | **small** | **small** |\n|    Vim (baseline)     |      0.00       |     76.1      |     80.5      |   1.45    |   5.08    |\n|   Token Recognition   |      0.17       |     71.3      |     74.8      |   1.28    |   3.57    |\n| Hidden State Alignmen |      0.17       |     75.1      |     78.8      |   1.29    |   3.60    |\n|  **AMVim-ToMe (ft)**  |    **0.17**     |   **75.3**    |   **79.5**    | **1.27**  | **3.60**  |\n\n\n\n**Analysis:** \n- Instead of relying on complex, task-specific pruning techniques, improving the architectural resilience of Mamba enables genric pruning strategies (e.g., ToMe, originally designed for ViTs) to achieve promising performance.\n- This finding suggests that resolving Mamba’s architectural fragility offers a more flexible and scalable solution for pruning robustness.\n\n\n\n---\n\n\n>**Q2**: Inconsistency in experimental results. In Table 1, the top-1 accuracy of AMViM-Tiny is reported as 75.3, while in Table 3, the same model is shown to achieve 76.3. Since these refer to the same architecture, this discrepancy should be clarified.\n\n\n**A2**: Thanks for the question. As clarified in **A1**, the 75.3% top-1 accuracy reported in Table 1 refers to AMVim with **ToMe**, under a **token reduction ratio of 0.17**, and is obtained after **fine-tuning**.\n\nIn contrast, Table 3 reports the 76.3% accuracy as the **non-pruned** (full-token) baseline performance of AMVim, without applying any token reduction. Additionally, the 72.8% result in Table 3 reflects the **training-free** accuracy of AMVim with ToMe under the same 0.17 reduction ratio.\n\n---\nThanks again for helping us improve the paper, and we hope the response can resolve these concerns! Please let us know if there are any further questions. We will be actively available until the end of the rebuttal period."}, {"review_id": "yxvOfLHb1t", "reviewer": "Reviewer_S9i4", "summary": "This paper proposes a new Mamba-based vision model, Asymmetric Multi-scale Vision Mamba (AMVim), to address the accuracy degradation that occurs when applying token reduction techniques to previous Mamba-based vision models. There are two core contributions in AMVim: asymmetric scanning paths to effectively mitigate the drop in token connection survival rate when pruning or merging tokens, and a window-aware scanning mechanism that helps capture local information. In this way, there are two different paths in the proposed Window-based Multi-Scale State Space (WMS3) block, one for global context and another for local dependencies. Experiments on ImageNet-1K demonstrate that AMVim achieves state-of-the-art pruning robustness, with AMVim-T showing a 34% improvement in training-free accuracy and AMVim-S experiencing only a 1.5% drop in accuracy during token reduction, outperforming Vim and even matching ViT in some cases, while also excelling in pruning-free settings.", "questions": "- As for the window-based multi-scale selective scanning mechanism, there is no overlap between different windows. Will it be helpful to add some new modules to exchange the information between different windows? Just like the shift in the swin transformer.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths\n\t- This paper is well-written and easy to follow.\n\t- Extensive evaluations make the proposed method solid and help readers understand the model architecture design. And the extensive ablation studies on all relevant components make it clear.\n- Weakness\n\t- This paper only evaluated the new model from the perspective of model accuracy, but it should also be important to test its real performance and speedup, not just by using FLOPs. And it'd be better to analyze the performance overhead of the proposed WMS3 block.\n\t- The evaluations are only conducted on the image classification task. It would be better to show more results on other benchmarks.", "was_revised": true, "final_justification": "From the throughput-vs-accuracy results, there is a big drop in accuracy (6%) with only a 1.1x speedup. So I will keep my rating.", "rebuttal": "We sincerely thank Reviewer S9i4 for the insightful suggestions and constructive feedback.\n> **Q1**: This paper only evaluated the new model from the perspective of model accuracy, but it should also be important to test its real performance and speedup, not just by using FLOPs. And it'd be better to analyze the performance overhead of the proposed WMS3 block\n\n**A1**: Thanks for the valuable suggestion. In response, we provide a detailed analysis of real hardware performance by reporting throughput-vs-accuracy results.\n\n**Setting**: All experiments are conducted fairly on a single NVIDIA GeForce RTX 4090 GPU with an image resolution of 224×224. We use ToMe as the default token reduction strategy. We choose Vim and LocalVim (window-scan based) as baselines. \n\n**Throughout (image/s):**\n\n\n| method \\ reduction ratio |   0.00    | 0.17   | 0.27   | 0.36   | 0.46   |\n|:----------------------:|:------:| :------: | :------: | :------: | :------: |\n|          Vim           | 2685.2 | 2630.9 | 2962.0 | 3291.0 | 3416.7 |\n|        LocalVim        | 955.3  | 1065.5 | 1168.0 | 1220.2 | 1548.5 |\n|         AMVim          | 2497.3 | 2452.5 | 2750.7 | 3063.1 | 3278.6 |\n\n\n**Top-1 Accuracy (%):**\n| method \\ reduction ratio |   0.00    | 0.17   | 0.27   | 0.36   | 0.46   |\n|:----------------------:|:------:| :------: | :------: | :------: | :------: |\n|          Vim           | 76.1| 38.7 | 41.6 | 37.6 | 34.2 |\n|        LocalVim        | 76.2  | 60.4 | 50.9 | 43.3 | 36.2 |\n|         AMVim          | **76.3** | **72.8** | **70.3** | **66.7** |**61.3** |\n\n**Analysis:**\n- **Efficiency:** AMVim incurs only a modest throughput drop (~180 img/s on average) compared to Vim. This slight overhead mainly stems from the directional switching introduced in one branch of the WMS3 block. However, AMVim achieves **approximately 2.5× higher throughput** than LocalVim, which applies per-layer scanning configurations that introduce greater computational overhead.\n- **Robustness:** Across all token reduction ratios, AMVim consistently achieves the highest accuracy, significantly outperforming both Vim and LocalVim. This confirms the robustness of the proposed architecture under aggressive token pruning.\n- **Conclusion:** These results demonstrate that **AMVim strikes a significantly better balance between efficiency and pruning robustness**, making it a more practical and deployable solution for real-world applications.\n\n\n\n---\n\n\n>**Q2:** The evaluations are only conducted on the image classification task. It would be better to show more results on other benchmarks.\n\n**A2:** Thanks for the helpful suggestion. In response, we add semantic segmentation as a downstream task to further validate the generalization ability of our method beyond image classification.\n\n**Semantic segmentation：**\n\n**Setting:** As no downstream-task checkpoints of Vim are publicly available, we follow the official codebase settings to fairly evaluate both Vim and AMVim. ToMe is used as the token reduction strategy, with training-free pruning results reported under identical conditions.\n\n**results:**\n| method  |   backbone    | image size | param (M) | val mIoU (%) |    $\\triangle$     |\n|:-------:|:-------------:|:----------:|:-----:|:--------:|:------------------:|\n| UperNet |    Vim-Ti     |  $512^2$   |  13M  |   40.0   |         0          |\n| UperNet |     Vim-S     |  $512^2$   |  46M  |   43.3   |         0          |\n| UperNet |   AMVim-Ti    |  $512^2$   |  13M  | **40.2** | $\\uparrow$ **0.2** |\n| UperNet |    AMVim-S    |  $512^2$   |  46M  | **43.4** | $\\uparrow$ **0.1** |\n| UperNet |  ToMe-Vim-Ti  |  $512^2$   |  13M  |   21.1   |         0          |\n| UperNet |  ToMe-Vim-S   |  $512^2$   |  46M  |   22.0   |         0          |\n| UperNet | ToMe-AMVim-Ti |  $512^2$   |  13M  | **27.0** | $\\uparrow$ **5.9** |\n| UperNet | ToMe-AMVim-S  |  $512^2$   |  46M  | **28.7** | $\\uparrow$ **6.7** |\n\n**Analysis:**\n  - AMVim achieves better performance than Vim in the non-pruned setting, indicating stronger representational capacity.\n  - Under token reduction, AMVim consistently outperforms Vim by a significant margin (**up to +6.7 mIoU**), validating its robustness and generalization ability in dense prediction tasks.\n\n**Object detection：**\nGiven the lack of publicly available Vim checkpoints for object detection, we would need to re-train both Vim and AMVim from scratch. Given the high resolution (1024×1024), each run takes **~8 days on 4× RTX 4090 GPUs.**\n\nDue to limited computational resources and other experiment priorities, this evaluation could not be completed within the rebuttal period. However, we acknowledge its importance and **plan to include it in the final version**.\n\n\n\n---\n>**Q3:** As for the window-based multi-scale selective scanning mechanism, there is no overlap between different windows. Will it be helpful to add some new modules to exchange the information between different windows? Just like the shift in the swin transformer.\n\n**A3:** Thanks for the insightful suggestion. To investigate the potential benefits of cross-window information exchange, we incorporated a Swin-style shift module into our architecture.\n\n**Setting:** \nThe shift is applied to the window-based branch. Specifically, odd-numbered WMS3 blocks perform cyclic shifts to enable cross-window interaction, while even-numbered blocks retain standard window-based scanning. We fine-tuned this modified variant from our AMVim checkpoint using the same training setup. ToMe is adopted as the pruning strategy, and we report training-free results. \n\n**Results:**\n\n| method            | image size | param (M) | FLOPs (G) | top-1 acc. (%) | $\\triangle$      |\n|:----------------- |:----------:|:---------:| --------- |:--------------:|:---------------- |\n| AMVim-T           |  $224^2$   |     7     | 1.5       |    **76.3**    | 0                |\n| AMVim-T-swin      |  $224^2$   |     7     | 1.5       |      73.7      | $\\downarrow$ 2.6 |\n| ToMe-AMVim-T      |  $224^2$   |     7     | 1.3       |   **72.8**    | 0                |\n| ToMe-AMVim-T-swin |  $224^2$   |     7     | 1.3       |      66.6      | $\\downarrow$ 6.2 |\n\n**Analysis:**  \n- Integrating the Swin-style shift results in a clear drop in both accuracy (↓2.6%) and pruning robustness (↓6.2%). This may stem from a misalignment between the uniform cyclic shift and the spatially structured design of our multi-scale mechanism, potentially disrupting intended token interactions.\n- We acknowledge the importance of cross-window communication. However, directly transferring the Swin shift from Swin Transformer is suboptimal in our setting. We will investigate more tailored and compatible strategies in future work.\n\n---\nThanks again for the professional, detailed, and valuable reviews! We have done our best to address each of the above concerns and hope our response can resolve them. Please let us know if there are any other questions. We will actively join the discussion until the end of the rebuttal period."}, {"review_id": "YRAmxlPl7A", "reviewer": "Reviewer_f6Uw", "summary": "This paper points out the fundamental problem that Mamba's chain-type scan structure easily breaks connections when tokens are reduced, causing cascading information loss and a sharp drop in accuracy as a result. To mitigate this, Vim introduced a symmetric dual-path scan, but it points out the limitation that both paths still rely on sequential (chain-like) connections, leaving the same vulnerability partially intact.\nThe proposed method, AMVim, asymmetrically inserts window-based multi-scale selective scanning into one of the two paths to ensure diversity in token connections. By simply changing the design, it achieves Top-1 +0.2 % based on Tiny and Small models without increasing parameters or FLOPs. Additionally, even without using a dedicated pruning technique, applying only the general-purpose Token Merging (ToMe) and Token Pruning methods results in higher accuracy than the specialized pruning method for Mamba, emphasizing its structural superiority.", "questions": "1. Is there a reason why the figures for AMVim in Table 1 and Table 3 are different?\n\n2. Have you tried using methods such as effective receptive fields (ERF) or activation maps to visualize whether global-local information is well reflected?\n\n3. The term “multi-scale” seems to be appropriate for situations where different image or feature map sizes are utilized, so it may be necessary to revise it to a more appropriate term.\n\nFor the rest, please refer to the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. This paper integrates dual-path scanning with a window-based multi-scale selective scanning mechanism, significantly reducing accuracy degradation under token reduction.\n\n2. It consistently achieves a +0.2% Top-1 improvement over Vim on both Tiny and Small models without additional FLOPs, outperforming the baseline at equal compute.\n\n3. Conducts systematic ablations on asymmetric window scanning to identify an optimal configuration, providing solid experimental justification for the design choices.\n\n# Weakness\n### 1. There is no evaluation of real-world latency or memory.\n- Results are limited to FLOPs and parameter counts; there is no throughput-vs-accuracy or memory-footprint measurement on actual hardware.\n- Given Mamba’s linear complexity, the paper also omits latency scaling tests as image resolution (token length) grows.\n\n### 2. Reliance on pretrained Vim checkpoints\n- Every experiment fine-tunes Vim pretrained weights. Training AMVim from scratch is not explored, so it remains unclear whether the reported gains are intrinsic or inherited.\n\n### 3. Restricted model scale\n- Only Tiny and Small variants are tested. Base (or beyond) models offered by Vim are absent, leaving scalability to higher-capacity settings unverified.\n\n### 4. No downstream-task evidence\n- The method’s potential for dense prediction tasks (e.g., object detection, semantic segmentation) is asserted but never demonstrated, so cross-task generality is still speculative.", "was_revised": true, "final_justification": "The authors have thoroughly addressed the concerns I raised, and their rebuttal satisfactorily resolved the previously incomplete aspects; consequently, I have adjusted my overall rating and the associated scores.", "rebuttal": "We sincerely thank Reviewer f6Uw for the constructive comments. Below, we provide our detailed responses.\n> Q1: There is no evaluation of real-world latency or memory. \n\n**A1**: Thanks for the valuable suggestion. In response, we include throughput-vs-accuracy experiments as well as latency scaling tests across different image resolutions to better reflect real-world performance.\n\n**1. Throughput vs. Accuracy:**\n\n**Setting:** All experiments are conducted fairly on a single NVIDIA GeForce RTX 4090 GPU with an image resolution of 224×224. We use ToMe as the default token reduction strategy. We choose Vim and LocalVim (window-scan based) as baselines. \n  \n**Throughout (image/s):**\n| method \\ reduction ratio |   0.00   | 0.17   | 0.27   | 0.36   | 0.46   |\n|:----------------------:|:------:| :------: | :------: | :------: | :------: |\n|          Vim           | 2685.2 | 2630.9 | 2962.0 | 3291.0 | 3416.7 |\n|        LocalVim        | 955.3  | 1065.5 | 1168.0 | 1220.2 | 1548.5 |\n|         AMVim          | 2497.3 | 2452.5 | 2750.7 | 3063.1 | 3278.6 |\n\n**Top-1 Accuracy (%):**\n| method \\ reduction ratio |   0.00    | 0.17   | 0.27   | 0.36   | 0.46   |\n|:----------------------:|:------:| :------: | :------: | :------: | :------: |\n|          Vim           | 76.1| 38.7 | 41.6 | 37.6 | 34.2 |\n|        LocalVim        | 76.2  | 60.4 | 50.9 | 43.3 | 36.2 |\n|         AMVim          | **76.3** | **72.8** | **70.3** | **66.7** |**61.3** | \n  \n**Analysis:**\n- **Efficiency:** AMVim shows only a slight throughput drop (\\~180 img/s) compared to Vim, yet delivers **~2.5×** higher throughput than LocalVim.\n- **Robustness:** AMVim consistently achieves the highest pruning robustness across all reduction ratios with a substantial margin.\n- **Conclusion:** AMVim **strikes a significantly better balance between robustness and efficiency**, making it more practical for real-world deployment.\n\n**2. Latency scaling tests:**\n\n  **Setting:** All experiments are conducted fairly on a single NVIDIA GeForce RTX 4090 GPU. These evaluations are performed under non-pruning settings.\n  \n**Throughput (image/s):**\n| method \\ reduction ratio |  256   |  512  |  640  |  738  | 1024 |\n|:------------------------:|:------:|:-----:|:-----:|:-----:|:----:|\n|           Vim            | 1660.1 | 414.9 | 310.0 | 201.1 | 95.9 |\n|          AMVim           | 1585.3 | 388.7 | 292.5 | 189.7 | 90.5 |\n\n**Analysis:**\n  - Throughput drops for both methods as resolution increases, reflecting Mamba’s linear complexity.\n  - AMVim incurs only a modest throughput drop compared to Vim, and this gap gradually narrows as the resolution increases.\n\n---\n>Q2: Reliance on pretrained Vim checkpoints.\n\n**A2:** Thanks for the helpful suggestion. To assess whether the performance gains of AMVim are intrinsic or merely inherited from Vim, we retrained AMVim from scratch under the official settings.\n\n**Setting:** We followed the default training configurations provided by Vim. ToMe was adopted as the pruning strategy, and we report training-free results.\n\n**Results:**\n| method \\ reduction ratio |    0.00     |   0.17   |   0.27   |   0.46   |   0.36   |\n|:------------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|           Vim            |   76.1   |   38.7   |   41.6   |   34.2   |   37.6   |\n|      AMVim-retrain       |  75.9    |  67.1   |  64.6    |   60.9    |  58.3    |\n|         AMVim-finetuned         | **76.3** | **72.8** | **70.3** | **66.7** | **61.3** |\n\n**Analysis:**\n- AMVim trained from scratch shows some degradation in robustness, confirming **the benefit of pretraining for pruning stability.**\n- Despite this, AMVim trained from scratch still outperforms Vim by **~25%** on average across all pruning ratios, confirming that its **core robustness primarily stems from the proposed multi-scale selective scanning mechanism** rather than inherited weights.\n---\n>**Q3:** Restricted model scale.\n\n**A3:** Thanks for the valuable suggestion. We include the AMVim-Base results to validate the scalability of our approach on larger model capacities.\n\n**Setting:** As no official training configuration publicly available for Vim-Base, we fine-tune the Vim-Base model for 150 epochs using a learning rate of 1e-5. ToMe was adopted as the pruning strategy, and we report training-free results.\n\n**Results:**\n| method       | image size | param (M) | FLOPs (G) | top-1 acc. (%) | $\\triangle$ |\n|:------------ |:----------:|:---------:| --------- |:-------------- |:----------- |\n| Vim-B        |  $224^2$   |    98     | 18.9      | 81.9           | 0           |\n| AMVim-B      |  $224^2$   |    98     | 18.9      | 81.7           |   $\\downarrow$ 0.2           |\n| ToMe-Vim-B   |  $224^2$   |    98     | 16.2      | 78.1           | 0           |\n| ToMe-AMVim-B |  $224^2$   |    98     | 16.2      | **80.0**       |  $\\uparrow$ **1.9**           |\n\n**Analysis:**\nOur method achieves competitive accuracy at full capacity and shows **notable robustness (+1.9%)** under token reduction, confirming the scalability and effectiveness of our approach when extended to higher-capacity models.\n\n---\n>**Q4:** No downstream-task evidence. \n\n**A4:** Thanks for the helpful suggestion. In response, we add semantic segmentation as a downstream task to further validate the generalization ability of our method beyond image classification. \n\n**Semantic segmentation：**\n\n**Setting:** As no downstream-task checkpoints of Vim are publicly available, we follow the official codebase settings to fairly evaluate both Vim and AMVim. ToMe is used as the token reduction strategy, with training-free pruning results reported under identical conditions.\n\n**results:**\n| method  |   backbone    | image size | param (M) | val mIoU (%) |    $\\triangle$     |\n|:-------:|:-------------:|:----------:|:-----:|:--------:|:------------------:|\n| UperNet |    Vim-Ti     |  $512^2$   |  13M  |   40.0   |         0          |\n| UperNet |     Vim-S     |  $512^2$   |  46M  |   43.3   |         0          |\n| UperNet |   AMVim-Ti    |  $512^2$   |  13M  | **40.2** | $\\uparrow$ **0.2** |\n| UperNet |    AMVim-S    |  $512^2$   |  46M  | **43.4** | $\\uparrow$ **0.1** |\n| UperNet |  ToMe-Vim-Ti  |  $512^2$   |  13M  |   21.1   |         0          |\n| UperNet |  ToMe-Vim-S   |  $512^2$   |  46M  |   22.0   |         0          |\n| UperNet | ToMe-AMVim-Ti |  $512^2$   |  13M  | **27.0** | $\\uparrow$ **5.9** |\n| UperNet | ToMe-AMVim-S  |  $512^2$   |  46M  | **28.7** | $\\uparrow$ **6.7** |\n\n**Analysis:**\n  - AMVim achieves better performance than Vim in the non-pruned setting, indicating stronger representational capacity.\n  - Under token reduction, AMVim consistently outperforms Vim by a significant margin (**up to +6.7 mIoU**), validating its robustness and generalization ability in dense prediction tasks.\n\n**Object detection：**\nGiven the lack of publicly available Vim checkpoints for object detection, we would need to re-train both Vim and AMVim from scratch. Given the high resolution (1024×1024), each run takes **~8 days on 4× RTX 4090 GPUs.**\n\nDue to limited computational resources and other experiment priorities, this evaluation could not be completed within the rebuttal period. However, we acknowledge its importance and **plan to include it in the final version**.\n\n---\n>**Q5:** Is there a reason why the figures for AMVim in Table 1 and Table 3 are different?\n\n**A5:** Thanks for the question. The Top-1 accuracies in Table 1 correspond to AMVim with ToMe under a token reduction ratio of 0.17, and are obtained after **fine-tuning**. These results are derived follow the Hidden State Alignment setting, as mentioned at the end of the first paragraph in Section 4.2. In contrast, all other results presented in the tables and figures (outside of Table 1) are based on **training-free** evaluations.\n\nTo avoid confusion, we will revise Table 1 in the final version for greater clarity. The updated table is shown below:\n|        method         | reduction ratio | top1-acc. (%) | top1-acc. (%) | FLOPs (G) | FLOPs (G) |\n|:---------------------:|:---------------:|:-------------:|:-------------:|:---------:|:---------:|\n|                       |                 |   **tiny**    |   **tiny**    | **small** | **small** |\n|    Vim (baseline)     |      0.00       |     76.1      |     80.5      |   1.45    |   5.08    |\n|   Token Recognition   |      0.17       |     71.3      |     74.8      |   1.28    |   3.57    |\n| Hidden State Alignmen |      0.17       |     75.1      |     78.8      |   1.29    |   3.60    |\n|  **AMVim-ToMe (ft)**  |    **0.17**     |   **75.3**    |   **79.5**    | **1.27**  | **3.60**  |\n\n---\n>**Q6:** Visual analysis via ERF or activation maps.\n\n**A6:** Thanks for the insightful suggestion. In response, we employed Effective Receptive Field (ERF) visualization to examine the activation patterns of Vim and AMVim. \n\n**Setting:** We use random noise as input to probe the model’s receptive behavior. The visualizations are generated using a jet colormap for clarity.\n\n**Results:**  Due to the official restriction on uploading figures during the rebuttal phase, we will include the comparative ERF visualizations in the final version.\n\n**Analysis:**\n- Vim exhibits activation primarily at the four corners, especially top-left and bottom-right, with almost no activation in the center, indicating a biased and spatially sparse response.\n- AMVim shows **significantly broader and more evenly distributed activation**, particularly in the center region, demonstrating enhanced spatial coverage and perception.\n---\n> Q7: The term “multi-scale” seems to be appropriate.\n\n**A7:** Thanks for the thoughtful advice. While we use “multi-scale” to reflect the use of different window sizes for capturing varied contextual ranges, we acknowledge the potential ambiguity. We will revise the term in the final version to better reflect the design for improved clarity."}, {"review_id": "3QQHBWXGaN", "reviewer": "Reviewer_tmKq", "summary": "This work investigates changing the Vision Mamba (Vim, Zhu et al., 2024) model architecture to make it more amenable to\ncontemporary token compression techniques. To this end, the authors propose to alter the order in which tokens are \ntraversed. The original Vim architecture traverses tokens in two symmetric paths (horizontal and inverse\nhorizontal scan order) before merging the resulting features. Instead, the authors investigate more complex asymmetric\npaths. In particular, they investigate splitting the image into non-overlapping windows, with the path traversing\nwindows in one scan order in the outer loop, and the tokens inside the window in another scan order in the inner loop.  \n\nThe authors find that using scan order in one path and their window-aware order in the other improves model robustness\nto state-of-the-art (SOTA) token pruning and merging techniques in the context of an ImageNet classification task.\nThe robustness benefits compared to vanilla ViM are found to increase with token reduction ratios, and to be \nparticularly pronounced for smaller network architectures (Vim-T). The authors conjecture that the observed effect is\ndue to a smaller proportion of token connections being disrupted in the proposed asymmetric path combination, compared\nto the vanilla symmetric one.", "questions": "- Table 1: The $75.3$ and $79.5$ accuracies are reported for AMVim with ToME? If so, which token reduction ratio? The\nvalues do not seem to match any in the rest of the tables/figures for AMVim, neither with nor without token reduction.\n- Table 4 and section 4.4: Is the random scanning order kept the same between the different token reduction ratio runs?\n- Have the authors also experimented with different scanning configurations per layer? This could potentially further\nimprove robustness under different token reduction ratios.\n- Have the authors also considered applying the method to Vision-LSTMs (ViL, Alkin et al., ICLR, 2025)? They outperform\nVim on many vision tasks, while also scaling better. The window-based token traversal idea seems to be directly\napplicable to ViLs.\n\nMinor observation: The multiscale scanning mechanism is fist mentioned on line 59 as an advantage in the introduction,\nbut at that point it is not explicitly linked to the window-based nature of the scan. I think making this link earlier\nmakes the contribution clearer, removing any early impression that there is an additional multiscale token extraction\nmechanism being proposed.", "limitations": "- Single search direction per layer (optimum may be layer-dependent)\n- The study only considers a single scanning process setting per layer (in contrast to the Adaptive Scan approach\nproposed in LocalMamba).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* Well-structured paper, that clearly places the method within the broader context and in relation to related work\n* The method is intuitive and easy to implement, but at the same time effective for the considered benchmark\n* In-depth ablation studies, focusing on key algorithmic choices \n\nWeaknesses:\n\n* Evaluation is only carried out for image classification on a single benchmark dataset (ImageNet-1K). It is not clear\nif the benefits still hold for other key tasks (e.g. image segmentation, object detection or video classification).\n* Somewhat limited technical novelty, as both varying scanning direction and window-based scanning are not new \n(as the authors also correctly cite). However, I am not aware of work employing asymmetric paths in conjunction with\nVim.\n* The experimental section is confusion at times:\n  - Table 1: it is not clear what AMVim performance is being reported (see questions to authors)\n  - Table 3: \"prune\" stands \"token merging\" via ToMe, which is inconsistent with the rest of the tables\n    (e.g. Table 2 uses \"prune\" to mean random pruning, and \"merging\" to refer to ToMe)\n  - Tables 2-3: FLOPs reported with 1 digit as opposed to 2 in the other tables (this is a minor comment, but it does\n  make it somewhat confusing to match results across tables/figures)\n  - Standard deviations not reported for experiments involving randomness\n* The authors only explore a single scanning direction pair configuration per layer, which may limit performance (see\nlimitations section)", "was_revised": true, "final_justification": "I thank the reviewers for their detailed rebuttal and additional results. Their responses address all of my concerns. I am revising my recommendation accordingly.", "rebuttal": "We sincerely thank Reviewer tmKq for the valuable suggestions. We make responses as follows.\n>**Q1:** Evaluation is only carried out for image classification on a single benchmark dataset (ImageNet-1K).\n\n**A1:** Thanks for the helpful suggestion. In response, we add semantic segmentation as a downstream task to further validate the generalization ability of our method beyond image classification. \n\n**Semantic segmentation：**\n\n**Setting:** As no downstream-task checkpoints of Vim are publicly available, we follow the official codebase settings to fairly evaluate both Vim and AMVim. ToMe is used as the token reduction strategy, with training-free pruning results reported under identical conditions.\n\n**results:**\n| method  |   backbone    | image size | param (M) | val mIoU (%) |    $\\triangle$     |\n|:-------:|:-------------:|:----------:|:-----:|:--------:|:------------------:|\n| UperNet |    Vim-Ti     |  $512^2$   |  13M  |   40.0   |         0          |\n| UperNet |     Vim-S     |  $512^2$   |  46M  |   43.3   |         0          |\n| UperNet |   AMVim-Ti    |  $512^2$   |  13M  | **40.2** | $\\uparrow$ **0.2** |\n| UperNet |    AMVim-S    |  $512^2$   |  46M  | **43.4** | $\\uparrow$ **0.1** |\n| UperNet |  ToMe-Vim-Ti  |  $512^2$   |  13M  |   21.1   |         0          |\n| UperNet |  ToMe-Vim-S   |  $512^2$   |  46M  |   22.0   |         0          |\n| UperNet | ToMe-AMVim-Ti |  $512^2$   |  13M  | **27.0** | $\\uparrow$ **5.9** |\n| UperNet | ToMe-AMVim-S  |  $512^2$   |  46M  | **28.7** | $\\uparrow$ **6.7** |\n\n**Analysis:**\n  - AMVim achieves better performance than Vim in the non-pruned setting, indicating stronger representational capacity.\n  - Under token reduction, AMVim consistently outperforms Vim by a significant margin (**up to +6.7 mIoU**), validating its robustness and generalization ability in dense prediction tasks.\n\n**Object detection：**\nGiven the lack of publicly available Vim checkpoints for object detection, we would need to re-train both Vim and AMVim from scratch. Given the high resolution (1024×1024), each run takes **~8 days on 4× RTX 4090 GPUs.**\n\nDue to limited computational resources and other experiment priorities, this evaluation could not be completed within the rebuttal period. However, we acknowledge its importance and **plan to include it in the final version**.\n\n---\n>**Q2:** Somewhat limited technical novelty.\n\n**A2:** Thanks for the question. We agree that both varying scanning directions and window-based mechanisms have been explored individually.  However, **our key novelty lies in introducing the multi-scale concept into SSM-based modeling**, which enables emergent architectural capabilities that cannot be achieved by either technique alone. We clarify this from two perspectives:\n\n**1. Pruning robustness:**\n\n**Setting:** We choose Vim and LocalVim (window-based scanning) as baselines. All models are evaluated using ToMe as the pruning strategy, and we report their training-free Top-1 accuracy across various token reduction ratios.\n\n**Top-1 Accuracy (%):**\n| method \\ reduction ratio |   0.00    | 0.17   | 0.27   | 0.36   | 0.46   |\n|:----------------------:|:------:| :------: | :------: | :------: | :------: |\n|          Vim           | 76.1| 38.7 | 41.6 | 37.6 | 34.2 |\n|        LocalVim        | 76.2  | 60.4 | 50.9 | 43.3 | 36.2 |\n|         AMVim          | **76.3** | **72.8** | **70.3** | **66.7** |**61.3** | \n\n**Analysis:**\n- Although LocalVim employs a window-based scanning strategy, AMVim consistently achieves higher accuracy across all token reduction ratios, particularly under aggressive pruning (e.g., 46% reduction). This indicates that **window-based scanning alone is insufficient for ensuring robustness**.\n- As shown in the ablation studies (Section 4.4), **simply varying scanning directions yield limited improvemen**t.\n\n**2. Efficiency:**\n\n**Setting:** All experiments are conducted fairly on a single NVIDIA GeForce RTX 4090 GPU with an image resolution of 224×224. We use ToMe as the default token reduction strategy.\n\n**Throughout (image/s):**\n| method \\ reduction ratio |   0.00   | 0.17   | 0.27   | 0.36   | 0.46   |\n|:----------------------:|:------:| :------: | :------: | :------: | :------: |\n|          Vim           | 2685.2 | 2630.9 | 2962.0 | 3291.0 | 3416.7 |\n|        LocalVim        | 955.3  | 1065.5 | 1168.0 | 1220.2 | 1548.5 |\n|         AMVim          | 2497.3 | 2452.5 | 2750.7 | 3063.1 | 3278.6 |\n\n**Analysis:**\n  - AMVim incurs a modest throughput drop (~180 img/s) due to window-based scanning.\n  - AMVim remains **~2.5× faster** than LocalVim, despite both using window mechanisms.\n  - This demonstrates that AMVim offers a **superior trade-off between robustness and efficiency** for practical deployment.\n \n---\n>**Q3:** Table 1: The 75.3 and 79.5 accuracies are reported for AMVim with ToME? If so, which token reduction ratio?\n\n**A3:** Thanks for the question. The Top-1 accuracies of 75.3% and 79.5% reported in Table 1 correspond to AMVim with ToMe under a token reduction ratio of 0.17, and are obtained after **fine-tuning**. These results are derived follow the Hidden State Alignment setting, as mentioned at the end of the first paragraph in Section 4.2. In contrast, all other results presented in the tables and figures (outside of Table 1) are based on **training-free** evaluations.\n\nTo avoid confusion, we will revise Table 1 in the final version for greater clarity. The updated table is shown below:\n|        method         | reduction ratio | top1-acc. (%) | top1-acc. (%) | FLOPs (G) | FLOPs (G) |\n|:---------------------:|:---------------:|:-------------:|:-------------:|:---------:|:---------:|\n|                       |                 |   **tiny**    |   **tiny**    | **small** | **small** |\n|    Vim (baseline)     |      0.00       |     76.1      |     80.5      |   1.45    |   5.08    |\n|   Token Recognition   |      0.17       |     71.3      |     74.8      |   1.28    |   3.57    |\n| Hidden State Alignmen |      0.17       |     75.1      |     78.8      |   1.29    |   3.60    |\n|  **AMVim-ToMe (ft)**  |    **0.17**     |   **75.3**    |   **79.5**    | **1.27**  | **3.60**  |\n\n\n---\n>**Q4:** Table 4 and section 4.4: Is the random scanning order kept the same between the different token reduction ratio runs?\n\n**A4:** Thanks for the question. For a fair comparison, the random scanning orders are generated in advance and kept fixed throughout training and during evaluation across all token reduction ratios.\n\n---\n>**Q5:** Have the authors also experimented with different scanning configurations per layer?\n\n**A5:** Thanks for the insightful suggestion. Indeed, LocalVim explores per-layer scanning variations. We address this question from both the robustness and efficiency perspectives:\n\n- **Robustness:**\nAs shown in the robustness results (refer to the top-1 accuracy table in **A2**), this strategy does improve robustness compared to the baseline Vim. However, it still lags significantly behind our method in terms of pruning robustness. \n- **Efficiency:**\nMore importantly, adopting layer-wise scanning variations typically incurs higher computational costs. This is reflected in the throughput results (see the throughput table in **A2**), where LocalVim shows a **~2.5× lower** than our AMVim. Such a design conflicts with our core objective of maintaining high efficiency.\n\n---\n> **Q6:** Have the authors also considered applying the method to Vision-LSTMs (ViL, Alkin et al., ICLR, 2025)? \n\n**A6:** Thanks for the valuable suggestion. Our window-based multi-scale selective scanning mechanism is model-agnostic and can be flexibly integrated into ViL-style architectures. Specifically, in ViL, the original flip operation in the even-numbered blocks could be replaced with our window-based scanning strategy to enhance multi-scale information exchange between global and local contexts.\n\nIn this work, we focus on Vim as a representative Mamba-based backbone to enable a controlled and fair comparison within the family of state-space models. While we have not incorporated our method into ViL within the limited rebuttal timeframe, we plan to explore a ViL-based variant in the camera-ready version to further demonstrate the generality of our approach.\n\n---\n> **Q7:** Minor observation: The multiscale scanning mechanism is fist mentioned on line 59 as an advantage in the introduction, but at that point it is not explicitly linked to the window-based nature of the scan.\n\n**A7:** Thanks for pointing out this potential ambiguity. To improve clarity and prevent misinterpretation, we will revise the beginning of the seventh paragraph (Line 50) in the introduction as follows:\n\n*\"In this work, we propose AMVim, a novel Asymmetric Multi-scale Vision Mamba for pruning robustness. To enrich spatial information diversity, we introduce a multi-scale scanning mechanism that integrates a window-based scanning strategy in one branch while retaining sequential scanning in the other.\"*\n\nThis revision explicitly links the multi-scale behavior to the window-based design from the outset, making it clear that the multi-scale perception arises directly from the window-based scanning mechanism, rather than from an additional or explicit multi-scale module. \n\n---\n>**Q8:** The experimental section is confusion at times.\n\n**A8:** Thanks for pointing out these sources of confusion. We address each issue in detail below:\n\n**Inconsistent terminology:** We will explicitly change the label \"prune\" to \"ToMe\" in Table 3 to clearly distinguish token merging from random pruning and ensure consistency across all tables.\n\n**Inconsistent precision:** We will unify the formatting of reported FLOPs to one decimal digit throughout the paper for clarity and consistency.\n\n**Missing standard deviations:** We acknowledge this oversight and will include standard deviation values for experiments involving randomness in the final version to improve the statistical reliability of the results."}]

Author rebuttal:

Meta-review: This paper received 4, 3, 4, 4 originally, which are elevated to 4, 5, 5, 4 after rebuttal and discussions. The main contribution is to modify the vision Mamba such that it becomes robust to pruning. The reviewers all praised the well-written nature and comprehensive ablation studies. After the rebuttal, most concerns have been resolved, but there is still concerns that the throughput increase may not be worthy the reduction in accuracy. However, overall reviewers all agree that the current method is already a valuable contribution to the community.
Decision: Accept (poster)