VMamba: Visual State Space Model
Authors: Yue Liu
Yunjie Tian
Yuzhong Zhao
Hongtian Yu
Lingxi Xie
Yaowei Wang
Qixiang Ye
Jianbin Jiao
Yunfan Liu
Venue: NeurIPS 2024 spotlight (2024)
Keywords: State Space Model
Transformer
Computer Vision
Foundation Model
Primary area: machine_vision
TL;DR: This paper introduces a new architecture with promising performance on visual perception tasks, which is based on mamba.

Abstract: Designing computationally efficient network architectures remains an ongoing necessity in computer vision. In this paper, we adapt Mamba, a state-space language model, into VMamba, a vision backbone with linear time complexity. At the core of VMamba is a stack of Visual State-Space (VSS) blocks with the 2D Selective Scan (SS2D) module. By traversing along four scanning routes, SS2D bridges the gap between the ordered nature of 1D selective scan and the non-sequential structure of 2D vision data, which facilitates the collection of contextual information from various sources and perspectives. Based on the VSS blocks, we develop a family of VMamba architectures and accelerate them through a succession of architectural and implementation enhancements. Extensive experiments demonstrate VMamba’s
 promising performance across diverse visual perception tasks, highlighting its superior input scaling efficiency compared to existing benchmark models. Source code is available at https://github.com/MzeroMiko/VMamba

VMamba: Visual State Space Model
Yue Liu
Hongtian Yu' Yaowei Wang Qixiang Ye
Yunjie Tian'
Yuzhong Zhao'
Lingxi Xie
Jianbin Jiao
Yunfan Liu
UCAS 2 Huawei Inc.
Pengcheng Lab.
(1iuyue171, tinyunniiee /
ucas
198808xc0gmail.com com, wangywepcl.ac. cn, (qxye,jiaojb, aln.llllunnnnnnnna Oucas. ac
Abstract
Designing computationally efficient network architectures remains ongoing necessity in computer vision. In this paper, we adapt Mamba, state-space language model, into VMamba, vision backbone with linear time complexity. At the core of VMamba is a stack Visual State-Space (VSS) blocks with the 2D Selective Scan (SS2D) module. By traversing along four scanning routes, SS2D bridges the gap between the ordered nature 1D selective scan and the non-sequential structure of 2D vision data, which facilitates the collection of contextual information from various sources and perspectives. Based on the VSS blocks, we develop family of VMamba architectures and accelerate them through succession of architectural and implementation enhancements, Extensive experiments demonstrate VMamba's s promising performance across diverse visual perception tasks, highlighting its superior input scaling efficiency compared to existing benchmark models. Source code is available at https: h /ttp//tgttttutt/t///// www. http///eeroooo/o////// /VMamba
Introduction
Visual representation learning remains as a fundamental research area in computer vision that has witnessed remarkable progress the era of deep learning. To represent complex patterns in vision data, two primary categories of backbone networks, i.e., Convolutional Neural Networks (CNNs) 44
27 29 53 37 and Vision Transformers (ViTs) 13 36 57 66), have been proposed and extensively utilized in variety of visual tasks. Compared to CNNs, VTTs generally demonstrate superior learning capabilities on large-scale data due to their integration of the self-attention mechanism [58] [3].
However, the quadratic complexity of self-attention w.r.t. the number of tokens imposes substantial computational overhead downstream tasks involving large spatial resolutions.
To address this challenge, significant efforts have been made improve the efficiency of attention computation [54 36 12]. However, existing approaches either restrict the size of the effective receptive field 36 or suffer from notable performance degradation across various tasks [30] 60). This motivates us to develop a novel architecture vision data, while maintaining the inherent advantages of the vanilla self-attention mechanism, i.e., global receptive fields and dynamic weighting
parameters 23
Recently, Mamba of natural language processing (NLP), has emerged as promising approach for long-sequence modeling with linear complexity. Drawing inspiration from this ad advancement, we introduce VMamba, a vision backbone that integrates SSM-based blocks enable efficient visual representation learning. However, the core algorithm of Mamba, i.e., the parallelized selective scan operation, is essentially designed for processing dieedimenii sequential data. This presents challenge when adapting for processing vision data, which lacks an inherent sequential arrangement of visual components. To address this issue, we propose Selective Scan (SS2D), four-way scanning mechanism designed for spatial domain traversal. In contrast to the self-attention mechanism (Figure | (a)). SS2D ensures
innovative State Space Model (SSM) 17 43 59
71
48 in the field
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Self-Attention
(b) SEDD--eeetiie-Seee Scann (SS2D)
Comparison of the establishment of correlations between image patches through (a) self- attention and (b) the proposed 21D-eelective-Scan (SS2D). The red boxes indicate the query image patch, with its opacity representing the degree of information loss.
Figure 1:
that each image patch acquires contextual knowledge exclusively through a compressed hidden state computed along its corresponding scanning path (Figure (b)), thereby reducing the computational complexity from quadratic to linear.
Building on the VSS blocks, we develop family of VMamba architectures (i.e., VMamba- Tiny/Smal//Base) and enhance their performance through architectural improvements and imple- mentation optimizations. Compared benchmark vision models built on CNNs (ConvNeXt (37), ViTs (Swin 336. HiViT [66]), and SSMs (S4ND 44], Vim 669 VMamba consistently achieves higher image classification accuracy ImageNet- K
[9] across various model scales. Specifically, VMamba-BB AMaamaaBaase achieves a top-1 accuracy of 83.9%, surpassing Swin by +0.4%, with a throughput ex- ceeding Swin's by substantial margin over 40% (646 vs. 458). VMamba's 's superiority extends across multiple downstream tasks, with VMamba- "Maaba-Tiny/Small/ Tiny/Small/Base achieving 47.3% 71....7%//99/// mAP in object detection on COCO 33 (1x training schedule). This outperforms Swin by 4.6% 19..99%%/2.../
and Co by 3. 3.1%/ 37//333%/2... 3 22, respectively. As for single-scale semantic segmentation on mloU, which surpasses respectively. Furthermore, unlike ViT-based models, which experience quadratic growth computational complexity with the number of input tokens, VMamba exhibits linear growth in FLOPs while maintaining comparable performance. This demonstrates its oate-of-theear input scalability.
ADE20K (68) VMamba- -Mamba-Tin/SSmaa Tiny/SmallBBase achieves 47. 77.9%%/5.66%/51. Swin by 0%%%%%%%/ and ConvNeXt by 1.9%/1.9%/11
The contributions of this study are summarized as follows:
We propose VMamba, an SSM-based vision backbone for visual representation learning with linear time complexity. A series of architectural and implementation improvements are adopted to enhance the inference speed of VMamba.
We introduce 2D Selective Scan (SS2D) to bridge ID array scanning and 2D plane traversal, enabling the extension of selective SSMs to process vision data.
VMamba achieves promising performance across various visual tasks, including image classification, object detection, and semantic segmentation. It also exhibits remarkable adaptability w.r.t. the length of the input sequence, showcasing linear growth in computa- tional complexity.
2 Related Work
Convolutional Neural Networks (CNNs). Since AlexNet 31 considerable efforts have been
devoted enhancing the modeling capabilities 44 52
27
29 and computational efficiency [8 [3
64 46] of CNN-based models across various visual tasks. Sophisticated operators like depth-wise 28 and deformable convolution (5] 70 have been introduced to increase the flexibility and efficacy of CNNs. Recently, inspired by the success of Transformers (58), modern CNNs [37 have shown promising performance by integrating long-range dependencies
convolution
and dynamic
weights 23 into their designs.
Vision Transformers (ViTs). As a pioneering work, ViT 13 explores the effectiveness of vi- sion models based on vanilla Transformer architecture, highlighting the importance of large-scale

---

pre-training for image classification performance. To reduce ViT's dependence on large datasets, DeiT [57] introduces teacher-student distillation strategy. transferring knowledge from CNNs to VITs and emphasizing the importance of inductive bias in visual perception. Following this approach,
subsequent studies propose hierarchical ViT's 36 12
39
55
Another research direction focuses on improving the computational efficiency of self-attention, which serves as the cornerstone of ViTs. Linear Attention [30] reformulates self-attention a linear dot-product of kernel feature maps, using the associativity property of matrix products to reduce computational complexity from quadratic to linear. GLA 65 introduces hardware-
variant of linear attention that balances memory movement with parallelizability. RWKV [45] also leverages the linear attention mechanism to combine parallelizable transformer training with the efficient inference of recurrent neural networks (RNNs). RetNet [S]] adds a gating mechanism to enable parallelizable computation path, offering an alternative to recurrence. RMT [S] further extends this for visual representation learning by applying the temporal decay mechanism to the spatial domain.
State Space Models (SSMs). Despite their widespread adoption in vision tasks, ViT architectures face significant challenges due to the quadratic complexity of self-attention, especially when handling long input sequences (e.g. high-resolution images). In efforts to improve scaling efficiency (8 7
45 5T 41, SSMs have emerged as compelling alternatives to Transformers, attracting significant models in handling the
research attention. Gu er [2] demonstrate the potential of
long-range dependencies using the HiPPO initialization [18]. To improve practical feasibility, S4 [20] proposes normalizing the parameter matrices into diagonal structure. Various structured SSM models have since emerged, each offering distinct architectural enhancements. such as complex-
diagonal structures 122 19,, support for multiple-input multiple-outpur (50), diagonal plus low-rank decomposition [24], and selection mechanisms [77. These advancements have also been integrated into larger representation models 43 16], further highlighting the versatility and scalability of
structured state space models in various applications. While these models primarily target long-range and sequential data such as text and speech, limited research has explored applying SSMs to vision data with two-dimensional structures.
3 Preliminaries
Formulation of SSMs. Originating from the Kalman filter (32). SSMs are linear time-invariant (LTI) systems that map the input signal u(t) E to the output response y(t) E IR via the hidden state h(f) E RN, Specifically, continuous-time SSMs can be expressed as linear ordinary differential equations (ODEs) as follows,
ht(t) Ah(t) Bu(t), v() = Ch(t) Du((),
where A E NRN B BERNII C RIXN and D E IR1 are the weighting parameters.
Discretization of SSM. To be integrated into deep models, continuous-time SSMs must undergo discretization advance. Concretely, for the time interval [ta: fa), the analytic solution of the hidden state variable htt) at = can be expressed
h(4)) h(taa) (al / B(T)u(T)e ((F))((Tje-AAt-ta) A((-tw)) dr.
By sampling with the time-scale parameter A (i.e, drlth Ad) h((l)) can be discretized by
h. e +d h DB,e AlAue-AAAtt..A)))
where [a. b) is the corresponding discrete step interval. Notably, this formulation approximates the result obtained by the zero-order hold (ZOH) method, which frequently utilized in the literature of models (please refer to Appendix A for detailed proof).
3

---

Cross-scan
96 blocks
Cross-merge
Des2.. ycl
Iatchea
Ouiput Paichwe
Figure 2: Illustration of AD-Selective-Seee (SS2D). Input patches are traversed along four different scanning paths (Cross-Scan), with each sequence independently processed by separate S6 blocks. The results are then merged to construct 2D feature map as the final output (Cross-Merge).
Selective Sean Mechanism. To address the limitation LTI SSMs (Eq. I in capturing the contextual information, Gu el al. [17] propose a novel parameterization method for SSMs, which in- corporates an input-dependent selection mechanism (referred to as S6). However, for selective SSMs, the time-varying weighting parameters pose challenge for efficient computation of hidden states, as convolutions cannot accommodate dynamic weights, making them inapplicable. Nevertheless, since the recurrence relation of h2 in Eq. can be derived, the response yb still be efficiently computed using associative scan algorithms 22 42 50), which has linear complexity (see Appendix B for detailed explanation).
VMamba: Visual State Space Model
4.1 Network Architecture
We develop VMamba in three scales: Tiny, Small, and Base (referred to as VMamba-T, VMamba-S, and VMamba-B, respectively). An overview of the architecture of VMamba-T is illustrated in Figure 3 (a), and detailed configurations are provided Appendix E The input image I E Hx first partitioned into patches by stem module, resulting in 2D feature map with spatial dimension of H/4 x W/4 /4. Without incorporating additional positional embeddings. multiple network stages employed to create hierarchical representations with resolutions of H/8 W/8, H H/16 x W/16, and H/32/ x /32.. Specifically, each stage comprises a down-sampling layer (except for the first stage), followed
stack of Visual State Space (VSS) blocks.
The VSS blocks serve as the visual counterparts to Mamba blocks [17 (Figure (b)) for representation learning. The initial architecture of VSS blocks (referred as the 'vanilla VSS Block' in Figure (c)) is formulated by replacing the S6 module. S6 is the core of Mamba and achieves global receptive fields, dynamic weights (i.e., selectivity), and linear complexity. We substitute it with the newly proposed AD-Selective-Sera (SS2D) module, and more details will be introduced in the following subsection. To further enhance computational efficiency, (highlighted by the red box in Figure 3 (c)), as the effect the gating mechanism has already been achieved by the selectivity of SS2D. As result, the improved VSS block (shown in Figure (d) consists a single network branch with two residual modules, mimicking the architecture of vanilla Transformer block (58). All results in this paper obtained using VMMmba models built with VSS blocks in this architecture.
remove the entire multiplicative branch
4.2 2D-Selective-Scan for Vision Data (SS2D)
While the sequential nature of the scanning operation in S6 aligns well with NLP tasks involving temporal data, it poses a significant challenge when applied to vision data, which is inherently non- sequential and encompasses spatial information (e.g., local texture and global structure). To address this issue, S4ND 44 reformulates SSM with convolutional operations, directly extending the kernel from ID to 2D through the outer-product. However, such modification restricts the weights from being input-dependent, resulting in a limited capacity for capturing contextual information. Therefore, we adhere to the selective scan approach [17 for input processing and propose the 2D-Selective-Ser (SS2D) module adapt S6 to vision data without compromising its advantages.

---

Thrnspppst
Vavilla Vianba
CSM
5B
407
VIS BRcc
VSS BluCK
Block
BICKE
Step
Image
638
81.1
54
$13
Arthitccure
Man
13.3
62:2
1379
92.3
4.5
1164
J942
61.6
9:4.
92.5
1340
Survedee2
82.6
1689
Mora Ligets
Dalcendd
Diccer
$2.1
1190
Linoer
Suie-
81.5
InageletTap3
92
88
(bi Mambe Blcck
(c) The Vanill VSS Block
(di VSS Block
lel Performence Comperison
Figure 3: Left: Illustration of (a) the overall architecture of VMamba, and (b) (d) the structure of Mamba and VSS blocks. Right: Comparison VMamba variants and benchmark methods in terms of classification accuracy and computational efficiency.
Figure 2 illlustaaee that data forwarding in SS2D consists of three steps: cross-scan, selective scanning with S6 blocks, and cross-merge. Specifically, SS2D first unfolds the input patches into sequences along four distinct traversal paths (i.e., Cross-Scan). Each patch sequence is then processed in parallel using a separate S6 block, and the resultant sequences are reshaped and merged to form the output map (i.e., Cross-Merge) Through the use of complementary ID traversal paths, SS2D allows each pixel in the image integrate information from all other pixels across different directions. This integration facilitates the establishment of global receptive fields in the 2D space.
4.3 Accelerating VMamba
As shown Figure (e),
(e), the VMamba-T model with vanilla VSS blocks (referred to as 'Vanilla
VMamba') achieves throughput of 426 images/s and contains 22.9M parameters with 5.6G FLOPs. Despite achieving a state-of-the-an classification accuracy of 82.2% (outperforming Swin-T [36] by 0.9% at the tiny level), the low throughput and high memory overhead present significant challenges for the practical deployment of VMamba.
In this subsection, improvements in both implementation details and architectural design. We evaluate the models with image classification ImageNet-Iie K. The impact of each progressive improvement is summarized as follows, where (%, img/s) denote the gains in top-1 accuracy on ImageNet- and inference throughput, respectively. Further discussion is provided in Appendix H
outline efforts to enhance its inference speed, primarily focusing
Step (a) (-0.0%, +41 img/s) by re-implementing Cross-Scan and Cross-Merge in Triton.
Step (b) (-0.0%, -3 img/s) by adjusting the CUDA implementation selective scan to accommo- date float16 input and float32 output. This remarkably enhances the training efficiency (throughput from 165 to 184), despite slight speed fluctuation at test time.
Step (c) (+0.0%, +174 img/s) by substituting the relatively slow einsum in selective scan with linear transformation (i.e, torch. nn wwnctional...... linear).. We also adopt the tensor layout of (B, C, H, W) to eliminate unnecessary data permutations.
5

---

Table 1: Performance comparison on ImageNet-IK. Throughput values are measured with an A100 GPU and an AMD EPYC 7542 CPU, using the toolkit released by 62 following the protocol proposed in [6.. All images are of size 224 x 224.
Params FLOPs
TP.
Top-1
Params FLOPs
TP. (img/s)
Top-1 (%)
Model
Model
(M)
(G)
(img/s)
(%)
(M)
(G)
Transformer- Barse
ConvNet-Based
DeiT-S
[5
22M
4.6G
1761
79.8
ConvNeXI-T 37
29M
4.5G
1198
82.1
DeiT-B
57
86M
17.5G
503
81.8
ConvNeXt-S 37
5UM
8,7G
684
83.1
HiVIT-T
666
19M
4.6G
1393
82.1
ConvNeXt-B (37
89M
15.4G
436
83.8
HiViT-S
66
38M
9.1G
712
83.5
SSM-Baser
HiViT-B
666
66M
15.9G
456
83.8
S4ND-Conv-T 44
30M
5.2G
683
82.2
Swin-T T 36
28M
4.5G
1244
81.3
S4ND-ViT-B 44
89M
17.1G
397
80.4
Swin-S
36
50M
8.7G
718
83.0
Vim-S 669
26M
5.3G
811
80.5
Swin-B
36
88M
15.4G
458
83.5
VMamba-T
30M
4.9G
1686
82.6
XCTT-S24
48M
9.2G
671
82.6
VMamba-S
50M
8.7G
877
83.6
XCIT-M24 [
84M
16.2G
423
82.7
VMamba-B
89M
15.4G
646
83.9
Step (d) (-0.6%, +175 img/s) by introducing MLP into VMamba due to its computational efficiency. We also discard the DWConv (depth-wise convolutional (23] layers and change the layer configuration from 22,2,9, ,,2] to [2, 2, 2] to lower FLOPs.
Step (e) (+0.6%, +-366 img/s) by reducing the parameter ssm-ratio (the feature expansion factor) from 2.0 to 1.0 (also referred to as Step (d.1)), raising the layer numbers to [2, 2,5, 2] (also referred to Step (d.2)), and discarding the entire multiplicative branch as illustrated in Figure (c).
Step (f) (-0.3%, +161 img/s) by introducing the DWConv layers (also referred to as Step (e. I)) and reducing the parameter d_state (the SSM state dimension) from 16.0 1.0 (also referred to as Step (c.2)), together with raising sam-ratio back to 2.0.
Step (g) (-0.1%, +346 img/s) by reducing the ssm-ratio to 1.0 while changing the layer configu- ration from [2, 22,2,5,21 to [2,2,8,2].
5 Experiments
In this section, we present a series of experiments to evaluate the performance of VMamba and compare it to popular benchmark models across various visual tasks. We also validate the effectiveness of the proposed 2D feature map traversal method by comparing with alternative approaches. Additionally, we analyze the characteristics of VMamba by visualizing its effective receptive field (ERF) and activation map, and examining its scalability with longer input sequences. We primarily follow the hyperparameter settings and experimental configurations used in Swin [36]. For detailed experiment settings, please refer to Appendix E and and for additional ablations, see Appendix All experiments were conducted server with 8 X NVIDIA Tesla-A A100 GPUs.
5.1 Image Classification
We evaluate VMamba's performance in image classification on ImageNet- K ②.. with comparison results against benchmark methods summarized Table With similar FLOPs, VMamba-T "Mamba-Tachieves achieves top-1 accuracy of 82.6%, outperforming DeiT-S by 2.8% and Swin-T by 1.3%. Notably, VMamba maintains its performance advantage at both Small and Base scales. For example, VMamba-B achieves a top-1 accuracy of 83.9%, surpassing DeÎT-B by 2.1% and Swin-B by 0.4%.
In terms of computational efficiency, VMamba-T achieves a throughput of 1,686 images/s, which is either superior or comparable to state-of-the-as methods. This advantage continues with VMamba-S and VMamba-B, achieving throughputs of 877 images/s and 646 images/s, respectively. Compared to SSM-based models, the throughput of VMamba-T is 1.47x higher than S4ND-Conv-T [4] and 1.08 higher than Vim-S (69, while maintaining clear performance lead of 0.4% and 2. over these models, respectively.

---

Table Left: Results for object detection and instance segmentation on MSCOCO. AP ah and APm denote box AP and mask AP, respectively. FLOPs are calculated with an input size of 1280 800. The notation '1 x" indicates models fine-tuned for 12 epochs, while *3 denotes multi-scale training for 36 epochs. Right: Results for semantic segmentation on ADE20K. FLOPs are calculated with an input size of 512 x 2048. 'SS' and 'MS' denote single-scale and multi-scale testing, respectively.
Mask R-CNN
1x schedule
ADE20K with crop size 512
Backbone
AP
Apm
Params
FLOPs
mIOU
mIOU
Backbone
Params
FLOPs
Swin-T
42.7
39.3
48M
267G
(SS)
(MS)
ConvNeXt-T
44.2
40.1
48M
262G
ResNet-50
42.1
42.8
67M
953G
VMamba-T
47.3
42.7
50M
271G
DeïT-S MLN
43.8
45.1
58M
1217G
Swin-S
44.8
40.9
69M
354G
Swin-T
44.5
45.8
60M
945G
ConvNeXt-S
45.4
41.8
70M
348G
ConvNeXt-T
46.0
46.7
60M
939G
VMamba-S
48.7
43.7
70M
349G
NAT-
48.4
58M
934G
Vim-S
44.9
46M
Swin-B
42.3
107M
496G
VMamba-T
47.9
48.8
62M
949G
ConvNeXi-B
47.0
42.7
108M
486G
ResNet- 101
43.8
44.9
86M
1030G
VMamba-B
49.2
108M
485G
DeiT-B MLN
45.5
47.2
144M
2007G
Mask R-CNN 3x MS schedule
Swin-S
47.6
49.5
8IM
1039G
Swin-T
46.0
41.6
48M
267G
ConvNeXI-S
48.7
49.6
82M
1027G
ConvNeXt-T
46.2
41.7
48M
262G
NAT-S
48.0
49.5
82M
1010G
NAT-T
47.7
42.6
48M
258G
VMamba-S
51.2
82M
1028G
VMamba-T
48.8
43.7
50M
271G
Swin-B
48.1
49.7
121M
I 1188G
Swin-S
48.2
43.2
69M
354G
ConyNeXt-B
49.1
49.9
122M
1170G
ConvNeXt-S
47.9
42.9
70M
348G
NAT-B
48.5
49.7
123M
I 1137G
NAT-S
48.4
43.2
70M
330G
RepLKNet-31B
49.9
50.6
112M
1170G
VMamba-S
49.9
44.2
70M
349G
VMamba-B
51.0
51.6
122M
1170G
5.2 Downstream Tasks
In this sub-section, we evaluate the performance of VMamba on downstream tasks, including object detection and instance segmentation on MSCOC02017 [33], and semantic segmentation on ADE20K 68 The training framework is based on the MMDetection K and MMSegmenation
libraries, following [35 in utilizing Mask R-CNN 26 and UperNet [63 as the detection and segmentation networks, respectively.
Object Detection and Instance Segmentation. The results on MSCOCO are presented in Table 2 VMamba demonstrates superior performance both box and mask Average Precision (APb and A across different training schedules. Under the 12-epoct fine-tuning schedule, VMamba- T/S/B achieves object detection mAPs of 47.3% / 48 19.2%. outperforming Swin-T/S/B by 4.6%/
mAP and ConyNeXt-T/SIH T/S//B by 3.1% 3.1%/3.3% 33.3% 3%/3 2.2%% mAP, respectively. VMamba- T/S/B achieves instance segmentation mAPs that exceed Swin-T/S/B by 3.4%/2.8% %1.8% mAP and ConvNeXt-T/SSA by 2.6% 6% /1.%% 1.4% mAP, respectively. Furthermore, VMamba's 's advantages persist with the 36-epoch fine-tume tne-ttunng schedule using multi-scale training, highlighting its strong potential in downstream tasks requiring dense predictions.
Semantic Segmentation. Consistent with previous experiments, VMamba demonstrates superior performance in semantic segmentation ADE20K with comparable amount of parameters. As shown in Table 2 VMamba-T achieves 3.4% higher mloU than Swin-T and 1.9% higher than ConvNeXt-T in the Single-Scale (SS) setting, and the advantage persists with Multi-Scale (MS) input. For models at the Small and Base levels, VMamba-S/E outperforms NAT-S/B 25 by 2.6%/2.5% mloU in the SS setting, and 1. 1.7% mIoU in the MS setting.
The experimental results in this subsection demonstrate VMamba's adaptability to object detection, instance segmentation, and semantic segmentation. In Figure we compare VMamba's performance with Swin and ConvNeXt, highlighting its advantages in handling downstream tasks with comparable classification accuracy on ImageNet- IK. This result aligns with Figure (b), where VMamba shows the most stable performance (i.e., modest performance drop) across different input image sizes, achieving a top-1 classification accuracy of 74.7% without fine-tuning (79.2% with linear tuning) an input resolution of 768 x 768. While exhibiting greater tolerance to changes
Discussion
7

---

Top acc. (8)
50-
Conditeit
2nd gli on ADT200
50
Conshedf
Sein
AP COCO
Apri COO0
DeffS
Cansfaattl
46
42
Swinil
72.5
02
224
3043
5121
640
765
InageNce-1K
Top-1 imagehez-1K
Tep-1 ImagoNot-1K
Top-1 occ. Imagchett
(a)
tretuning w.Z. resolution fises
Figure 4: Illustration of VMamba's adaptability (a) downstream tasks and (b) input images with progressively increasing resolutions. Swin-T" denotes Swin-T tested with scaled window sizes.
FLOPs
Throughout (ingls)
Memory (N)
200
20000
CondRadleT
1235
Comvlean
Delf-S
Conalactt1
REIT 512/15
1000
512111218
VineS
GATUTY
730
Sain-T
10000
300
Vitanta
10
250
51.1
2243
2047
0131
0/07
(0) FLOPs w.r. resoutton rises
(6) Throughout resolution rises
(c) Memory cost wirt resolution rises
Figure 5: Illustration of VMamba's resource consumption with progressively increasing resolutions. Swin-T" denotes Swin-T tested with scaled window sizes.
in input resolution, VMamba also maintains linear growth in FLOPs and memory-consumption (see Figure and (c)) and maintains high throughput ( F (b)), making it more effective and efficient compared to ViT-based methods when adapting to downstream tasks with inputs of larger spatial resolutions. This aligns with Mamba's 's advanced capability in efficient long sequence
modeling 17
5.3 Analysis
Relationship between SS2D and Self-Attention. To formulate the response Y within the time interval (a, of length T, we denote the corresponding SSM-related variables u1 A, RIxD. B; e I RIIDs and C; E Dx as V E RTXD. E RTx,, and Q E TR T respectively. Therefore, the j-th slice along dimension D, of yb. denoted as yb (i) RR can be written as
Yb 5 (QT wT ha (1) + QT Σ
WT
Ki
+ (vi
(4)
w/
Input Image
(b) Activation map of (c) Activation map (K/W)
(di Activation map of (K/W) for indivdual scannin path
Figure 6: Illustration of the activation map for query patches indicated by red stars. The visualization results in (b) and (c) are obtained by combining activation maps from each scanning path in SS2D.

---

After briinin Before training
1.0
0.8
0.6
0.4
0.2
0.0
Resnet-50
ConvNeXt-T
Swin-T
DeiT-S
HiVIT-T
Vim-S
VMamba-T
Figure 7: Comparison of Effective Receptive Fields (ERF) 40 between VMamba and other bench- mark models. Pixels with higher intensity indicate larger responses related the central pixel.
where hn E DR is the hidden state at step a, @ denotes element-wise product. Particularly, V,6) only a scalar. The formulation each element in w == : E RTxDxxD ie.. W; PR xD, be written as [T eAA 1-45, representing the cumulative attention weight at step computed along the scanning path.
Consequently, the j-th dimension of Y, ie.. Y E RT11 can be expressed as
v = Q
ha (
(
(Qowln) w
K
T
M
V
(5)
w(i)
where M denotes the temporal mask matrix size T with the lower triangular part set and elsewhere O. Please refer to Appendix Cer more detailed derivations.
In Eq. the matrix multiplication process involving Q, K. and V closely resembles the self-attention mechanism, despite the inclusion of w.
Visualization of Activation Maps.
To gain an intuitive and in-depth understanding of SS2D,
further visualize the attention values in QK and (Q @ w) (K/w)² corresponding to a specific query patch within foreground objects (referred to the 'activation map'). As shown in Figure 6 (b), the activation map of QK demonstrates the effectiveness of SS2D in capturing and retaining traversed information, with all previously scanned tokens in the foreground region being activated. Furthermore, the inclusion of w results in activation maps that are more focused on the neighborhood of query patches (Figure ( (c)), which is consistent with the temporal weighting effect inherent in the formulation of w. Nevertheless, the selective scan mechanism allows VMamba to accumulate history along the scanning path, facilitating the establishment of long-term dependencies across image patches. This is evident in the sub-figure encircled by a red box (Figure (d)). where patches of the sheep far the left (scanned in earlier steps) remain activated. For visualizations and further discussion, please refer to Appendix D
Visualization of Effective Receptive Fields. The Effective Receptive Field (ERF) 440 refers
to the region in the input space that contributes to the activation of a specific output unit. We conduct comparative analysis of the central pixel's ERF across various visual backbones. both before and after training. The results presented in Figure iillsstrate that among the models examined, only DeiT, HiViT, Vim and VMamba demonstrate global ERFs, while the others exhibit local ERFs despite their theoretical potential for global coverage. Moreover, VMambb's linear time complexity enhances its computational efficiency compared to DeiT and HiViT, which incur quadratic costs w.r.t. the number of input patches. While both VMamba and Vim are based on the Mamba architecture, VMamba's ERF more uniform and 2D-aware than that of Vim, which may intuitively explain its superior performance.
Diagnostic Study on Selective Scan Patterns. We compare the proposed scanning pattern (i.e. Cross-Scan) to three benchmark patterns: unidirectional scanning (Unidi-Scan), bidirectional scan- ning (Bidi-Scan), and cascade scanning (Cascade-Secd Scan, scanning data row-wise and column-wise successively). Feature dimensions adjusted to maintain similar architectural parameters and

---

TP. (img/s)
DW dwconv
Dw/o dwconv
Top-1 (%)
w/ diwconv
O w/o dwconv
1900
16822716
16871719
16861717
83%
82.26%
82.49%
82.42%
60.6
$2.25%
1700
1500
81.80%
81.71%
82%
1300
1100
998 1007
80.88%
900
81%
Unidi
Bidi
Cascade
Cross
Unidi
Bidi
Cascade
Cross
(a)
Figure 8: Performance comparison of different scanning patterns. The proposed Cross-Scan achieves superior performance in speed while maintaining the same number of parameters and FLOPs.
FLOPs for fair comparison. As illustrated in Figure 8 Cross-Scan outperforms the other scanning patterns both computational efficiency and classification accuracy, highlighting effectiveness in achieving "D-Selective-Seae Removing the DWConv layer, which has been shown to aid the model learning 2D spatial information, further enhances this advantage. This underscores the inherent strength of Cross-Scan in capturing 2D contextual information through its adoption of four-way scanning.
Conclusion
This paper presents VMamba, efficient vision backbone model built with State Space Models (SSMs). VMamba integrates the advantages of selective SSMs from NLP tasks into visual data processing, bridging the gap between ordered 1D scanning and non-sequential 2D traversal through the novel SS2D module. Furthermore, we have significantly improved the inference speed of VMamba through a series of architectural and implementation refinements. The effectiveness the VMamba family has been demonstrated through extensive experiments, and its linear time complexity makes VMamba advantageous for downstream tasks with large-resolution inputs.
Limitations. While VMamba demonstrates promising experimental results, there still room for improvement this study. Previous research has validated the efficacy unsupervised pre-training on large-scale datasets (e.g., ImageNet-2) (K). However, the compatibility of existing pre-training methods with SSM-based architectures like VMamba, as well as the identification of pre-training techniques specifically tailored for such models, remain unexplored. Investigating these aspects could serve as promising avenue for future research in architectural design. Additionally, limited computational resources have prevented from exploring VMamba's architecture at the Large scale and conducting a fine-grained hyperparameter search to further enhance experimental performance. Although SS2D, the core component of VMamba, does not make specific assumptions about the layout or modality of the input data, allowing generalize across various tasks, the potential of generalized tasks remains unexplored. Bridging the gap between
VMamba for integration into SS2D and these tasks, along with proposing a more generalized scanning pattern for vision tasks, represents a promising research direction.
7
Acknowledgments
This work was supported by National Natural Science Foundation of China (NSFC) under Grant no.62225208 and 62406304, CAS Project for Young Scientists in Basic Research under Grant No. YSBR-117, China Postdoctoral Science Foundation under Grant No.2023M743442, and Postdoc- toral Fellowship Program CPSF under Grant No. GDB20240730.
10

---

References
[1] Alaacidin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, al. Xcit: Cross-covariance image trans- formers. NeurIPS, 34:20014-20027 2021.
[2] Guy Blelloch. Prefix and their applications. 1990.
[3] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 99960000055555 2019.
[4] MMSegmentation Contributors. MMScgmentation: Openmmlab semantic segmentation toolbox and benchmark. betpp://ggtthbs com/ heem//pen---ae www.m/maaamaaaa. mbmemmmmmttmtttiiionnon 2020.
[5] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li. Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, pages 764-773, 2017.
[6] Zihang Dai, Hanxiao Liu, Quoe Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurIPS, 34:3965 3977, 2021.
Tri Dao. Flashattention-22 2: Faster attention with better parallelism and work partitioning. ICLR, 2023.
[8] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Neur/PS, 35:16344 16359, 2022.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei- Sai-FFii Li. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248 255, 2009.
[10] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. In ECCV, pages 74-92, 2022.
[11] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling your kernels to 31x31: Revisiting large kernel design in cnns. In CVPR, pages 11963-11975, 2022.
[12] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, pages 12124 12134, 2022.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[14] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. weigoid-weighted linear units for neural network function approximation reinforcement learning. Neural Networks, 107:3-11, 2018.
[15] Qihang Fan, Huaibo Huang. Mingrui Chen, Hongmin Liu, and Ran He. Rmt: Retentive networks meet vision transformers. In CVPR, 2024.
[16] Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re, Hungry hungry hippos: Towards language modeling with state space models. In ICLR, 2022.
[17] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 2020::2312.00752. 2023.
[18] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. NewrIPS, 33:1474- 1487, 2020.
[19] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. NeurIPS, 13:35971-35598 2022.
[20] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In ICLR, 2021.
[21] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré, Combining recurrent, convolutional, and continuous time models with linear state space layers. NewrlPS, 34:572-585, 585, 2021.
11

---

[22] Ankit Gupta. Albert Gu, and Jonathan Berant. Diagonal state spaces are effective structured state spaces. NeurlPS, 35:22982-229944 2022.
[23] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming- Ming Cheng. Jiaying Liu, and Jingdong Wang. On the connection between local attention and dynamic depth- wise convolution. In ICLR. 2021.
[24] Ramin Hasani, Mathias Lechner, ''uun-Hsuan Wang. Makram Chahine, Alexander Amini, and Danicla Rus. Liquid structural state-space models. In ICLR, 2022.
[25] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In CVPR, pages 6185-6194, 2023.
[26] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cmn. In ICCV, pages 2961-s2969, 2017.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[28] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: MexXiv::70.0000 7704.04861, 2017.
[29] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, pages 4700-4708, 2017.
[30] Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Flcuret. Transformers are mns: Fast autoregressive transformers with linear attention. In ICML, pages 5156-5165, 2020.
[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey F. Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS, pages 1014,114, 2012.
[32] Rudolf Emil Kálmán. A new approach linear filtering and prediction problems. Journal of Basic Engineering, 82(1):35-45, 1960.
[33] Tsung-Y Lin, Michael Maire. Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pages 740-755, 2014.
[34] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Tommi Kärkkäinen, Mykola Pechenizkiy, Decebal Constantin Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kemelss beyond 51x51 using sparsity. In ICLR, 2023.
[35] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, pages 12009-12019, 2022.
[36] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[37] Zhuang Liu, Hanzi Mao, Chao-Yure Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, pages 11976 11986, 2022.
[38] Ilya Loshchilov and Frank Hutter. 1711v:171.00100,,2 2017.
Decoupled weight decay regularization. arXiv preprint
[39] Jiasen Lu, Roozbeh Mottaghi, Aniruddha Kembhavi, et al. Container: Context aggregation networks. NeurlPS, 14:19661111 2021.
[40] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. NeurIPS, 29:4898-4906 2016.
[41] Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neuhig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In ICLR, 2022.
[42] Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In ICLR, 2018.
[43] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. In ICLR, 2023.
12

---

[44] Eric Nguyen. Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao. Stephen Baccus, and Christo- pher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. NeurIPS. 35:2846-2861, 2022.
[45] Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. RWKV: reinventing rns for the transformer era. In EMNIP, pages 14048-14077, 2023.
[46] Ilia Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, pages 10428-10436, 2020.
[47] Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser Nam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. NeurlPS, 35: 10353-10366, 2022.
[48] Mark Schöne, Neeraj Mohan Sushma, Jingyue Zhuge, Christian Mayr, Anand Subramoney, and David Kappel. Scalable event- processing of neuromorphic sensory signals with deep state-space models. arXiv preprint arXiv:2404. 18508, 2024.
[49] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. 2015.
[50] Jimmy TH Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In ICLR, 2022.
[51] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma. Yuqing Xia, Jilong Xue, Jianyong Wang. and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:3307.08621 2023.
[52] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.
[53] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-61 14, 2019.
[54] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6), 2022.
[55] Yunjie Tian, Lingxi Xie, Zhaozhi Wang, Longhui Wei, Xiaopeng Zhang, Jianbin Jiao, Yaowei Wang. Qi Tian, and Qixiang Ye. Integrally pre-trained transformer pyramid networks. In CVPR, pages 18610- 18620, 2023.
[56] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakoh Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 14:24261-24272, 2021.
[S7] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pages 10347-10357, 2021.
[58] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 30:5998 6008, 2017.
[59] Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In CVPR, pages 6387-6397, 2023.
[60] Sinong Wang, Belinda Z Li, Madian Khabsa. Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
[61] Wenhai Wang, Enze Xie, Xiang Li, Deng- Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In ICCV, pages 568-578, 2021.
[62] Ross Wightman. Pytorch image models. 2019.
[63] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unitied perceptual parsing for scene understanding. In ECCV, pages 418 434, 2018.
13

---

[64] Jianwei Yang, Chunyuan Li. Pengchuan Zhang, Xiyang Dai, Bin Xiao. Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641, 2021.
[65] Songlin Yang, Bailin Wang. Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXi::3312.06335, 2023.
[66] Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang. Qi Dai, Qixiang Ye, and Qi Tian. Hivit: A simpler and more efficient design of hierarchical vision transformer. ICLR, 2023.
[67] Weixi Zhao, Weigiang Wang, and Yunjic Tian. Graformer: Graph-oriented transformer for 3d pose estimation. In CVPR, pages 20438-20447, 2022.
[68] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 5122-5130, 2017.
[69] Lianghui Zhu, Beneheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In ICML, 2024.
[70] Xizhou 7hu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, hetter results. In CVPR, pages 9308-9316, 2019.
[71] Nikola Zubic, Mathias Gchrig, and Davide Scaramuzza. State space models for event cameras. In CVPR, pages 5819-5828, 2024.
14

---

A Discretization of State Space Models (SSMs)
In this section, explore the correlation between the discretized formulations of State Space Models (SSMs) obtained in Sec. and those derived from the zero-order hold (ZOH) method [17] which is frequently used in studies related to SSMs.
Recall the discretized formulation of SSMs derived Sec. as follows,
he- A(Astt..
h2
B,ue AlA. A(A
(6)
Let = a + 1, then the above equation can be re-written as
ha+1 = e h. + Ba
(7)
where A, == A is the exact discretized form of the evolution matrix A obtained by ZOH, and Ba = B, represents the first-order Taylor expansion the discretized B acquired through ZOH.
B Derivation of the Recurrence Relation of Selective SSMs
In this section, derive the recurrence relation of the hidden state in selective SSMs. Given the expression of hy shown in Eq. 6 let us denote eAAA, +.-+)-11)) as PAA.. Then, its recurrence relation can be directly written
PAAa e AA,
'AA,a'
i-1
(8)
For the second term of Eq, 6 we have
b-1
PB.a An-11
(9)
=
PB.
B 106-1111111111
(10)
Therefore, with the associations derived in Eq. 8 and Eq. 9 hy P ha + PR.. can be efficiently
computed in parallel using associative scan algorithms 12 22 [5], which are supported by numerous modern programming libraries. This approach effectively reduces the overall computational com- plexity to lincar, and VMamba further accelerates the computation by adopting hardware-aware implementation 17]
C Details of the relationship between SS2D and Self-attention
In this section, elarify the relationship between SS2D and the self-attention mechanism commonly employed in existing vision backbone models. Subsequently, visualization results are provided to substantiate explanation.
Let denote the length of the sequence with indices from a to b, we define the following variables
V Vi; VTl E T. where Vi Unti-1 Aati-1 E RIXD.
(11)
K = K KT] E RRRxx TxDx where K; := Bati-1 E R1
(12)
Q = Q Qrl e Tx where Q; := Cati-1 E 1x
(13)
= [wi wTl E R Tx D where wi = II J=1
e AAT E D&XD,
(14)
hat1 hpl I xxDxxD, where h DaxD,
(15)
Y =
E TxD, where y1 E RR P.
(16)
Note that in practice the parameter in Eq. is simplified RIID., Consequently, h((1)
Ah(t) + Bu(f) is simplified to ht(t)) = A h + Bu(t), which is the reason why wi R D.
15

---

Based on these notations, the discretized solution of time-varying SSMs (Eq. 6 can be written as
T
hp WT ha
WT Wi
K;
ViI)
(17)
where @ denotes the element-wise product between matrices, and the division is also elements-wise.
Based on the expression of the hidden state hb, the first term of the output of SSM, i.e., Yb. can be computed by Yb = -Qrbb
(18)
T
Qr (WT ④ ha) + QT
WT ①
Ki
(K]T)))
(19)
Here, we drop the skip connection between the input and the response for simplicity. Particularly, the j-th slice along dimension D, of yb. denoted yb (i) R can be written as
T
QT & WT 6 w₂()
yb (i) = (or c wT ha 4
Σ
K;
V
(20)
Similarly, the j-th slice along dimension D, of the overall response Y, denoted as Y E Rx can be expressed
YÚI (Qowo)) h,
(1) ( (ol
K w(3
M
VO)
(21)
where M = tr)) 1) denotes the temporal mask matrix with the lower triangular portion of a T matrix set to and elsewhere O. It is evident that how matrices Q, K, and V are multiplied in Eq. 22 closely resembles the process in the self-attention module of Vision Transformers. Moreover, if w is in shape (T, D&) rather than (T, DR, Dv). then Eq. 18 and Eq. [2 reduce to the form of Gated Linear Attention (GLA) (65), indicating that GLA is also special case of Mamba.
D Visualization of Attention and Activation Maps
In the preceding subsection, we illustrated how the computational process of selective SSMs shares similarities with self-attention mechanisms, allowing us delve into the internal mechanism of SS2D through the visualization of its weight matrices.
Given the input image shown in Figure (a), illustrations of four scanning paths in SS2D are presented in Figure 9 (d). The visualizations of the corresponding attention maps, calculated using QK" and (Q O w) (K <</)" are shown in Figure (e) and Figure (g) respectively. These results underscore the effectiveness of the proposed scanning approach (i.e., Cross-Scan) in capturing and retaining the traversed information, as each row in a single attention map corresponds to the attention between the current patch and all previously scanned foreground tokens impartially. Additionally, in Figure 9 (f), we showcase the transformed activation maps, where the pixel order corresponds to that of the first route, traversing the image row-wise from the upper-left to the bottom-right.
By rearranging the diagonal elements of the obtained attention map in the image space, we de- rive the visualization results shown in Figure 9 (b) and Figure 9
(c) corresponding QK and (Q @ w) (K/ (<) respectively. These maps illustrate the effectiveness of VMamba in accurately distinguishing between foreground and background pixels within an image.
Moreover, given selected patch the query, we visualize the corresponding activation map by reshaping the associated row in the attention map (computed by QK or (Q O w) (K/w) T) This reflects the attention score between the query patch and all previously scanned patches. To obtain complete visualization for a query patch, we collect and combine the activation maps from all four scanning paths in SS2D. The visualization results of the activation map for both QK' and shown in Figure 10. We also visualize the diagonal elements of attention
(Q O w) (K /w) maps computed by (Q & w) w) (K/w) where all foreground objects are effectively highlighted and separated from the background.
16

---

Route
Route
Route 3
Route 4
M
M
(a)
Input Image
(e)
Attention Map QKT
SF
Aligned
Diagonal of QK"
Attention Map QKт
(g)
Altention Map of (Q O w)
Diagonal of (2 O
cc)
Figure 9: Illustration of the attention maps obtained by SS2D.
(x2) Imput Image
Acttvivatiion Map (c)tviattooon Map
(d) Image
Activatio Map
(f Activation Map of O
(g) Diegonal of Attendion Map
Figure 10: Illustraioon of activation maps for the query patch (marked with a red star).
17

---

1.5
M
Before Tranning
Epocho
Epoch6
Epoch9
Epoch12
Epoctó0
Afeer
Figure 11: Illustration of ERF maps throughout the training process Mimila- (with EMA).
E
Detailed Experiment Settings
Network Architecture. The architectural specifications of Vanilla-VMambal are outlined in Table while detailed configurations of the VMamba series are provided Table The Vanilla-VMamba series is constructed using the vanilla VSS Block, which includes multiplicative branch and does not have feed-forware network (FFN) layers. In contrast, the VSS Block in the VMamba series removes the multiplicative branch and introduces FFN layers, Additionally, we provide alternative architectures for VMamba at Small and Base scales, referred to as VMamba-S[s 1/20] and VMamba- [[s]/ respectively. The notation 'szly' indicates that the sem-ratio is set and the number of layers in stage 3 is set to y. Consequently, the versions presented in Table can also be referred to as VMamba-S[ss S[s2115 and "Mamba-B((22) B[s2/15].
Experiment Setting. The pypper-parameter for training VMmmba ImageNet inherited from Swin [3), except for the parameters related to drop_ path_rate and the exponential moving average (EMA) technique. Specifically, VMamba- T/S/B models trained from scratch for 300 epochs, with a 20-epoch warm-up period, using a batch size of 1024. The training process utilizes the Adam optimizer (38 with betas
(0.9, 0.999). initial learning rate of 10-3 weight decay of 0.05, and a cosine decay learning rate scheduler. It is noteworthy that this not the optimal setting for VMamba. With learning rate of 10-3 the Top-1 accuracy of VMamba-T can reach 80.7%.
Additional techniques such as label smoothing (0.1) and EMA (decay ratio of 0.9999) are also applied. The drop_path_ree path_ratio is set to 0.2 for Miailla-VMamba-TT and VMamba-T, 0.3 for Vanilla- MMmbaa-S, (Mamba-S[s2/11) and VMamba-S[ S[/20)] 0.6 for Vanilla- VMamba-B and VMamba- ([smbaa and 0.5 for "Mamba-B[s122)) No additional training techniques employed.
Throughput Evaluation. Detailed performance comparisons with various models are presented in T Throughput (referred as TP. ) was assessed on an A 1100 GPU paired with an AMD EPYC 7542 CPU, utilizing the toolkit provided by (22. Following the protocol outlined in [66. we set the batch size 128. The training throughput (referred as Train TP.) is tested on the same device with mix-resolutionn excluding the time consumption of optimizers. The batch size for measuring training throughput is also set 128.
Table provides detailed configurations of the intermediate variants in the acceleration process from Momill--Vaambe to VMamba-T.
Accelerating VMamba.
Evolution ERF. We further generate the effective receptive field (ERF) maps throughout the training process for Vanilla- Manilla-VMamiie VMamba-T. These maps intuitively illustrate how VMamba's pattern of ERF evolves from being predominantly local to predominantly global, epoch by epoch.
F Performance of the VMamba Family on Downstream Tasks
In this section, we present the experimental results of Vanilla- VMamba and VMamba on the MSCOCO and ADE20k datasets. The results summarized in Table 7 and Table respectively.
For object detection and instance segmentation, we adhere to the protocol outlined by Swin (36) and construct our models using the mmdetection framework 33. Specifically, we utilize the AdamW optimizer (38) and fine-tune the classification models pre-trained on ImageNet- IK for both and 36 epochs. The learning rate is initialized 10 and decreased by a factor of 10 at the 9-th

---

Architectural overview of the Vanilla-VMambo series. Down-sampling is performed through patch merging [36 operations in stages 1. and 3. The term Linear refers linear layer, while DWConv denotes depth-wise convolution 23 operation. The proposed 2D sDsselective-scan is labeled as SS2D.
Table 3:
layer name output size
"aaill-VManibe
Vanilla-VMnnbe-S
Vanilla- VMamba-B
112x112
4x4, 96, stride
4x4, strido
conv 4x4, stride
vanilla VSSBLock
vanilla VSSBLock
vanilla VSSBLock
Linear 96 - 2x96
Linear 96 - 2x96
Linear 128 - 2x 128
56x56
DiConv 3x 2x96
DWConv 3x 2x96
TWConv 3xc3, 2x128
Linear 2x96 96
8320, dim 2x96
x2
Linear 2x - 96
S822, dim 2x
x2
Linear 2x8 128
SS2D, dim 1x8
x2
hultsspiicattvee
Mulesplicativv Linear 2x - 96
Multiplicatti9e Linear 2x 128 - 128
Linear 2x96 -+ 96
patch merping
patch merging
patch merging -> 256
vanilla VSSBLock
vanilla VSSBLock
vanilla VSSBLock
Linear 192 1x.92
Linear 192 - 192
Linear 256 2x256 DWConv 3x
stage?
28x228
DWConv 3x33,
DWCony 3x.3, 2xx192
Linear 2x 192 192
SS20, dim 192
x2
Linear 2x
9320, dim 2x 192
192
x2
Linear 256 256
$920, dim 2x256
x2
Multsppiccattvee
Multspli
Nultiplicatiiv
Linear 2x2192 192
Linear 2x2192 + 192
Linear 2x256 256
putch merging > 384
patch merging > 384
patch merging 512
vanilla VSSBLock
vanilla VSSBLock
vanilla VSSBLock
384
2x384
512 2x512
stage
DWCcov 3x.3, 2x384
DiCaov 3x 2x384
DVConv 3x3,, 2x512
2820, dim 384
3820, dim 384
x27
S320. dim 2x512
x27
Linear 2x3
Linaar 2x3384
Multaplicativee
Moltiplicative
Hultiplicatare
Linear 2xx384
Linear 2x384
Linear 2x:12
putch merging - 768
patch merging 768
pulch merging 1024
vanilla VSSBLock
vanilla VSSBLock
vanilla VSSBLock
Linear 768 - 2x6
Linear 768 - 2x668 DWCony 26688
Linear 1024 - 2x1024
stuge
7x7
DWCcar 3x3, 3x 268
DVConv 3x3,, 2x 1024
9520, dim 2x768
2x
S320, 2x1144
Linear 768 -> 768
Linear 2x 768 - 768 mullippicatiiee
Linear 2x 1024 1024 Multiplicatiie Linear 2x 1024
Multiplicativve Linear 2x768
average pool, 1000-d softmax
Param. (M)
22.9
44,4
76.3
FLOPSs
5.63xx10*
11,23xx 10"
18.02
and 11-th epoch. We incorporate multi-scale training and random flipping with batch size of 16, following established practices for object detection evaluations.
For semantic segmentation, we follow Swin [6)] and construct a UperHead [S
network on top of
the pre-trained model using the MMSegmentation library (4]. We employ the AdamW optimizer (38) and set the learning rate to 6 x 10-5, The fine-tuning process spans a total of 160k iterations with batch size of 16. The default input resolution is 512 x 512.
G Details of VMamba's Scale-Up Experiments
Given Mamba's exceptional ability in efficient long sequence modeling, we conduct experiments to assess whether VMamba inherits this characteristic. We evaluate the computational efficiency and classification accuracy of VMMmba with progressively larger input spatial resolutions. Specifically, following the protocol in XCIT D. we apply VMamba, trained on 224 x 224 inputs, to images with resolutions ranging from 288 x 288 to 768 768. We measure the generalization performance in terms of the number of parameters, FLOPs, throughput during both training and inference, and
19

---

Table 4: Architectural overview of the VMamba series.
Tayer nome
oupur sine
VMamba-T
VMamba-8
VMimba-R
dom
112x112
cone 5t:de 2, LayerNoem, GRUA, comr 3x3 strice 2, Layerhorm
MaaiBliaokaaa-ttttiooo sip-ration)) tatorr 5880, dim den ratio x86 Eineer ane-ratto x96 BES sip rrato
VSSBlocktann watieaadddeaarraddndn. elp-rationl)
VSSBLock wwt.e--rttoo,, aip-xution))
Lineer
stag:
56x:5
x96
x2
Bobouz 3x33, 5920, dim cem-ratio x96
X2
DNlDoay 3x33,
8820, dim con-ratto
X2
Lineer cae-ratto x96 - 96
Lineer sar-ratio 128 128
FFM mip-rati x96
PP8 aip-ratii x128
outy atrdd LayerNonn
Vasai oetteee-ratttoll oct, itp-ratioon ooo))
VSSiBllcktemm-ree elp-ratteca)
VSsttLnckeeee wtt.onccteeerrtioaaaaa ratioss, alp-rationd)
Linear 192 ratio x192
Linear
vatio DiAton 363, aee-ratie x192
Linear
ratio x256
stag 2
23 28
biktem 3 xxm-matso x
s220, dim ram-ratip--.. x192
x=2
5820, dim san crstio x x92
X2
Dillowy 3x3, amm-ratto x256
se2n, dis sam-ratse x256
x2
Linear cen rratto 192 192
Linear dex ratio 192 - 192
Linear esx-ratio x25 - 256
pntnlp-ratt 192
1FM Prrotp-ratoo X12
FPR al p-rabio x226
arale LayehNun
VeSBLockkoox crttoo-ttooolo raprratioo))
VSSiBloctteco ratioed, elp-ratiowd)
VSSBLockiere raclosd, mip alp-ratiow))
Linear 394 zatio x384
Linear 394 son- zatio x384
Liseur 512 ase- rario x512
stage
14x14
Dittas 3x33, ose. rasio X354
3920, dmn sew ratio x38
DNDoan 3x33, oon. rario x354
8520, din sex xario x388
DVDowy 3x.,, rotio x512
8920, dim rario x512
x15
Linser aun-ratio x 384 384
Linear see-ratie a 384 384
Liseae san- catso x512 512
FP mip-zatiio 384
F98 puttootio X384
283 alg-ratio X512
3X3 ande LayenNom
VSSBI saslllasconnnttiooee mip-rwtiond) Linner 768t - aan-rattn x766
VissBlick elp-eationd)
VSSBLack alp-rattond)
Lineer 768 - aes-ratto 766k
Linear 1004 - ser-ratin x 1004
stage
7x7
nattore fo aen-ratto x768
s820, din ooa zatio x768
X2
naConw fo aoe-racio 768
se2n, dmm son vatio 768
X2
mitooy 3x sem-ratin 1034
8820, dim ose gasdo x 1024
X2
Linear san ratio x768 7668
Linear son ratio 768 766
Lincor den retio x x102 1024
FFM ropoooooo x7188
FFN rrrtio X768
7P9 rap-r X1004
Lxl
uvergge pool, 1000-4 fo. soflmax
Purmm (M)
30.2
50.1
88.6
FLOPs
4'1
6.72x
15.36x100
Table 5: Details of accelerating VMamba.
Model
d_state sen-ratio
DWCoav
multiculative branch
layers numbers
FFN
Params
(M)
FLOPS]
(G)
(img/s)
TP
Train
(img/s)
Top-T
(%)
Vanilla- VMamba-
2.0
[222,22]
22.9M
5,63G
426
138
82.17
Step(a)
16
2.0
[2,2.9.2]
22.9M
5.63G
467
165
82.17
Stepth)
2.0
[2.2.922]
22.9M
5.63G
464
184
82.17
Stepte)
16
2,0
[2,2,922]
22.9M
5.63G
638
195
82.17
Step(d)
16
2.0
12.2.2.2
<
29.0M
5.63G
813
248
81.65
Steptd.1)
16
1.0
12.2.2.2
22.9M
4.02G
1336
405
81 1..05
Steptd.2)
1.0
[..2.5.2]
/
28. 2M
5.18G
1137
348
82.24
Steple)
16
1.0
[2,2,5
/
26.2M
4.86G
1179
360
82.17
Stepte.
1.0
[2.2.5.2]
/
26.3M
4.87G
358
82.31
Stepie.2)
1.0
[2.2,5.2]
/
25.6M
3.98G
1942
647
81.87
Step(i)
2.0
122
7
30.7M
4.86G
464
82.49
Stepig)
1.0
[222,82
7
30.2M
4.9GG
1686
571
82.60
the top-1 classification accuracy on ImageNet-IK, IK. We also conduct experiments under the 'linear tuning' setting, where only the header network, consisting of a single linear module, is fine-tuned from random initialization using features extracted by the backbone models.
According to the results summarized in Table 9 VMmmba demonstrates the most stable performance across (i.e,, modest performance drop) different input image sizes, achieving a top-1 classification accuracy of 74.7% without fine-tuning (79.2% with linear tuning). while maintaining relatively high throughput of 149 images per second at an input resolution of 768 x 768. In comparison, Swin (36] achieves the second-highest performance with a top-1 accuracy of 73.1% without fine- tuning (77.5% under linear tuning) at the same input size, using scaled window sizes (set as the resolution divided by 32). However, its throughput significantly drops to 53 images per second. Furthermore, ConvNeXt [3]] maintains relatively high inference speed (i.e., throughput of 103 images per second) at the largest input resolution. However, its classification accuracy drops to 69.5% when directly tested on images of size 768 x 768, indicating its limited adaptability to images with large spatial resolutions. Deit-S also shows a dramatic performance drop, primarily due to the interpolation used in the absolute positional embedding.
20

---

Table Performance comparison on ImageNet- 1K with an image size of 224. f indicates that Vim is
trained solely in float32 in practice, with a training throughput of 232.
[69]
Image Size
Params
FLOPs
TP.
Train TP.
Top-1 (%)
Model
(M)
(G)
(img/s)
(img/s)
DeîT-S [57
224
22M
4.6G
1761
2404
79.8
DefT-B B [57
224
86M
17.56
503
1032
81.8
ConvNeXt-T
224
29M
4.5G
1198
702
82.1
ConvNeXt-S
37
224
50M
8.7G
684
445
83.1
ConvNeXI-B
224
89M
15.4G
436
334
83.8
HiViT-T 66
224"
19M
4.6G
1393
1304
82.1
HiViT-S 66
224
38M
9.1G
712
698
83.5
HiVIT-B 66
224
66M
15.8G
456
544
83.8
Swin-T
36
224
28M
4.5G
1244
987
81.3
Swin-S
36
224
50M
8.7G
718
642
83.0
Swin-B 36
224
15.5G
458
496
83.5
XCTT-S12/16
224
26M
4.9G
1283
935
82.0
XCIT-S24/16
224
9.2G
671
509
82.6
XCIT-M24/16
224
84M
16.2G
423
385
82.7
S4ND-ConNNeXI-T
224
30M
5.2G
683
369
82.2
S4ND-ViT-B 44
224
89M
17.1G
398
400
80.4
Vim-S
224
26M
5.3G
811
344
80.5
Vanilla- Mamba-T
224"
23M
5.6G
638
195
82.2
Vanilla- MMamba-S
224
44M
11.2G
359
83.5
Vanilla-VMaminn Mamba-B
224
76M
18.0G
268
84
83.7
VMamba-T
224
30M
4.9G
1686
571
82.6
VMamba-S S[$2/15]
224
50M
8.7G
877
314
83.6
VMmmba VMamba-B(s2118 B[s2|15]
224
89M
15.4G
646
247
83.9
VMamba S[s1/20]
224
49M
8.6G
1106
390
83.3
VMamba B[s1/20]
224
87M
15.2G
827
313
83.8
Notably, VMamba displays linear increase in computational complexity, as measured by FLOPs, which is comparable to CNN-based architectures. This finding aligns with the theoretical conclusions drawn from selective SSMs [17
H
Ablation Study
H.1
Influence of the Scanning Pattern
In the main submission, we validate the effectiveness of the proposed scanning pattern (referred to as Cross-Scan) in SS2D by comparing it to three alternative image traversal approaches, i.e., Unidi-Scan, Bidi-Scan, and Cascade-Scan (Figure 12}. Notably, since Unidi-Scan, Bidi-Scan, and Cross-Scan are all implemented in Triton, they exhibit minimal differences in throughput. The results in Table I0 indicate that Cross-Sean demonstrates superior data modeling capacity, as reflected by its higher classification accuracy. This advantage likely stems from the two-dimensional prior introduced by the four-way scanning design. Nevertheless, the practical implementation of Liscade-Scan is significantly constrained by its relatively slow computational pace, primarily due the inadequate compatibility between selective scanning and high-dimensione data, which is further affected by the multi-step scanning procedure.
Figure 13 indirectly demonstrates that among analyzed scanning methods, only Bidi-Scan, Cascade Scan, and Cross-Scan showcase global ERFs. Moreover, only Cross-Scan and Cascade-Sea exhibit woodimmessional (2D) priors. It also worth noting that DWConv (23) plays a critical role in establishing 2D priors, thereby contributing to the formation of global ERFs.
H.2 Influence of the Initialization Approach
In our study, we adopted the initialization scheme originally proposed the SS2D block S4D [19]. Therefore, is necessary to investigate the contribution of this initialization method to the effective-
21

---

Table 7: Object detection and instance segmentation results on COCO dataset. FLOPs are calculated using inputs of size 1280 800. Here, AP and APm denote box AP and mask AP, respectively. "1x" indicates models fine-tuned for 12 epochs, while "3xMS" signifies the utilization of multi-scale training for epochs.
Mask R-CNN 1x schedule
Backhone
Apb
APp
AP
Apm
AP
AP
Params
FLOPPs
Swin-T
42.7
65.2
46.8
39.3
62.2
42.2
48M
267G
ConvNeXt-T
44.2
66.6
48.3
40.1
63.3
42.8
48M
262G
Vanilla- VMamba-T
46.5
68.5
50.7
42.1
65.5
45.3
42M
286G
VMamba-T
47.3
69.3
52.0
42.7
66.4
45.9
50M
271G
Swin-S
44.8
66.6
48.9
40.9
63.4
44.2
69M
354G
ConvNeXI-S
45.4
67.9
50.0
41.8
65.2
45.1
70M
348G
Vanilla- V
48.2
69.7
52.5
43.0
66.6
46.4
64M
400G
VMamba-S
48.7
70.0
53.4
43.7
67.3
47.0
70M
349G
Swin-B
46.9
42.3
107M
496G
ConvNeXt-B
47.0
69.4
51.7
42.7
66.3
46.0
108M
486G
Vanilla- VMamba-B
48.6
70.0
53.1
43.3
67.1
46.7
96M
540G
VMamba-B
49.2
71.4
54.0
44.1
68.3
47.7
108M
485G
Mask R-CNN 3
MS schedule
Swin-T
46.0
68.1
50.3
41.6
65.1
44.9
48M
267G
ConvNeXt-T
46.2
67.9
50.8
41.7
65.0
44.9
48M
262G
Vanilla- VMamba-T
48.5
70.0
52.7
43.2
66.9
46.4
42M
286G
VMamba-T
48.8
70.4
53.5
43.7
67.4
47.0
50M
271G
Swin-S
48.2
69.8
52.8
43.2
67.0
46.1
69M
354G
ConvNeXt-S
47.9
70.0
52.7
42.9
66.9
46.2
70M
348G
Vanilla- VMamba-S
49.7
70.4
54.2
44.0
67.6
47.3
64M
400G
VMamba-S
49.9
70.9
54.7
44.2
68.2
47.7
70M
349G
Unidi-Scan
Cascade- Scann Row and Cal
Bidi-Scan
Soan
M
Figure 12: Illustration of different scanning patterns for selective scan.
ness of VMamba. To explore this further, we replaced the default initialization with two alternative methods: random initialization and zero initialization.
For both initialization methods, we set the parameter D in equation To vector of all ones, mimicking a basic skip connection (thus we have y = Ch Du). Additionally, the weights and biases associated with the transformation to the dimension D., (which matches the input size), are initialized as random vectors. In contrast, Mamba [17] employs more sophisticated initialization.
22

---

Table 8: Semantic segmentation results on ADE20K using UperNet [63
evaluate the performance FLOPs are calculated with input sizes of 512 x 2048. "SS" and "MS" denote single-scale and multi-scale testing, respectively.
of semantic segmentation on the ADE20K dataset with UperNet
method
crop size
mloU (SS)
mloU (MS)
Params
FLOPs
Swin-T
5122
44.5
45.8
60M
945G
ConvNeXI-T
5122
46.0
46.7
60M
939G
Vanilla- VMamba-T
512²
47.3
48.3
55M
964G
VMamba-T
5122
48.0
48.8
62M
949G
Swin-S
5122
47.6
49.5
81M
1039G
ConvNeXt-S
5122
48.7
49.6
82M
1027G
Vanilla- SMamba-S
5122
49.5
50.5
76M
1081G
VMamba-S
5122
50.6
51.2
82M
1028G
Swin-B
5122
48.1
49.7
121M
1188G
ConvNeXt-B
5122
49.1
49.9
122M
1170G
Vanilla- -
5122
50.0
51.3
110M
1226G
VMMmba-B
5122
51.0
51.6
122M
1170G
w/ DWConv
w/o DWNConv
1.0
Before trining
0.8
0.4
After training
0.2
0.0
Unidi-Scan
Bidi-Scan
Cascade-Scan
Cross-Scan
Unidi-Scan
Bidi-Scan
Cascade-Scan
Cross-Scan
Figure 13: The visualization of ERF for models with different scanning patterns.
The main distinction between random and zero initialization lies in the parameter A in equation 6 which is typically initialized as a HiPPO matrix in both Mamba [17 19 and our implementation of VMamba. Given that we selected the hyper-parameter d_ d_state to be 1, the Mamba initialization for (A) can be simplified to all zeros, which aligns with zero initialization. In contrast, random initialization assigns random vector log( ()). We choose to initialize log(A) rather than A directly to keep A near the all-ones matrix when the network parameters are close to zero, which empirically enhances the training stability.
The experimental results in Table indicate that, at least for image classification with SS2D blocks, the model's performance is not significantly affected by the initialization method. Therefore, within this context, the sophisticated initialization method employed Mamba [17] can be substituted with simpler, more straightforward approach. We also visualize the ERF maps of models trained with different initialization methods (see Figure 14], which intuitively reflect SS2D's 's robustness across various initialization schemes.
H.3 Influence the d_state Parameter
Throughout this study, we primarily set the value of statee to to optimize VMamba's computa- tional speed. To further explore the impact of d_ ststae the model's performance, conduct series of experiments.
As shown in Table 12 with all other prrametssssssers fixed, we increase d_ d_state from to 4. This results in slight improvement in performance but substantial decrease throughput, indicating significant negative impact the VMamba's 's computational efficiency. However, increasing d_ state
23

---

1.0
Before training
0.8
0.8
After Aftortraining
0.2
0.0
Mamba-Init
Rand-Init
Zero-Init
Figure 14: The visualization of ERF of VMamba with different initialization.
to 8, while reducing sam sam-ratio to maintain computational complexity, leads to improved accuracy. Moreover, when d_state further increased to 16, with sam-ratio set to I, performance declines.
These findings suggest that modest increases in d_state may not necessarily lead to better per- formance. Instead, selecting the optimal combination of d d_state and ssm-ratio crucial for achieving good trade-off between inference speed and performance.
H.4 Influence of ssm-ratio, mlp-ratio, and layer numbers
In this section, we investigate the trade-off's among sam-ratio, layer numbers, and
Experimental results shown in Table 13 indicate that reducing ssm-ratio significantly decreases performance but substantially improves inference speed. Conversely, increasing layer numbers enhances the performance while slowing down the model.
As the hyper-parameter ssm-ratio represents the dimension used by the SS2D module, the trade-off between rsm-ratio and layer numbers can be interpreted a balance between coanel-mixing and token-mixing [56] Furthermore, we reduce mlp-ratio from 4.0 to 2.0 and progressively increase ssm-ratis ratio to maintain constant FLOPs, as shown in Table [4 The results presented in Tables 13 and 14 highlight the importance of an optimal combination of ssm-ratio, mlp-ratio, and layer numbers for constructing a model that balances effectiveness and efficiency.
H.S Influence the Activation Function
In VMamba, the SiLU [4] activation function is utilized to build the SS2D block. However, experimental results in Table 13 show that VMamba maintains robustness across different activation functions. This implies that the choice of activation function does not substantially affect the model's performance. Therefore, there flexibility to choose an appropriate activation function based on computational constraints or other preferences.
24

---

Table Comparison generalizability inputs with increased spatial resolutions. The throughput and training throughput are measured with a batch size of 32 using PyTorch 2.0 on an A100 GPU paired with an AMD EPYC 7542 CPU. Unlike throughput, the model's forward pass, loss calculation, and backward pass included in calculating the training throughput, with mixed precision. We re-im re-implemented the Hi as the checkpoint for HiVIT-T has not been released. f denotes that the batch size & 16 due to out-of out-of-memory memory (OOM) issues.
Image Size
Param.
FLOPS (G)
TP:
Train TP. (img/k)
Top-1 acc. (%)
Top-1 acc. (%) (w/ Winear tuning)
Model
(M)
(img/s
SSM-Based
224
30M
4.91G
1490
418
82.60
82.64
288
30M
8.11G
947
303
82.95
83.03
384
30M
14.4IG
566
187
82.41
82.77
VMambu-Tiny
512"
30M
25.63G
340
121
80.92
81.88
640
30M
40.04G
214
75
78.60
80.62
768
30M
57.66G
53
74.66
79.22
31M
4.86G
1227
399
82.49
82.52
288
31M
8.03G
761
255
82.81
82.93
384
31M
14,27G
452
155
82.51
82.74
VMamha- VMamba-Tiny(=222 [=255
5122
31M
25.38G
272
100
81.07
82.02
640
31M
39.65G
170
60
79.30
81.02
7682
31M
57.09G
117
42
76.06
79.69
224'
23M
5.63G
628
189
82.17
82.09
288'
23M
9.30G
390
117
82.74
82.76
384
23M
16.53G
212
65
82,40
82.72
Vanilla- VMamba- Tiny
5122
23M
29.39G
138
53
81.05
81.97
640
23M
45.93G
87
27
78.79
80.71
768
23M
66.14C
52
18
75.09
79.12
Transformer-Bin
224
28M
4.51G
1142
769
81.19
81.18
288
28M
7.60G
638
489
81.46
81.62
3842
28M
14.05G
316
268
80.67
81.12
Swin-Tiny
5122
28M
26.65G
176
131
78.97
80.21
640
28M
45.00G
88
68
76.55
78.89
768
29M
70.72G
53
38
73.06
77.54
224"
26M
4.87G
1127
505
81.87
81.89
288
26M
8.05G
724
462
82.44
82.44
3842
26M
14,31G
425
308
81.84
82.21
XCIT-S12/16
5122
26M
25.44G
244
185
79.80
80.92
640
26M
39.75G
158
122
76.84
79.00
768
26M
57.24G
111
87
72.52
76.92
224
19M
4.60G
1261
1041
81.92
81.85
288
19M
7.93G
750
614
82.45
82.42
19M
15.21G
388
333
81.51
81.91
HiViT-Tiny
20M
30.56G
186
150
79.30
80.49
640
20M
$4,83G
93
71
76.09
78.58
7682
20M
91.41G
55
371
71.38
76.47
224
22M
4.61G
1573
1306
80.69
80.40
288
22M
7.99G
914
1124
80.80
80.63
3842
22M
15.52G
502
697
78.87
79.54
DeiT-Small
5122
22M
31,80G
261
387
74.21
76.91
640
23M
58.17G
149
244
68.04
73.31
768
23M
98.70G
90
156
60.98
69.62
ConvNet-Based
2242
29M
4.47G
1107
614
82.05
81.95
2882
29M
7.38G
696
403
82.23
82,30
3842
29M
13.12G
402
240
81.05
81.78
ConvNeXii-Ton Tiny
29M
23.33C
226
140
78.03
80.37
640
29M
36.45G
147
90
74,27
78.77
7682
29M
52.49G
103
63
69.50
76.89
25

---

Table 10: The performance of VMamba-T with different scanning patterns.
Params
FLOPs
TP.
Train TP.
Top-1
Model
(M)
(G)
(img/s)
(img/s)
(%)
VMamba dwconv
Unidi-Scan
30.2M
4.91G
1682
571
82.26
Bidi-Scan
30.2M
4.91G
1687
572
82,49
Cascade-Scan
30.2M
4.91G
998
308
82.42
Cross- -Saan
30.2M
4.91G
1686
571
82.60
VMamba w/o dwconv
Unidi-Scan
30.2M
4.89G
1716
578
80.88
Bidi-Scan
30.2M
4.89G
1719
578
81.80
Cascade-Scan
30.2M
4.90G
1007
309
81.71
Cross-Scan
30.2M
4.89G
1717
577
82.25
Table 11: The performance of VMamba-T with different initialization.
Params
FLOPs
TP.
Train TP.
Top-1
initialization
(M)
(G)
(img/s)
(img/s)
acc. (%)
mamba
30.2
4.91
1686
571
82.60
rand
30.2
4.91
1682
570
82.58
zero
30.2
4.91
1683
570
82.67
Table 12: The performance of VMamba- with different d_state.
Params
FLOPs
Train TP.
Top-I
d_ state asm-ratio
(M)
(G)
(img/s)
(img/s)
acc. (%)
2.0
30.7
4.86
1340
464
82.49
2
2.0
30.8
4.98
1269
432
82.50
4
2.0
31.0
5.22
1147
382
82.51
8
1.5
28.6
5.04
1148
365
82.69
16
1.0
26.3
4.87
1164
358
82.31
Table 13: The performance of VMamba-T under different combination of ssm-ratio and layer numbers.
layer numbers
Params (M)
FLOPs
TP.
Train TP.
Top-1
BBm sem-ratio
(G)
(img/s)
(img/s)
(%)
[2,2,5.2]
30.7
4.86
1340
464
82.49
1.0
[2,2,5,2]
25.6
3.98
1942
647
81.87
1.0
[2.2.8.2]
30.2
4.91
1686
571
82.60
Table 14: The performance of VMamba under differrnn combination ssm-ratio and mlp mlp-ratio.
Params
FLOPs
TP.
Train TP.
Top-I
m mlp-ratio ssm-ratio
(M)
(G)
(img/s)
(img/s)
acc. (%)
4.0
1.0
30.2
4.91
1686
571
82,60
3.0
1.5
28.5
4.65
1419
473
82.75
2.0
2.5
29.9
4.95
1075
361
82.86
Table 15: The performance VMamba-1 T with different activation functions in SS2D.
Params
FLOPs
TP.
Train TP.
Top-1 acc. (%)
activation
(M)
(G)
(img/s)
(img/s)
SiLU
30.2
4.91
1686
571
82.60
GELU
4.91
1680
82,53
ReLU
30.2
4.91
1684
82.65
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction provide a comprchensive overview of the background and motivation of this study, effectively outlining its main contributions point- by-point, thus accurately reflecting the paper's scope and significance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
We primarily focused on discussing the limitations associated with this study in section
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: The paper includes the full set of assumptions and correct proofs for each theoretical result, primarily presented in the appendix. Notably, covers the formulation of State Space Models, the discretization process, the derivation of recurrence relationships, and explanations concerning self-attention computations, ensuring completeness and accuracy in theoretical presentation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All information regarding the key contribution of this paper, i.e., the intro- duction of selective SSMs (or S6) to processing vision data, as well as architectural and experimental configurations, have been fully disclosed (to the extent that it affects the other
main claims and/or conclusions of the paper). Furthermore, the implementation components within the proposed VMamba framework, such Vision Transformers and the parallelized Selective Sean operation, facilitated by the plenty of support available from existing open-source resources within the community.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce
28

---

reproducible, they the model (e.g., with an spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The supplementary material submitted with the manuscript includes open access to all source code and scripts necessary to faithfully reproduce the main experimental results. Instructions for running the code are also provided within the scripts.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
CoodeSubiiiieeee wetiebbmissiie aessonPollin Pollicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies detailed experimental configurations in Section in Appendix, providing readers with essential information to comprehend the results. Following established conventions the field of vision backbone models, the evaluation protocol encompasses standard practices commonly found in the relevant literature, ensuring readers can refer established methodologies.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
29

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not include an analysis of the statistical significance of the experiments mainly due to the prohibitively expensive training cost of vision backbone models and our limited computing resources. However, we have provided the code, hyperparameters, and random seeds used in our experiments to facilitate the reproducibility of our findings. We would like to point out that, due to the extensive amount of training data, the statistical patterns of the experiment results are likely to remain consistent across different trials. Consequently, reporting error bars other information about statistical significance is not common practice in studies developing deep vision backbones.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етrоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were carried out on 8 A100 GPU server, as detailed at the beginning of the experiment section (Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/::////uuus.eeooo http:///wwrrgecco/oaa/a///eeeeea
Answer: [Yes]
30

---

After carefully reviewing the referenced document, we certify that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper primarily focuses on vision backbones trained using publicly available datasets that have undergone thorough validation. While the vision backbone itself is not directly applicable to everyday scenarios, it serves as a neutral and valuable toolkit for further development and research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The proposed models are vision backbone networks trained on benchmark datasets such as ImageNet- 1K, MSCOCO, and ADE20K. These datasets have been exten- sively used in the computer vision community and have undergone comprehensive safety risk assessments.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
31

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In the paper, we specified the datasets and code sources used (e.g.. mmdet), and provided appropriate citations in the reference section. Additionally, we ensured transparency by including the sources of any modified code files, making the changes traceable.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included the code, along with detailed usage instructions, in the supplementary materials. After the review process is completed, we will make the code publicly available to the community.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
32

---

This study does not involve any crowdsourcing experiments or research with human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing experiments research with human subjects were involved in this study. All experiments were conducted using code and GPU servers.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
33


Reviews:
[{"review_id": "AGIG6PRHJ7", "reviewer": "Reviewer_dvTH", "summary": "### Summary\n\nThis paper proposes VMamba, which adopts the recently proposed selective linear state space model, Mamba, in the domain of computer vision. The paper evaluates variants of VMamba on tasks such as image classification, object detection, and semantic segmentation. To improve performance and efficiency, VMamba incorporates several architectural and implementation enhancements.\n\n---\n\npost-rebuttal: score 6 -> 7", "questions": "### Questions\n\n1. Is positional embedding used when encoding the patches? Apologize if this is already state somewhere in the paper.\n2. If I understand correctly, Figure 3 for section 4.3 shows that performance improved with smaller d_state and expand ratio. This is quite surprising since one might expect degrading performance when network capacity is reduced. Could you provide any insights into this phenomenon?", "limitations": "Yes, the author adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Strengths\n\n- The writing is simple and clear, quite accessible to readers.\n- After implementing enhancements, VMamba achieves good performance - computational efficient and quantitatively well-performed.\n- Additional analysis such as the effective receptive field and relationship between attention and the updates in state space are insightful.", "weaknesses": "### Weaknesses\n\n- As an architecture exploration paper I don’t see many weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer dvTH\n\nWe thank the reviewer for the constructive comments and are glad they appreciate the performance of VMamba. Below, we clarify the reviewer’s concerns regarding the detailed structure and the influence of hyper-parameters on VMamba.\n\n### **Usage of Positional Embedding**\n\nTo clarify, VMamba does not use positional embedding. Sorry for any confusion caused, and we will make this clear in Section 4.1 Network Architecture (lines 129-136) of the revised version as follows: \n\n\"Subsequently, multiple network stages are employed to create hierarchical representations\"\n$\\rightarrow$ \n\"Without further incorporating positional embedding, multiple network stages are employed to create hierarchical representations.\"\n\n### **Explanation of Performance Improvement**\n\nIn step (e) shown in Figure 3 for Section 4.3, we manage to save parameters and FLOPs by reducing the expansion ratio and eliminating the entire multiplicative branch. This allows us to increase the number of layers from [2,2,2,2] to [2,2,5,2], resulting in the observed performance improvement. Similarly, in step (g), lowering the expansion ratio enables us to increase the depth of the model with additional layers. For step (f), the performance improvement is due to the larger expansion ratio and the addition of extra DWConv blocks. By using a smaller d\\_state value, we keep parameters and FLOPs comparable. We will provide more details on these points in the revised version. \n\n**Influence of d\\_State.** In Section H.3 of the Appendix, we explore the impact of adjusting the d\\_state parameter on VMamba. Table 12 shows that increasing d\\_state from 1 to 4 yields only marginal performance gains while significantly reducing throughput, indicating a substantial negative impact on VMamba's computational efficiency. To mitigate this, we propose lowering the ssm\\_ratio parameter to reduce overall network complexity. We find the best performance at (d\\_state=8, ssm\\_ratio=1.5).\n\n**Influence of ssm\\_ratio.** We also analyze VMamba's sensitivity to the ssm\\_ratio parameter, with results presented in Table 13 of Appendix H.4. The results clearly indicate that lowering the ssm\\_ratio significantly reduces performance but also greatly increases the inference speed. On the other hand, adding more layers boosts performance but also decelerates the model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SlFHoVKWEr", "reviewer": "Reviewer_ZfkW", "summary": "This paper transplants the Mamba (Selective State Space Model), a linear complexity model originally designed for 1D language processing, into VMamba to process image data. It introduces the 2D selective scan and various acceleration techniques to facilitate the modeling of 2D data and enhance the speed of the network. The proposed VMamba model is trained and evaluated on a number of representative downstream tasks including ImageNet-1K classification, COCO object detection, and ASE20K semantic segmentation, and it is compared with strong baselines. A range of analyses and visualizations on the theoretical perspectives, design choices, and behavior of the model are also presented.", "questions": "Could the authors explain more on the statement in Lines 153-154, “such modification prevents the weights from being input-independent, resulting in a limited capacity for capturing contextual information”?\n\nOthers: Repetitive reference entries: [50] and [51], [50] and [60].", "limitations": "The authors adequately discuss the limitations and potential societal impact of this work. This paper also points out several potential improvements and future directions, with which I highly agree.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tVMamba is one of the first papers to attempt using Mamba, one of the most efficient and performant linear complexity models to date, to learn visual data and demonstrate effectiveness. \n2.\tThe paper proposes a series of innovations to adapt the original Mamba's 1D sequential scanning to process 2D image data (SS2D) and increase the model's processing speed (image throughput) without compromising performance.\n3.\tIn-depth deductions, comprehensive analyses, experiments, and visualizations on design choices, theoretical aspects (e.g., the relationship between SSM and Self-Attention), and model behaviors have been presented, carrying a huge volume of insightful findings that are valuable for future research.\n4.\tThe proposed VMamba is compared to representative downstream tasks, including ImageNet-1K classification, COCO object detection, and ASE20K semantic segmentation. It shows comparable or better (and consistent) results to strong baselines (e.g., Swin, DeiT, and the concurrent Vim) and superior efficiency.\n5.\tAs a general and simple visual model, the proposed VMamba potentially carries huge extension and generalization potential, which could inspire and impact a wide range of visual research.", "weaknesses": "I didn’t find any critical weakness in this paper. Apart from some limitations that have already been mentioned by the authors, such as large-scale experiments, training strategies, and hyperparameter search, the only part that I hope the paper can show more results is the ablation of some design choices. For instance, the performance change of removing the entire multiplicative branch, where Table 5 does not show a straight ablation because of more than 1 variable change. This problem also exists in some other tables of some other design choices and hyperparameters. But again, I think neither of these weaknesses is critical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer ZfkW\n\nWe appreciate the reviewer’s thoughtful review and positive comments about our study. In the following sections, we address the reviewer’s primary concern regarding the lack of ablation on design choices and clarify several other issues raised.\n\n### **More Ablation on Design Choices**\n\nFirst of all, we would like to clarify that our primary reason for modifying multiple hyper-parameters simultaneously is to ensure that the number of parameters and FLOPs remain comparable, facilitating a fair comparison between different model variants. In Table 5 (corresponding to Figure 3 (e) in Section 4.3 of the main submission), we detail the configurations used to optimize the overall performance of VMamba, balancing both effectiveness and efficiency rather than isolating the impact of each hyperparameter.\n\nHowever, we sincerely acknowledge the importance of analyzing the significance of each individual hyperparameter and architectural design choice on the overall performance. We plan to conduct more comprehensive experiments to extend the results of experiments isolating each hyperparameter in Table 5, and include those results in future versions of this study.\n\nTo address the issue mentioned in the reviewer's comment, we have conducted additional experiments to analyze the influence of changing a single variable. The results are reported in the following table. Values for Step (e.1) and Step (e.2) are copied from Table 12 and Table 14 in the appendix, respectively, while Step (d.1) and Step (d.2) present new results obtained during the rebuttal process.\n\n| Model | d\\_state | ssm\\_ratio | DWConv | Multiplicative Branch | Layers | FFN |  Params (M) | FLOPs (G) | TP.  (img/s) | Train TP. (img/s)| Top-1 (\\%) |\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|Vanilla-VMamba-T|16 |2.0 |True |True |[2,2,9,2] |False|22.9M |5.63G |426        |138       |82.17        |\n|Step(a)         |16 |2.0 |True |True |[2,2,9,2] |False|22.9M |5.63G |467        |165       |82.17        |\n|Step(b)         |16 |2.0 |True |True |[2,2,9,2] |False|22.9M |5.63G |464        |184       |82.17        |\n|Step(c)         |16 |2.0 |True |True |[2,2,9,2] |False|22.9M |5.63G |638        |195       |82.17        |\n|Step(d)         |16 |2.0 |False|True |[2,2,2,2] |True |29.0M |5.63G |813        |248       |81.65        |\n|Step(d.1)       |16 |1.0 |False|True |[2,2,2,2] |True |22.9M |4.02G |1336 $\\dagger$|405 $\\dagger$|81.05 $\\ddagger$|\n|Step(d.2)       |16 |1.0 |False|True |[2,2,5,2] |True |28.2M |5.18G |1137 $\\dagger$|348 $\\dagger$|82.24 $\\ddagger$|\n|Step(e)         |16 |1.0 |False|False|[2,2,5,2] |True |26.2M |4.86G |1179       |360       |82.17        |\n|Step(e.1)       |16 |1.0 |True |False|[2,2,5,2] |True |26.3M |4.87G |1164       |358       |82.31        |\n|Step(e.2)       |1  |1.0 |True |False|[2,2,5,2] |True |25.6M |3.98G |1942       |647       |81.87        |\n|Step(f)         |1  |2.0 |True |False|[2,2,5,2] |True |30.7M |4.86G |1340       |464       |82.49        |\n|Step(g)         |1  |1.0 |True |False|[2,2,8,2] |True |30.2M |4.91G |1686       |571       |82.60        |\n\nDetails of accelerating VMamba. $\\dagger$ and $\\ddagger$ indicate the value is obtained from `[SERVER 1]` and `[SERVER 2]`, respectively. All other experiments are conducted on `[SERVER 0]`.\n\n\n### **Clarification of the Statement**\n\nThere is a typo in the mentioned statement, and the correct version is \"such modification prevents the weights from being input-dependent, resulting in a limited capacity for capturing contextual information\" (i.e., change from \"input-independent\" to \"input-dependent\"). We will fix this typo and conduct thorough proofreading to prevent further errors in the revised version.\n\n**Detailed Explanation of the Referred Statement.** S4ND [2] extends S4 [1] to higher-dimensional contexts through a straightforward outer product, with the essential condition being that the SSM in S4 is implemented using 'accelerated convolution'. Specifically, S4 utilizes a global convolutional operation to compute the output of the SSM, denoted as $\\mathbf{y}$, given the input data $\\mathbf{u}$ and the kernel function $\\mathbf{K} = \\mathbf{C}e^{\\mathbf{A\\Delta}}\\mathbf{B}$.\n\nEfficient computation is achieved if $\\mathbf{A}$ has an 'Normal Plus Low-Rank' (NPLR) form and $\\Delta$ is constant, enabling the low-rank approximation of $\\mathbf{K}$ in the spectral domain, allowing the convolution to be efficiently computed with Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT). Conversely, if $\\Delta$ is input-dependent or context-aware, the kernel function will no longer maintain a low-rank form in the spectral domain, leading to a substantial increase in the convolution computation time.\n\nThe efficacy of a recurrent model is significantly limited by its capacity to effectively compress context [3]. By leveraging the task of selective copying and employing Induction heads, Mamba [3] illustrates that LTI models lack content awareness. Consequently, it is concluded that a fundamental principle in developing sequence models is selectivity: the context-aware capability to emphasize or disregard specific inputs within a sequential state.\n\n### **Repetitive Reference Entries**\n\nWe will address the issues mentioned in the comment and conduct thorough proofreading to prevent further errors in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FsFsv0wrPE", "reviewer": "Reviewer_c4kg", "summary": "This paper presents VMamba, a novel vision backbone model inspired by the famous Mamba state-space sequence model. The main contribution of VMamba is its ability to achieve efficient visual representation learning with linear computational complexity. The core of VMamba is the VSS block, which incorporates the 2D-Selective-Scan module (SS2D), thereby extending the Mamba model that is a 1D selective scan good for NLP tasks. With SS2D, we can work nicely with inductive biases associated with 2D image space. \n\nVMamba's architecture consists of multiple stages with hierarchical representations (similar to ViT). The authors introduce three model sizes: Tiny, Small, and Base. The VSS blocks replace the S6 module from Mamba with the SS2D module, and further enhancements are made by eliminating unnecessary components and optimizing the architecture for better computational efficiency - using the Triton language.\n\nExtensive experiments demonstrate VMamba's promising performance across various visual perception tasks, including image classification on ImageNet-1K, object detection, instance segmentation on MSCOCO, and semantic segmentation on ADE20K. VMamba consistently achieves superior accuracy and throughput compared to existing benchmark models, showcasing its scalability and adaptability to different input resolutions and downstream tasks.", "questions": "- Could you provide more detailed comparisons with other state-space models (SSMs) used in the vision domain, such as S4ND and Vim? Specifically, how does VMamba perform in terms of accuracy, computational efficiency, and memory usage compared to these models?\n- How well does VMamba generalize to other types of visual tasks beyond image classification, object detection, and segmentation? Have you considered evaluating VMamba on tasks such as video analysis, 3D vision, or more complex scene understanding? How does it scale on these tasks?\n- How sensitive is VMamba to various hyperparameters? It would be helpful to know if specific hyperparameters are critical to achieving the reported performance and if there are guidelines or best practices for tuning them.\n- How easily can VMamba be integrated into existing deep learning frameworks and pipelines? Are there any specific requirements or modifications needed for seamless integration?", "limitations": "Authors addressed everything regarding limitations section.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* 2D-Selective-Scan Module: The introduction of the 2D-Selective-Scan (SS2D) module is a creative solution to bridge the gap between 1D selective scan and 2D vision data.\n* Comprehensive Experiments: The paper provides extensive experimental results on multiple benchmarks, including ImageNet-1K, MSCOCO, and ADE20K, demonstrating the effectiveness and robustness of VMamba across various tasks.\n* Clear Explanation: The paper is well-written, with clear explanations. The authors provide detailed descriptions of the architecture, modules, and experimental setups, making it accessible to readers.\n* Visualization: The use of visualizations, such as activation maps and effective receptive fields (ERF), helps in understanding the SS2D mechanism and the model's behavior, which is very important part of all the ablation studies.\n* Impact on Visual Representation Learning: VMamba addresses a critical issue in vision models by reducing computational complexity from quadratic to linear, which can significantly impact the field of visual representation learning.", "weaknesses": "* Limited Comparison with Other SSM-based Models: While the paper does compare VMamba with several benchmark models, it would benefit from a more detailed comparison with other state-space models (SSM) in the vision domain. Specifically, models like S4ND and Vim are mentioned, but the comparisons are somewhat brief. Providing more in-depth analysis and results would strengthen the argument for VMamba's superiority.\n* Adding more interesting works to Related Work section: There are some interesting works on neuromorphic vision and processing with SSMs that authors should cite and mention:\n\n[1] State Space Models for Event Cameras. Nikola Zubić, Mathias Gehrig, Davide Scaramuzza - CVPR 2024, Spotlight\n\n[2] Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Models. Mark Schöne, Neeraj Mohan Sushma, Jingyue Zhuge, Christian Mayr, Anand Subramoney, David Kappel - ICONS 2024\n\n* Generalization to Other Tasks: The experiments focus mainly on standard benchmarks for image classification, object detection, and segmentation. However, it is not clear how well VMamba generalizes to other types of visual tasks such as video analysis, 3D vision, or more complex scene understanding. Including some preliminary results or at least discussions on these aspects could highlight the versatility of VMamba further.\n* Clarity in Mathematical Derivations: Some of the mathematical derivations, especially in the relationship between SS2D and self-attention, are complex and may not be easily accessible to all readers. Simplifying the explanations or providing more intuitive visual insights alongside the formal derivations could enhance understanding. Also, they are not rigorously mathematically proven.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer c4kg\n\nWe appreciate the reviewer’s thoughtful review and constructive comments. In our responses, we address the following concerns: a detailed comparison with SSM-based methods, the generalizability of VMamba, sensitivity to hyper-parameters, and potential for integration into various frameworks.\n\n### **Detailed Comparison with SSM-based Methods**\n\nIn Table 1 of the main submission, we have already compared our method to S4ND [2] and Vim [4] in terms of the number of parameters, train throughput, and the Top-1 accuracy on ImageNet-1K. To provide a more comprehensive evaluation, we additionally conduct comparison on FLOPs and the memory usage, and the results are reported in the following table.\n\nMoreover, we also compare the performance (both effectiveness and efficiency) change with increasing input resolution in Figure 1 in the `attachment`. For qualitative comparison, we visualize the ERF of S4ND and Vim, and the results are shown in Figure 2 in the `attachment`.  We will include these results and the associated analysis in the revised manuscript.\n    \n| Model | Hierarchical | Params (M) | FLOPs (G) | TP. (img/s) | Test Mem. (M) | Train TP. (img/s) | Train Mem. (M) | Top-1 (\\%) |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|DeiT-S      |False |22M |4.6G  |1761 |582  |2404 |4562  |79.8 |\n|DeiT-B      |False |86M |17.5G |503  |1032 |1404 |9511  |81.8 |\n|S4ND-ViT-B  |False |89M |17.1G |398  |2221 |400  |15868 |80.4 |\n|Vim-S       |False |26M |5.3G  |811  |1055 |344 $\\dagger$ (232) |9056 $\\dagger$ (16150) |80.5 |\n|Swin-T      |True  |28M |4.5G  |1244 |3092 |987  |9798  |81.3 |\n|ConvNeXt-T  |True  |29M |4.5G  |1198 |2498 |702  |9450  |82.1 |\n|S4ND-Conv-T |True  |30M |5.2G  |683  |3945 |369  |18843 |82.2 |\n|Vanilla-VMamba-T |True  |23M |5.6G  |638  |6042 |195  |16452 |82.2 | \n|VMamba-T    |True  |30M |4.9G  |1686 |3064 |571  |12394 |82.6 |\n\n[Performance comparison between VMamba and benchmark methods. $\\dagger$ indicates the value is measured with mix-resolution while Vim does not support training with mix-resolution (values in the brackets are results obtained with fp32).]\n\n### **Additional Related Studies**\nWe thank the reviewer for bringing these inspiring studies to our attention. We will include references to these papers in the revised version.\n\n### **Versatility of VMamba**\nDue to our limited computational resources, we have focused on conducting experiments on benchmark tasks in vision modeling. However, we recognize the importance of illustrating the potential of the proposed method in more generalized tasks.\n\nA preliminary literature review of recently proposed SSM-based approaches in vision tasks, along with our private communications with researchers in the field, highlights the potential of the 2D selective scan technique (SS2D) introduced in this study. SS2D does not make specific assumptions about the layout or modality of the input data, which allows it to be generalized to various tasks. For example, SS2D can process video data by traversing a spatial-temporal plane of frame patches. To our knowledge, recent studies leveraging scanning patterns analogous to SS2D have shown success in various tasks, including image restoration and multimodal data understanding, in addition to those mentioned in the question. We will add these results to the final version and cite their works if they are published by then.\n\nDespite not being inherently prohibited, we anticipate challenges in directly migrating SS2D to diverse downstream tasks due to varying requirements. Bridging the gap between SS2D and these tasks, along with proposing a more generalized scanning pattern for vision tasks, is a promising research direction. We will include this discussion in the revised version, hopefully to provide readers with some inspiration.\n\n### **Clarity in Mathematical Derivations**\n\nDue to limited space, we have included detailed proofs in the appendix and will provide more rigorous and clearer derivations in the revised version. We also recognize the significance of providing more intuitive and accessible explanations, and will include them in the revised version.\n\n### **Sensitivity ot Hyper-parameters**\nAccording to our experience, we have not found any hyperparameter to which VMamba is particularly sensitive. This observation is also supported by the ablation results on single hyper-parameters (initialization approach in Table 11 and activation function in Table 15) as well as different combinations (Tables 12, 13, and 14) included in the Appendix. \n\nWe conducted additional experiments on the influence of the learning rate, and the results are reported in the following table. We will include this discussion in the revised version.\n\n|Model       |Params (M) |FLOPs (G) |lr | Top 1. (\\%) |\n|:--:|:--:|:--:|:--:|:--:|\n|VMamba-Tiny        |30M |4.91G |2e-3 |82.70 |\n|VMamba-Tiny $\\dagger$ |30M |4.91G |1e-3 |82.62 |\n|VMamba-Tiny        |30M |4.91G |5e-4 |82.16 |\n\n[The performance of VMamba-T with different learning rate. Results marked by $\\dagger$ is the default setting used in the submission. All the models here are trained on `[SERVER 2]`.]\n\n### **Potential of Integrating into Various Frameworks**\n\nThe core of VMamba lies in the design of the SS2D module, which aims to bridge the gap between 1D sequence scanning and 2D plane traversing, rather than specific architectural configurations. SS2D can function as an end-to-end token mixer, allowing it to be integrated into various mainstream backbone networks in computer vision.\n\nIndeed, integrating SS2D into existing frameworks requires additional considerations. One critical aspect is the numerical precision settings in the model, which significantly impact performance and computational speed. Another important factor is the inclusion of normalization layers to stabilize the training process. We will include these points in the revised version to assist researchers who may want to build upon our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Response to all

We thank the reviewers for their thoughtful reviews and constructive suggestions. We’re glad that the reviewers recognized the innovation and influence of the proposed 2D-Selective-Scan (SS2D) module, as well as the extensive experiments and thorough analysis supporting VMamba. In the following, we provide a shared response to common concerns raised by the reviewers, and also include a PDF file (referred to as the `attachment`) with additional experimental results to support our discussion. Additional results are included in the `attachment` (figures) as well as in the separate responses to each reviewer (tables).

### **Ablation Study on Hyper-parameters**

All reviewers have raised concerns regarding the influence of hyper-parameters. Due to the mismatch between our limited computational resources and the extensive range of design choices, we did not initially conduct a comprehensive ablation study on all hyper-parameters, focusing instead on a subset included in the appendix. As suggested by the reviewers, we have now conducted additional experiments on this topic. 

### **Comparison with SSM-based Models**

Another focus of the reviewers is the need for a more in-depth comparison between VMamba and SSM-based models, such as S4ND [2] and Vim [4]. We recognize the importance of these comparisons and have conducted additional experiments as suggested. The results include comparisons of FLOPs, visualizations of the Effective Receptive Fields (ERFs), and analyses of the changes in performance (both effectiveness and efficiency) with increasing input resolution.

### **Statement on Experiment Platforms**

Please note that there are slight differences between the platforms we used for the original study and this rebuttal. 

| Usage | CPU | GPU | Notation |
|:--:|:--:|:--:|:--:|
|Original Work|AMD EPYC 7542| 8 $\times$ Tesla A100 GPU|`[SERVER 0]`|
|Rebuttal (Testing)|Intel Xeon Platinum 8358|Tesla A800 GPUs|`[SERVER 1]`|
|Rebuttal (Training)|Intel Xeon Platinum 8480C|8 $\times$ Tesla H100 GPU|`[SERVER 2]`|

We investigate the influence of computational platforms on evaluation results as follows. For `[SERVER 0]` and `[SERVER 1]`, we test the generalizability to inputs with increased spatial resolutions, and the results are shown in the following table. Both training and inference throughput values are measured with a batch size of $32$ using PyTorch 2.2. The training throughput calculations include only the model forward pass, loss forward pass, and backward pass.

| Model | Image Size | Params (M) | FLOPs (G) | [SERVER 0] TP.  (img/s) |  [SERVER 0]  Train TP. (img/s)| [SERVER 1] TP.  (img/s) |  [SERVER 1]  Train TP. (img/s)|
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|VMamba-Tiny|$224^2$ |30M|4.91G |1490|418|1463|453|
|VMamba-Tiny|$288^2$|30M|8.11G |947 |303|952 |305|
|VMamba-Tiny|$384^2$|30M|14.41G|566 |187|563 |187|
|VMamba-Tiny|$512^2$|30M|25.63G|340 |121|339 |120|
|VMamba-Tiny|$640^2$|30M|40.04G|214 |75 |216 |75 |
|VMamba-Tiny|$768^2$|30M|57.66G|149 |53 |149 |53 |

We also compare the differences between VMamba-T trained on `[SERVER 0]` and `[SERVER 2]` in the following table.

| Model | Params (M) | FLOPs (G) | LR | Top 1. (\%) |
|:--:|:--:|:--:|:--:|:--:|
|VMamba-Tiny [SERVER 0]|30M|4.91G|1e-3|82.60|
|VMamba-Tiny [SERVER 2]|30M|4.91G|1e-3|82.62|

According to the results shown in the above two tables, there is only a subtle difference between the results obtained on `[SERVER 0]` and `[SERVER 1]`/`[SERVER 2]`. Therefore, we disregard the influence of computational platforms and will include results obtained with consistent machines in the revised version.

### **Citations:**
    [1] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In ICLR, 2021.

    [2] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. NeurIPS, 35:2846–2861, 2022.

    [3] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.

    [4] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In ICML, 2024.

    [5] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021.

Meta-review: This paper introduces VMamba, a new vision backbone model inspired by the Mamba state-space model, designed for efficient visual representation learning with linear computational complexity. The key innovation is the VSS block, which incorporates a 2D-Selective-Scan (SS2D) module, adapting the 1D selective scan of Mamba for 2D image data. Experiments show that VMamba outperforms existing models in image classification, object detection, instance segmentation, and semantic segmentation, demonstrating superior accuracy, scalability, and adaptability. All reviewers are satisfied with the authors' responses.
Decision: Accept (spotlight)