Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model
Authors: Yuheng Shi
Minjing Dong
Chang Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Classification
Generic vision model
State Space Model
Primary area: machine_vision

Abstract: Despite the significant achievements of Vision Transformers (ViTs) in various vision tasks, they are constrained by the quadratic complexity. Recently, State Space Models (SSMs) have garnered widespread attention due to their global receptive field and linear complexity with respect to the input length, demonstrating substantial potential across fields including natural language processing and computer vision. To improve the performance of SSMs in vision tasks, a multi-scan strategy is widely adopted, which leads to significant redundancy of SSMs. For a better trade-off between efficiency and performance, we analyze the underlying reasons behind the success of the multi-scan strategy, where long-range dependency plays an important role. Based on the analysis, we introduce Multi-Scale Vision Mamba (MSVMamba) to preserve the superiority of SSMs in vision tasks with limited parameters. It employs a multi-scale 2D scanning technique on both original and downsampled feature maps, which not only benefits long-range dependency learning but also reduces computational costs. Additionally, we integrate a Convolutional Feed-Forward Network (ConvFFN) to address the lack of channel mixing. Our experiments demonstrate that MSVMamba is highly competitive, with the MSVMamba-Tiny model achieving 83.0% top-1 accuracy on ImageNet, 46.9% box mAP, and 42.5% instance mAP with the Mask R-CNN framework, 1x training schedule on COCO, and 47.9% mIoU with single-scale testing on ADE20K.  Code is available at https://github.com/YuHengsss/MSVMamba.

Multi-Scale VMmbaa Hierarchy in Hierarchy Visual State Space Model
Yuheng Shi City University of Hong Kong yuhengsh1990@mail.ccmm
Minjing Dong City University of Hong Kong minjdongocityu. edu. .k
Chang Xu University of Sydney cxxu@sydney.eduu
Abstract
Despite the significant achievements of Vision Transformers (ViTs) in various vi- sion tasks, they are constrained by the quadratic complexity. Recently, State Space Models (SSMs) have garnered widespread attention due to their global receptive field and linear complexity with respect to the input length, demonstrating sub- stantial potential across fields including natural language processing and computer vision. To improve the performance of SSMs in vision tasks, a multi-scan strategy is widely adopted, which leads to sigmificant redundancy of SSMs. For better trade-off between efficiency and performance, we analyze the underlying reasons behind the success of the multi-scan strategy, where long-range dependency plays important role. Based on the analysis, we introduce Multi-Scale Vision Mamba (MSV Mamba) preserve the superiority of SSMs in vision tasks with limited parameters. It employs multi-scale 2D scanning technique on both original and downsampled feature maps, which not only benefits long-range dependency learning but also reduces computational costs. Additionally, we integrate Convo- lutional Feed-Fore Forward Network (ConvFFN) to address the lack of channel mixing. Our experiments demonstrate that MSVMamba is highly competitive, with the MS MaMMa-Tinnng model achieving 83.0% top-1 accuracy on ImageNet, 46.9% box mAP, and 42.5% instance mAP with the Mask R-CNN framework, 1x training schedule on COCO, and 47.9% mloU with single-scale testing on ADE20K. Code is available htthts:://// / /gg.ttit....... com www..oo//eeeese//eeeeeeeeeee /MSVMambal
Introduction
In the domain of computer vision, the extraction of features plays a pivotal role in the performance of various tasks, ranging from image classification to more complex applications like detection and
33 have been the backbone of feature extraction methodologies, prized for their linear scaling complexity and proficiency in capturing local patterns. However, CNNs often fall short encapsulating global context, limitation that becomes increasingly apparent in tasks requiring comprehensive understanding
segmentation. Traditionally, Convolutional Neural Networks (CNNs) 22 40 18
of the entire visual field. In contrast, Vision Transformers (ViTs) 6. 32 50 43 have emerged
as a compelling alternative, boasting an inherent global receptive field that allows for the direct capture of long-range dependencies within an image. Despite their advantages, VITs are hampered by their quadratic scaling complexity concerning the input size, which significantly constrains their applicability to downstream tasks such as object detection and segmentation, where efficiency is paramount. Recently, State Space Model (SSM)-based approaches 41 8 have garnered attention for their ability to combine the best both worlds: a global receptive field and linear scaling complexity. Notably, Mamba [9] introduces hardware-aware and input-dependent algorithm that significantly enhances the performance and efficiency of SSMs. Inspired by Mamba's success, a burgeoning body of work has sought leverage its advantages for vision tasks, pioneering efforts such as ViM [60] and VMamba [30].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

85
85
Top-1 (11Arraayy))))
Top- Arraaay))))))
83
Canrhati
VMumba
VMaamba
82
MBV Mamba
M5VMamba
5
10
15
20
0.4
0.7
1.3
1.6
1.9
2.2
25
28
31
FLOPs(G)
Latency(ms)
Figure 1: FLOPs and latency comparison ImageNet. The latency tested RTX 4090 GPU with batch size of 128 using FP32 precision at an image resolution 224.
The S6 block, developed by Mamba [9]. was originally designed for NLP tasks. To adapt S6 for vision tasks, images are first divided into patches and then flattened into patch sequence along the scanming path. To accommodate the non-causal nature of image data, the multi-scan strategy is widely adopted for vision tasks, such ViM [60] which enhances the sequence by summing in both forward and reverse directions, and VMamba [3 which integrates horizontal and vertical scans. However, unlike NLP models, which have up billions of parameters, current vision backbones always take computational costs into consideration, i.., the trade-off between accuracy and efficiency. This constraint model size inherently limits the long-range modeling capabilities of SSMs vision tasks. Taking ViM-Tiny 60) as an example, placing the cls token in the middle of the sequence yields markedly better results than positioning it the ends. This suggests that central placement compensates for the model's limited ability to integrate distant information, highlighting vision models. We refer
the difficulties handling long-range dependencies parameter-constraiiee to this as the long-range forgetting problem. In this work, we analyze how the multi-scan strategy in (60 30 helps alleviate this problem. Compared to the single-scan strategy, the multi-scan one allows long-range decay to manifest in various directions within 2D images. However, the increased scanning routes bring multiples of computations, significantly increasing redundancy and limiting efficiency. Thus, we aim to pursue better trade-off between the performance and efficiency of Mamba in vision tasks.
The most direct and effective method to address the long-range forgetting problem is to shorten the sequence length, which can be achieved by downsampling the feature map. However, placing all scanning routes on a downsampled feature map could result in the loss of fine-grained features and the downstream task performance. Through the visualization of different scans, we show that the decay rates could vary for different scanning routes, which motivates us to develop hierarchical design of multi-scan scal. In this work, we propose a Multi-Scar 2D (MS2D) scanning strategy to alleviate the long-ran range forgetting problem with limited computational costs. Specifically, we divide the scanning directions of SS2D 30) into two groups: one retains the original resolution and processed by the S6 block, while the others are downsampled, processed by the S6 block, and then upsampled, which not only shortens the sequence length for long-range dependencies learning but also alleviates redundancy. Building on the VMmmba with its hierarchical architecture, we incorporate another hierarchical design within the block, creating hierarchy within a hierarchy. Furthermore, we introduce a Convolutional Feed-Forwan Network (ConvFFN within each block to bolster the model's s capability for channel-wise information exchange and local feature capture.
We conduct extensive experiments to validate the effectiveness of MSVmamba across a spectrum of tasks, including image classification, object detection, and semantic segmentation. Detailed comparisons on the ImageNet- K [3] dataset are illustrated in Fig. MSVMamba achieves notable improvement over VMmmba across different model sizes.
2
Related work
2.1 Generie Vision Backhone
The evolution of generic vision backbones has significantly shaped the landscape of computer vision,
transitioning from CNNs 25 54
3k] to ViT' 6 32
51
5
43 57
CNNs
have been the cornersoone of models, dominating vision tasks in the early era of deep

---

learning. The classic CNNs such as AlexNet 22
VGG
40 and ResNet [18), have paved the way
for numerous subsequent innovations [54] 21
19
42
20
These designs have significantly
improved performance on a wide range of vision tasks by enhancing the network's ability to capture
complex patterns and features from visual data. The Vision Transformer [6] drawing inspiration
from the success of transformers 47] in natural language processing, has emerged as formidable contender to conventional CNNs for vision-relates tasks. ViT reimagines image processing by segmenting an image into patches and employing self-attention mechanisms process these segments. This innovative approach enables the model to discern global dependencies across the entire image, a significant leap forward in understanding complex visual data. However, the ViT architecture demands considerable computational resources and extensive datasets for effective training. Moreover, its performance is intricately tied to the input sequence length, exhibiting quadratic complexity that can escalate processing costs. In response, subsequent research has focused on developing more efficient 51
training strategies [4
45 24 hierarchical network structures 32 31
50
15 refined spatial
attention mechanisms these issues.
57 59
16 5
and convolution-based design
2
29
36
to address
2.2 State Space Models
41 g have garnered increasing attention from researchers due to their computational complexity, which grows linearly with the length of the input sequence, and their inherent global awareness properties. To reduce the computational resource consumption of SSMs, S4 [ introduced a diagonal structure and combined it with a diagonal plus low-rank approach to construct structured SSMs. Subsequently, S5 [41] and H3 {8] further enhanced the efficiency of SSM-based models by introducing parallel scanning techniques and improving hardware utilization. Mamba p then introduced the S6 block, incorporating data-dependent parameters to amend the Linear Time Invariant(LTT) characteristics of previous SSM models, demonstrating superior performance over transformers on large-scale datasets. In the realm of vision tasks, S4ND (35] pioneered the application of SSM models in vision tasks by treating visual data as 1D, 2D, and 3D) signals. U-Mamba (34 combined CNNs with SSMs for medical image segmentation. ViM [60] and VMamba [30] integrated the S6 block into vision backbone design, employing multiple scanning directions to accommodate the non-casual nature of image data, achieving competitive results against ViTs and CNNs. Motivated by the success of ViM and VMamba, a plethora of hamba-based works 37 23
State Space Models (SSMs) TO 22
7
55
56 39] have emerged across various vision tasks, including vision 23 55 medical image segmentation [39) (27), and video classification (26).
26
backbone design 37
showcasing the potential of SSM-based approaches advancing the field of computer vision.
3 Method
In this section, we first summarize the state space model in Section 3.1 Subsequently, in Section 3.2 we provide an in-depth analysis of the multi-scan strategy in existing vision Mamba models. Following the analysis, Section 3.3 tackles the redundancy and long-range dependency issue by introducing Multi-Scale 2D (MS2D) scanning strategy. Finally, Section 3.4 details the integration of the Multi-Scale State Space (MS3) block, which incorporates the MS2D technique alongside a ConvFFN Building upon the MS3 block, various model configurations are developed across different scales, illustrating the adaptability and scalability of our proposed approach.
3.1 Preliminaries
State Space Models. Classical State Space Models (SSMs) represent a continuous system that maps an input sequence æ(() E RR_ to a latent space representation h(() E R and subsequently predicts an output sequence y()) R_ based on this representation Mathematically, an SSM can be described
as follows:
N'(() = Ah(t) + Br((), v(t) Ch(t),
where A E NRNNX BRNXN,BERRA and C E I are learnable parameters.
Discretization. To adapt continuous State Space Models (SSMs) for use within deep learning frameworks, it is crucial to implement discretization operations. By incorporating a timescale parameter A E I and employing the widely utilized zero-order hold (ZOH) the discretization rule,
3

---

96
(a) Horizontal Scan
(b) Vertical Scan
(c) Decay Ratio
(d) Binary Decay Ratio
Figure 2: Illustration of decay along horizontal, vertical scanning routes and their ratio.
the discretized versions of A and B (denoted as A and B, respectively) can be derived, with which, Eq. Can be reformulated into a discretized manner as:
(t) = Ah(t - + Br(t), v(t) = Ch(t). where = م B (AA) (A I]AB
AB,
where denotes the identity matrix. Afterward, the process of Eq. could be implemented in a global convolution manner as:
y v= K. K (CB, CAB ... CA
"AB...,CAA"A CA''BB) 'B)
where K E R is the convolution kernel.
Selective State Space Models. The Selective State Space (S6) mechanism, introduced Mamba 99]
renders the parameters B, C, and A input-dependent, thereby enhancing the performance of SSM- based models. After making B, C, and A input-dependent, the global convolution kernel in Eq.
could be rewritten as:
L-1
K C.BL,
C₁ II A,B, 1=1
(4)
3.2 Analysis of Multi-Scan Strategy
When processing image data using the S6 block, the 2D feature map E HxW'DDD is flattened into 1D sequence of image tokens, denoted as X E LRDD According to Eq. 4 and Eq. 2 the contribution of the men token to the construction of the na token (m < n) in S6 be expressed as:
C, A,B, (,A Ban where A (m-nn)
A.A
Typically, the learned A,A is negative, which biases the model towards prioritizing recent tokens' information. Consequently, as the sequence length increases, the exponential term eEi... A:A in Eq. decays significantly, resulting in minimal contributions from distant tokens. We refer to it as the long-range forgetting issue, which has also been observed in recent studies applying S6 to vision tasks [13]. Although this problem can be mitigated by increasing the number of parameters and depth of the model, such adjustments introduce additional computational costs. Furthermore, the causal property of the S6 block ensures that information can only propagate unidirectional manner between tokens, preventing earlier tokens from accessing information from subsequent tokens.
The inherent non-causal nature of images renders the direct application of the S6 block vision- related tasks less than optimal, as identified by ViM [60]. To mitigate this limitation, ViM [60] and VMamba 30 have introduced methodologies that entail scanning image features across various directions and then integrating these features. Generally, the updated token along one of the scanning routes, denoted as Scan (Z (p.)). where (p. q) indicates the coordinate, could be obtained by:
Scan(Z(p.)) C. Σ D A(i-a) (Z)
(6)
In Eq. 6 represents the transformation that converts a feature map into a ID sequence, and a denotes the corresponding index of Z(p, q) in the transformed ID sequence. Afterward, results from

---

K=3, S=1 DW Conv
S6
Interpolate
K=7, S=2 DW Conv
Figure 3: Illustration of the Multi-Scale 2D-Selective-Soee on an image
multi-scan routes are added together to produce enhanced feature Z' (p. q), which can be denoted as:
Z' (p.g)
(Za.)(Z(pps)) ECaz
Cas
Σ A ar (Z)
This multi-scan strategy allows tokens to access information from each other. In VIM 60), two distinct scanning routes correspond two different transformations in Eq. specifically, the flatten and the flatten with flip transformations. Similarly, VMMamb (30) extends the basic bidirectional scanning by incorporating both horizontal and vertical scanning directions, yielding four distinet scanning routes. Besides, the multi-scan strategy also alleviates the long-range forgetting problem by minimizing the effective distance between tokens. For tokens coordinates (P1, 91) and (p2. 92). the strategy employs multiple scanning routes, each potentially altering their relative positions. The minimum distance across these routes is given by min, dę ((p1. 91) + (p2, 92)). where de represents the distance between the tokens in the sequence generated by the k-th scan. By reducing this distance, the multi- scan strategy reduces the decay of influence between distant tokens, thereby enhancing the model's 's ability to maintain and utilize long-range information.
To more intuitively demonstrate the relationship between the multi-scan strategy and long-range decay, visualize the exponential term eE A;A along the horizontal and vertical scanning directions VMamba-Tiny with respect to the central token in Fig. 2a and Fig. 26 Specifically, randomly select 50 images from the ImageNet (3) validation and compute the average decay along scanning routes at the last layer of the final stage across these images and feature dimensions. We use higher input resolution to enhance the quality of the visualization.
According to these observations, the success of the multi-scan strategy in VMamba can be attributed to its mitigation of the non-causal properties of image data and alleviation of the long-range forgetting problem. However, as the number of scanning routes increases, the computational cost also rises linearly, introducing computational redundancy. In Fig. 2c we illustrate the maximum ratio of Fig. 2a to Fig. 2bb and vice versa. While in Fig. 2d we present a binarized version of Fig. [2c] applying threshold of 10, which covers more than 40% of the entire figure. This phenomenon indicates that the varying decay rates across different scanning routes lead to certain routes dominating the decay dynamics, which can also be attributed to the long-range forgetting problem. The existence of dominant scanning routes implies that some scans contribute significantly more to information retention than others, leading to computational redundancy in the multi-scan strategy.
3.3 Multi-Scale 2D Scanning
As discussed in the last subsection. the contribution of tokens decays with increasing scanning distance. The most effective and direct way to alleviate the long-range forgetting problem is to reduce the number of tokens. Simultaneously. since the computational complexity of the S6 block is linearly dependent the number of tokens, reducing the token count also enhances efficiency. Thus, an alternative approach to address the aforementioned issue is to apply the multi-scan strategy downsampled feature map. However, setting all scans downsampled feature map will ignore fine-gre grained features and result in unavoidable information loss. Thus, scanning along the full-resolution feature map is also essential.
Motivated by these considerations, we introduce a simple yet effective Multi-Scale 2D scan- ning(MS2D) strategy, as depicted in Fig. 3 Our approach commences with the generation of
5

---

Sulti-Scale Space State
ConVFFN
DWCo
1x1 Conv
DWConv
GELU
1x1 Conv
Linear
MMS2D
Linear
SE
Linear
Figure 5: Detailed architecture of Multi-Scale State Space (MS3) block, consisting of Multi-Scale Vision Space State (MSVSS) block and a Convolutional Feed-Forward Network (ConvFFN) block.
hierarchical feature maps at varying scales, achieved through the application Depthwise Convolu- tion (DWConv) with distinct stride values. These multi multi-scale feature maps are then processed through four distinct scanning routes within VMamba. Specifically, we uitlize DWConvs with strides of respectively. Afterwards, Z₁ and
and to obtain feature map Z₁ ##xxxD and Z2 E R Ze are processed by two S6 blocks as:
Y1 = (Z)), 22, Ys Y. S6( (22])2(ZZ2,, (Z2) 0 (Zz))),
(8)
(9)
where a is transformation that convert 2D feature maps into 1D sequences used in SS2D, and Y is the processed sequence. These processed sequences converted back into 2D feature maps, and the downsampled feature maps are interpolated for merging:
Z % (Y,)), E 2,, 4},
(10)
Z' = Z + Interpolate E(Z))).Je (2,3,,4
(11)
where 7 is the inverse transformation of and Z' is the feature map enhanced by MS2D. The downsampling operation reduces the sequence length by factor of s2 which also shortens the distance be- tween tokens in Eq. 5 factor of s² thereby alleviating the long-range forgetting problem. As the computa- tional complexity of single S6 block O(9DDN) 2,, where N denotes the SSM dimension, replacing SS2D with MS2D reduces the total sequence length across four scans from 4L to (L+3/s²)L thereby improves the effi- ciency. Practically, the downsampling rate is set to 2. is worth noting that sequences from Z2 processed by the same S6 block. This approach maintains the same accuracy as using multiple S6 blocks for different scanning routes while ef- fectively reducing the number of parameters.
Srant
Scand
Scan1
Scan2
Scand
Scand
Scon3
Scand
04
(a) SS2D
(b) MS2D
Figure Illustration of the decay with different scanning routes in SS2D and MS2D.
To better illustrate the alleviation of the long-range forgetting problem, we also provide empirical evidence, as shown in Fig. 4 We compare the decay along scanning routes in the SS2D of VMamba and our MS2D, focusing
the last token with the same configuration Fig. The decay maps in downsampled features are interpolated back. As observed, the decay rate along scanning routes in downsampled maps is significantly alleviated, enhancing the capability capture global information.
3.4 Overall Model Architecture
In this study, we extend the capabilities of the VMamba framework by substituting its VSS block with our Multi-Scale State Space (MS3) block. The architectural framework of the MS3 block is delineated in Fig. 55 comprising Multi-Scale Vision Space State(MSVSS, component and a Convolutional Feed-Forwant Network (ConvFFN). The MSVSS component is devised by adapting the vision

---

state space framework in VMamba, substituting the SS2D with an MS2D to further introduce hierarchy design in a single layer. Additionally, Squeeze-Excitation (SE) 20 block integrated
subsequent to the multi-scale 2D scanning, as informed by related literature 23
37
Diverging
from the conventional focus on token mixing in prior vision Mamba architectures 60
30
23 our
design introduces channel mixer to augment the flow of information across different channels, concordance with preceding studies [52, 48 [1, (51, the ConvFFN which consists of a depth-wise convolution and two fully connected layers is employed as the channel mixer. Upon the amalgamation MSVSS and ConvFFN within the MS3 block, meticulous adjustments
aligning with the structural paradigms of typical vision transformers.
made to the number of blocks ensure comparable computational budget, facilitating a fair comparison.
To empirically validate the efficacy of our proposed modifications, we introduce model variants across different scales. These variants, namely Nano, Micro, Tiny, Small and Base, are characterized by their parameter counts of 7M, 12M, 32M, 50M and 88M respectively. In terms of computational expenditure, these models require 0.9, 1.5, 5.1, 8.8 and 15.5 GFLOPs correspondingly, demonstrating scalable approach to model design that accommodates varying computational constraints. For models above tiny size, the multiplicative branch the MSVSS block is removed as informed by the VMambav3 30 for better performance. Detailed architectures are shown in Appendix A
4 Experimental Validation
4.1 ImageNet Classification
Settings. Our models are trained and tested on the ImageNet-IK dataset 3 In alignment with previous works 32
Table 1: Accuracy comparison across various mod- els on ImageNet-IK.
30 23 all models un-
dergo training for 300 epochs, with the initial 20 epochs dedicated to warming up. The train- ing utilizes a batch size of 1024 across GPUs. We employ the AdamW W optimizer, setting the betas to (0.9, 0.999) and momentum 0.9. The learning rate is managed through cosine decay scheduler, starting from an initial rate of 0.001, coupled with weight decay of 0.05. Addition- ally, we leverage the exponential moving aver- age (EMA) and implement label smoothing with a factor of 0.1 to enhance model performance and generalization. During testing, images center cropped with the size of 224x224. When dealing with the routes for multi-scale scanning, we select top-left to the bottom-right for dealing with the full-resolution feature map while the other three scans are responsible for scanning the downsampled feature map.
Top-1 Acc(%).
Method
#param.
FLOPs
RegNetY-800M
6M
0.8G
76.3
RegNetY-1166
38
11M
1.6G
78.0
RegNetY-44
38
21M
4.0G
80.0
DeiT-S
43
22M
4.6G
79.8
DeiT-B
4
86M
17.5G
81.8
Swin-T
32
29M
4.5G
81.3
Swin-S
32
50M
8.7G
83.0
Swin-B
88M
15.4G
83.5
ViM-T
60
7M
1.5G
ViM-S
60
26M
5.1G
80.5
VMambav3-T
30M
4.9G
82.6
VMambav3-S
30
50M
8.7G
83.6
VMambav3-B
30
89M
15.4G
83.9
LocalVMamba-TT
23
26M
5.7G
82.7
Results. Tab. D showcases our MSV- Mamba models against established CNNs, ViTs, and SSM-based models on ImageNet- 1K. MSVMamba-T, with 32M parameters and 5.1G FL.OPs, achieves 83.0% top-1 accu- racy, outperforming similar-cost SSM-based Lamaa-Ma The MSVMamba-B model
LocalVMamba-S
23
50M
11.4G
83.7
MSVMamba-N
7M
0.9G
77.3
MSVMamaa-M
12M
1.5G
79.8
MSVMamba-T
32M
5.1G
83.0
MS MSVMamba-S
50M
8.8G
84.1
MSVMamba-B
91M
16.3G
84.4
attains 84.4% accuracy with 91M parameters and 16.3G FLOPs, exceeding VMambav3-B by 0.5%.
These results highlight MSVMamba's 's efficiency and scalability, offering robust option for high- accuracy, resource-efficient model design.
4.2 Object Detection
Setup. We evaluate our MSVMa on the MSCOCO 28 dataset using the Mask R-CNN
framework for object detection and instance segmentation tasks. Following previous works 32 30
7

---

Table Object detection and instance segmentation with Mask R-CNN on COCO. The FLOPs are computed for an input size of 1280 x 800. Multi-scale training is exclusively implemented in the 3x schedule, All backbones are pre-trained on the ImageNet-11 K dataset.
Mask R-CNN 1x Schedule
Mask R-CNN 3x Schedule
Backbone
fiparam. FLOPs
AP" AP
A5
AP" AP% AP
AP"
AP
APis
AP"
AP AP7s
PVT-T |00
36.7 59.2 39.3 35.1
56.7 37.3
39.8
62.2
43.0
37.4
59.3
39.9
33M
208G
LightViT-- 22
37.8 60.7
40.4
35.9
57.8
38.0
41.5
64.4
45.1
38.4
61.2
40.8
28M
187G
EffV Mamba-S S 37
39.3
61.8
42.8
36.7
58.9
39.2
41.6
63,9
45.6
38.2
60.8
40.7
31M
197G
MSVMamba-NN M
43.8
65.8
47.7
39.9
62.9
42.9
46.3
68.1
50.8
41.8
65.1
44.9
32M
201G
Swin-T
82
42.7 65.2
46.8
39.3
62.2
42.2
46.0
68.1
50.3
41.6
65.1
44.9
48M
267G
ConyNeXt-T 33
44.2 66.6
48.3
40.1
63.3
42.8
46.2
67.9
50.8
41.7
65.0
44.9
48M
262G
VMambav3-T 33
47.3
69.3
52.0
42.7
66.4
45.9
48.8
70.4
53.5
43.7
67.4
47.0
50M
271G
LocalVMamba-T [23
46.7
68.7
50.8
42.2
65.7
45.5
48.7
70.1
53.0
43.4
67.0
46.4
45M
291C
MSVMamba-T
46.9
68.7
51.4
42.5
66.2
45.8
48.7
69.8
53.3
43.4
67.2
46.8
52M
275G
Swin-S (32]
44.8 66.6
48.9
40.9
63.2
44.2
48.2
69.8
52.8
43.2
67.0
46.1
69M
354G
ConvNeXI-S 33
45.4
67.9
50.0
41.8
65.2
45.1
47.9
70.0
52.7
42.9
66.9
46.2
70M
348G
VMambav3-S 30
48.7
70.0
53.4
43.7
67.3
47.0
49.9
70.9
54.7
44.2
68.2
47.7
70M
349G
MSVMamba-S
48.1 70.1
52.8
43.2
67.3
46.5
49.7
70.9
54.3
44.2
68.0
47.9
70M
349G
we utilize backbones pretrained on ImageNet-IK for initialization. We employ standard training strategies of 1x (12 epochs) and 3x (36 epochs) with Multi-Scale (MS) training for a fair comparison.
Results. Tab. presents performance comparison of our method against CNNs, ViTs, and SSM- based models. Our model achieve competitive results across various variants and training settings. Specifically, MSVMamba-T outperforms Swin-T by +4.2 box AP and +3.3 mask AP under the 1x schedule and also shows improvements in both box AP and mask AP under the 3 schedule.
4.3 Semantic Segmentation
Table 3: We present the results of semantic seg- mentation on the ADE20K dataset [58] using the UperNet framework [33] The computational com- plexity, measured in FLOPs, is calculated for input dimensions of 512 2048. The abbreviations "SS" and "MS" refer to single-scale and multi-scale test- ing, respectively.
Setup. Consistent with the methodologies used in Swin [32) and VMamba [30]. we utilize the UperHead (53] framework atop ImageNet pre-trained MSVMamba backbone. The train- ing process conducted over 160K iterations with batch size of 16. We employ the AdamW 10 10-5 Our experiments are primarily conducted using a default input resolution of 512 x 512. Addi- tionally, we also incorporate Multi-Scale (MS) testing to assess performance variations.
optimizer with learning rate set
mloU SS
mloU MS
Method
#param. FLOPs
ResNet-50 [8]
42.1
42.8
67M
953G
DeiT-S+MLN
44
43.8
45.1
58M
1217G
Swin-T T 332
44.4
45.8
60M
945G
Results. We present the detailed results of our model and other competitors Tab. which in- cludes both single-scale and multi-scale testing. Our MSVMambe consistently outperforms the Swin and ConNeXt models in the tiny variant by margins of +2.2 and +1.6 mloU, respectively.
ConvNeXt-
33
46.0
46.7
60M
939G
VMamhav3-T
30
47.9
48.8
62M
949G
MSVMamba-M
45.1
45.4
42M
875G
MSVMamba-T
47.9
48.5
63M
953G
4.4 Ablation Study
To validate the effectiveness of the proposed modules, we conducted a comprehensive ablation study, Specifically. we scaled the VMamba- model by setting its embedding dimension d to 48, the state space dimension N to 8, and the number of blocks in the four different stages to [1, 2, 4, 2). The scaled model, referred to as VMamba-Nano, has parameters and computational costs of 4.4M and 0.87GFLOP respectively. This model and the standard tiny-sized models serve as the baseline for our ablation experiments. Models in ablation study are conducted over a training schedule of 100 epochs on ImageNet- to reduce training time. Besides, the AP, and APm. pm under Mask R-CNN with Ix schedule COCO dataset are also reported for nano-size models.
8

---

Table 4: Evolutionary trajectory from VMamba to MSVMamba on nano-sized model.
Top-1 Acc(%)
Model
MS2D SE ConvFFN N =
#param. FLOPs
AP,
APm
VMamba-Nano
4.4M
0.87G
69.6
38.1
35.6
4.8M
0.89G
71.9/2.3
39.111.0
36.3+o.7
5.3M
0.89G
72.4
39.5
36.5
MSVMamba-Nano
6.6M
0.94G
74.4
41.0p2.9
37.8,2.2
6.9M
0.88G
75.1
41.44a.3
37.9/2.3
Table 5: Performance ablation on tiny-size models. FPS and Memory are tested on a 4090 GPU with batch size of 128 and FP32 precision. The symbol indicates model inherit optimization used in VMambav3 version.
Top-1 Acc(%)
Model
MS2D SE ConvFEN N N= #param. FLOPs
FPS
Memory (MB)
VMambavl-Ting
23M
5.6G
80.3
603
6639
24M
4.8G
80.9tu.6
866
4780
MSVMamba-Ting
33M
4.6G
81.441.2
1092
4533
MSVMamba- Tiny
/
32M
5.1G
81.7
1097
2413
On Multi-Scale 2D Scan. For the nano-size variant, replacing SS2D with our MS2D in the VMamba framework resulted in an increase in accuracy ImageNet-11 from 69.6% to 71.9%. Additionally, the AP, and AP,, metrics improved from 38.1 and 35.6 to 39.1 and 36.3, respectively, as shown in Tab. 4 Furthermore, we conducted an ablation the number of scans in the multi-scale scan, considering both full-resolution and half-resolution branches on nano-size models. The results are shown in Tab. 6 Placing all scans in the half-resolution branch led significant loss of fine- grained features, resulting in substantial decrease in model accuracy. Positioning two three scans in the full resolution branch, compared just one, resulted in accuracy improvements of 0.1% and 0.6%, respectively, but introduced an additional computational cost of approximately 12% and 25%. Allocating four scans the full resolution branch, effectively reverting to the SS2D method, increased the computational cost by 34% while only improving accuracy by 0.4%. For an optimal trade-off between computational cost and accuracy, we select one scan in the full-resolution branch as the default setting. Building upon the MS2D foun- dation, we introduce SE block following EfficientV- Mamba (37), which further enhanced accuracy by 0.5% with minimal additional computational cost.
Table 6: Ablation study on MS2D.
Top-1 Acc(%)
Full Half #param
FLOPs
0
5.1M
0.74G
63.1
Experiments related tiny-size model are reported in Tab. 5 Our findings indicate that the proposed MS2D
1
3
4.8M
0.89G
71.9
2
5.0M
1.00G
72.0
module contributes to an improvement of 0.6% in Top-1
3
1
5.3M
1.11G
72.5
accuracy for the tiny-size model while other components of our model collectively contribute an additional 0.5% in- crease in accuracy. The MS2D module not only enhances performance but also contributes to further speed gains and reductions in memory usage. Since MS2D is orthogonal to the updates in VMambav3, they can be combined for further enhancements. In the last row of Tab. 5 we present the results after adopting optimization used in VMambav3, which include reducing the 5sm ratio and climinating the entire multiplicative branch.
4
5.1M
1.19G
72.3
On COnvFFN. Upon replacing SS2D with MS2D and incorporating a SE block, constructed model that utilizes ConvFFN as a channel mixer. When only using SSM, the model exhibited insufficient information exchange between channels. The integration of ConvFFN as a channel mixer significantly enhanced the model's 's capability for inter-channel information interaction. As indicated Tab. the addition of ConvFFN resulted an additional accuracy improvement of 2.0%. Besides, we set the state space dimension N = 1 and stacked one more block to further enhance the capability of capturing long-range information while maintaining roughly constant computational cost. This operation resulted in an additional accuracy improvement of 0.7%, as shown in Tab, To maintain

---

roughly equivalent computational cost, we adjusted the number of blocks within the model. When integrating the ConvFFN and setting the state space dimension N 1, we meticulously calibrate the quantity of blocks to maintain nearly constant computational cost, measured in GFL.Oss.
5
Limitations
The design multi-scale VMamba aims to tackle the long-range forgetting problem of Mamba models with limited parameters vision tasks. Although the proposed model has proven to be effective, its scalability remains to be explored since this issuc can also be alleviated by increasing model sizes.
6 Conclusion
In this paper, we introduced Multi-Scale VMamba (MSVMamba), an SSM-based vision backbone that leverages the advantages of linear complexity and global receptive field. We developed the Multi-Scale 2D (MS2D) scanning technique to minimize computational redundancy and alleviate the long-range forgetting problem parameter-limited vision models. Additionally, we incorporated the Convolutional Feed-Forward Network (ConvFFN) to enhance the exchange of information between channels, thereby significantly improving the performance of our model. Our experiments demonstrate that MSVMamba consistently outperforms popular models from various architectures, including ConvNeXt, Swin Transformer, and VMamba, image classification and downstream tasks.
Acknowledgements
This work supported in part by the Australian Research Council under Projects DP240101848 and FT230100549, and by the Start-up Grant (No. 9610680) the City University of Hong Kong.
10

---

References
[1] Tianxiang Chen, Zhentao Tan, Tao Gong, Qi Chu, Yue Wu, Bin Liu, Jieping Yc, and Nenghai Yu. Mim-istd: Mamba-in-mamba for efficient infrared small target detection. arXiv preprint arXiv:2403.02 148, 2024.
[2] Zihang Dai, Hanxiao Liu, Quoe Le, and Mingxing Tan, Coatnet: Marrying convolution and attention for all data sizes. NeurlPS, 2021.
[3] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[4] Xiaohan Ding, Xiangyu Zhang, Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In CVPR, 2022.
[5] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
[7] Chengbin Du, Yanxi Li, and Chang Xu. Understanding robustness of visual state space models for image classification. arXiv preprint arXiv:2403.1092: 2024.
[8] Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.1440 10052, 2022.
[9] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00733 2023.
[10] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. NeurIPS, 2020.
[11] Albert Gu. Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.003300 2021.
[12] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. NeurlPS, 2021.
[13] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: A simple baseline for image restoration with state-space model. arXiv preprint 2rXiv:2402.1564B 2024.
[14] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In CVPR, pages 12175-12185, 2022.
[15] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In NewrIPS, 2021.
[16] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. CVPR, 2023.
[17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[19] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1484000004. 2017.
11

---

[20] Jie Hu, Li Shen, and Gang Sun. Squeeze-and -nd---andddee-diieee networks. In CVPR. 2018.
[21] Gao Huang, Zhuang Liu, Geoff Pleiss, Laurens Van Der Maaten, and Kilian Weinberger. Convolutional networks with dense connectivity. IEEE TPAMI, 2019.
[22] Tao Huang, Lang Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Lightvit: Towards light-weight convolution-free vision transformers. arXiv preprint arXiv:2207.05557. 2022.
[23] Tao Huang. Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. arXiv preprint arXiv:2403.09338, 2024.
[24] Zihang Jiang. Qibin Hou, Li Yuan, Daquan Zhou, Xiaojic Jin, Anran Wang, and Jiashi Feng. Token labeling: Training a 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet. arXiv preprint arXiv:2104. 10858, 2021.
[25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS, 2012.
[26] Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Video- mamba: State space model for efficient video understanding, arXiv preprint arXiv:2403.06977, 2024.
[27] Weibin Liao, Yinghao Zhu, Xinyuan Wang, Cehngwei Pan, Yasha Wang, and Liantao Ma. Lightm-unet: Mamba assists in lightweight unet for medical image segmentation. arXiv preprint arXiv:2403.05246, 2024.
[28] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
[29] Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lianwen Jin. Scale-aware modulation meet transformer. In ICCV, 2023.
[30] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint 20X1v:2401.10166, 2024.
[31] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning. Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022.
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[33] Zhuang Liu, Hanzi Mao, Chao- Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022.
[34] Jun Ma, Feifei Li, and Bo Wang- U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint 2rXiv:2401.04722, 2024.
[35] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao. Stephen Baccus, and Christopher Ré. $4nd: Modeling images and videos as multidimensional signals with state spaces. NeurlPS, 2022.
[36] Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, and Gao Huang. On the integration of self-attention and convolution. In CVPR, 2022.
[37] Xiaohuan Pei, Tao Huang, and Chang Xu. Efficientvmamba: Atrous selective scan for light weight visual mamba. arXiv preprint arXiv:2403. 2024.
[38] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In CVPR, 2020.
[39] Jiacheng Ruan and Suncheng Xiang. Vm-unet: Vision mamba unet for medical image segmen- tation. arXiv preprint 4924422402.0249), 2024.
12

---

[40] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale acale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[41] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208. 04433,, 2022.
[42] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019.
[43] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[44] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In ECCV, 2022.
[45] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021.
[46]
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NearlPS, 2017.
[48] Wang, Chen, Lin, Pu, and G Ding. Repvit: Revisiting mobile enn from vit perspective. arxiv 2023, arXiv preprint vrXiv:2307.09283 2023.
[49] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scal vision foundation models with deformable convolutions. arXiv preprint arXiv:2211.05778, 2022.
[50] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[51] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 2022.
[52] Zhuofan Xia, Xuran Pan, Shiji Song. Li Erran Li, and Gao Huang. Dat++ Spatially dynamic vision transformer with deformable attention. arXiv preprint arXiv:2309.01 2023.
[53] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
[54] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
[55] Chenhongyi Yang. Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot J Crowley. Plainmamba: Improving non-hie oon-hierarchica mamba in visual recognition. arXiv preprint 24t.:2200.17699 2024.
[56] Yuhuan Yang, Chaofan Ma, Jiangchao Yao, Zhun Zhong, Ya Zhang, and Yanfeng Wang. Re- mamber: Referring image segmentation with mamba twister. arXiv preprint arXiv:2403. /7839, 2024.
[57] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022.
[58] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[59] Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson Lau. Biformer: Vision transformer with bi-level routing attention. In CVPR, 2023.
[60] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint 2024.
13

---

A Network Architecture
Table 7: Specifications of MSVMamba varints.
Model
Blocks
Channels
8sm ratio
FFN ratio
#param.(M) GFLOPs
Nano
[1,2, 5, 2]
48, 96, 192, 384]
2
2
7
0.9
Micro
[1, 5,
64, 128, 256,
512]
2
2
12
1.5
Tiny
[2, 2, 9,2]
96, 192, 384,
768]
4
32
5.1
Small
[2, 3, 20, 2]
96, 192, 384, 768]
4
50
8.8
Base
[2, 4, 18, 2]
[128, 256, 512, 1024]
4
91
16.3
Tab. 7 we present the detailed architecture of our model variants, including the Nano, Micro, Tiny, Small and Base versions, each with varying channels, block numbers 88m ralio and FFN ratio.
B More Ablations
To further demonstrate the effectiveness of our MS2D, we add more baselines that involve only Scanl (Uni-directional Scan) and combination of Scanl I and Scan3 (Bi-directional Scan). These results are presented in Tab. 8 Concretely, our MS2D further outperform Uni-directional Scan and Bi-directional Scan baselines by 3.0% and 2.4% top-1 acc respectively.
Tab. D we explore the impact of full-resolution scanning directions. The results indicate that while different scans yield similar accuracy, Scanl was selected for its marginally superior performance consistency.
Table 8: Ablation with more baselines. The CrossScan is utilized in VMamba, while the Uni-Scan and Bi-Scan denote Uni-directional Scan and Bi-directional Scan respectively.
Setting
#param
GFLOPs
Accuracy (%)
Uni-Scan
4.4
0.87
68.9
Bi-Scan
4.4
0,87
69.5
CrossScan
4.4
0.87
69.6
MS2D
4.8
0.89
71.9
Table 9: Ablation for full-resolution branch.
Full
Scanl
Scan2
Scan3
Scan4
Top-1 Acc(%)
71.9
71.8
71.8
71.9
C Efficiency Comparison
We report detailed throughput comparison of Swin 12,, ConyNeXt [33. and VMamba 30 in
Tab, TÔJ All models are tested on RTX 4090 GPU with a batch size of 128 and FP32 precision at an image resolution of 224.
D Qualitative Analysis
We provide visualizations in Fig. 6 comparing the proposed MS2D and SS2D configurations in VMamba. These visualizations are generated by converting the S6 layer into an attention format, as demonstrated by VMamba [30]. The results clearly show that the full-resolution in MS2D scan captures more detailed features, whereas the seans at half resolution primarily focus on broader architectural details, compared to SS2D. The proposed hierarchical scanning pattern facilitates the current layer's ability to discern and amalgamate features across various levels of abstraction.
14

---

Table 10: Efficiency comparison with bascline VMamba 30
and widely-used Swin Trans-
former 32 and ConvNeXt (3].
Model
Top-1 Acc(%)
#Params
FLOPs
Thru. (imgs/sec)
Memory (MB)
(G)
Swin-T
32
81.3
28 M
4.5
986
2402
ConvNeXt-1
33
82.1
29 M
4.5
1062
1670
VMambav 1-T 30
82.2
23 M
5.6
603
6639
VMambav3-T
30
82.6
30 M
4.9
1456
3204
MSVMamba-T
83.0
32 M
5.1
1097
2413
Swin-S 32
83.0
50 M
8.7
561
2596
ConvNeXt-S
33
83.1
50 M
8.7
605
1753
VMambav1
30
83.5
44 M
11.2
425
6882
VMambav3-S
300
83.6
50 M
8.7
764
5780
MSVMamba-S
84.1
50 M
8.8
708
2545
Swin-B [32
83.5
88 M
15.5
363
3362
ConvNeXt-B
33
83.8
89 M
15.4
387
2380
VMambav I-B
30
83.7
76 M
18.0
314
8853
VMambav3-B
30
83.9
89 M
15.4
555
7826
MSVMamba-B
84.4
91 M
16.3
514
3699
SS2
Soan
Scan2
Scan
Scan 4
MS2D
Figure 6: Attention maps from four distinct scanning directions, generated by SS2D and our MS2D in the last layer of the second stage. In the second row, full-resolution (first scan) captures fine-grained features, whereas scans at half resolution capture coarse-grained features. Maps are rendered higher resolution enhance visualization quality.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions of the paper, including the theoretical and experimental results. The claims made are consistent with the findings presented the paper and accurately reflect the scope of the research.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in Section of the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: The paper provides the full set of assumptions and a complete (and correct)
proof for theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper discloses all the information needed to reproduce the main experi- mental results. Besides, code, models and logs will be released after review.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code, models and logs provided in the github repo Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ccippi.cc guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
All the training and test details are properly stated.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not report eтror bars or statistical significance information because the results of multiple experiments are consistent and show minimal variation. Therefore, the precision of the results is deemed sufficient without the need for additional statistical measures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources needed to reproduce the experiments are stated in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp//./aarrpp...c/o/o////ccce
Answer: [Yes]
Justification:
The research presented in the paper adheres to the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper focuses on the design and development of a classification model, which foundational research in the field of machine learning. As such, does not directly address specific applications or deployments that could lead to societal impacts. The work is primarily technical and does not include discussions on potential positive or negative socictal impacts.
19

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in the paper, including code, data, and models, properly credited to their original creators. The licenses and terms use for these assets are explicitly mentioned in the paper, and their usage complies with the specified terms. Proper citations and acknowledgments are provided to ensure that the contributions the original creators are recognized and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
20

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "Lj6U11wgm1", "reviewer": "Reviewer_kG7u", "summary": "The paper introduces Multi-Scale VMamba (MSVMamba), a novel vision backbone model that leverages State Space Models (SSMs) to address the challenges of quadratic complexity in Vision Transformers (ViTs). The proposed Multi-Scale 2D Scanning (MS2D) and Convolutional Feed-Forward Network (ConvFFN) contributes to the final performance.", "questions": "Please refer to the weakness", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper presents a multi-scale design of VMamba, reducing the sequence length by downsampling the input features.\n- The SSM module proposed in this paper incorporated the strength of residual, dwconv, ssm, se-block.\n- This paper achieves good results on classification and dense prediction benchmarks.", "weaknesses": "- This paper lacks the efficiency comparison among training/inference FPS and memory usage on real GPU devices.\n- The proposed multi-resolution processing seems to contain too many inductive biases. I wonder if the gains would be reduced when applying to a standard-ablation-size model, i.e., VMamba-T.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and for the time you have dedicated to reviewing our manuscript. We appreciate your feedback, which helps us improve the quality of our paper. Below, we address your concerns regarding the **efficiency comparison(Q1)** and the **ablation of our model to a tiny-size variant(Q2)**.\n\n### **Q1: Efficiency Comparison Among Training/Inference FPS and Memory Usage.**\n\nThanks for the suggestion. As suggested, we have complemented the efficiency comparison, including training/inference FPS and memory usage, with our baseline VMamba and widely-used SwinTransformer and ConvNeXt in **Table1** and **Table 2** of the uploaded PDF for your reference.  Compared to our baseline, our proposed model achieves nearly **1.5x speedup in inference** and **2.0x speedup in training**. Additionally, it requires approximately **30% less memory.**  The efficiency of our models at a 224x224 image resolution did not match that of well-established architectures such as Swin Transformer. However, when the image resolution is increased, our model achieves comparable efficiency to the Swin Transformer, which can be found in Table 2 of the uploaded PDF.\n\n\n### **Q2: Ablation for tiny-size model.**\n\nThanks for this nice concern. In response to this query about the performance of our model on a standard-ablation-size, we have conducted additional ablation studies with a 100-epoch training schedule. The results are detailed in the **table below** for your reference:\n\n\n| Model    | Param(M) | GFLOPs | Top-1 Acc(%) | Thru. (imgs/sec) | Train Thru. (imgs/sec) | Memory (MB) |\n|----------|----------|--------|--------------|------------------|------------------------|-------------|\n| VMamba   | 23       | 5.6    | 80.3         | 603              | 151                    | 6639        |\n| +MS2D    | 24       | 4.8    | 80.9         | 866              | 205                    | 4780        |\n| +Others  | 33       | 4.6    | 81.4         | 1092             | 331                    | 4532        |\n\n\nOur findings indicate that the proposed **Multi-Scale 2D (MS2D)** module contributes to an improvement of **0.6%** in Top-1 accuracy for the tiny-size model. **Other components** (SE, ConvFFN and N=1 in Table 4 of our paper) of our model collectively contribute an additional **0.5%** increase in accuracy. This ablation reveals that MS2D is more important in accuracy gain compared to other components on tiny-size model. Furthermore, the MS2D module not only enhances performance but also contributes to further speed gains and reductions in memory usage.\n\nWe hope that these revisions could satisfactorily address your comments and **thank you once again for your constructive feedback**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nw5xoQvA86", "reviewer": "Reviewer_TQwz", "summary": "The paper introduces a novel vision backbone model, MSVMamba, which incorporates State Space Models (SSMs) to address limitations in computational efficiency and long-range dependency capture in vision tasks. The model utilizes a multi-scale 2D scanning technique and a Convolutional Feed-Forward Network (ConvFFN) to improve performance with limited parameters.", "questions": "Please refer to the weakness.", "limitations": "Please refer to the weakness.", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper is easy to follow.", "weaknesses": "1. Lack of novelty. \n\nThe paper propose MSVMamba, the main contribution is shown in table.4. MS2D, SE, ConvFFN have already been proposed in previous papers. The main improvements come from existing knowledge. \n\n2. Speed\n\nThe paper does not systematically measure the speed of their model on all the tasks and scales. The training and inference of MSVMamba could be slow compared with current models, like CAFormer, Conv2Former, CSwinTransformer.\n\n3. Scalability\n\nThe model in the experiments is relatively small. Model with more than 300M or 600M parameters could show whether MSVMamba does perform better than current sota model. Now the experiments just show that the model converges quickly w.r.t. model size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and the time you have dedicated to reviewing our manuscript. We appreciate your feedback, which helps us improve the quality of our paper.  We address your concerns in a few points as described below:\n\n### **Q1: Lack of novelty.**\n\nWe agree that multi-scale strategy, SE, and ConvFFN have been introduced in previous CNN or ViT-based works. However, the utilization of these techniques, especially the multi-scale strategy, is well-motivated by the analysis and empirical observation in this paper, which could provide insights to Mamba model design for vision tasks. Specifically, we take our core contribution MS2D as an example. To begin with, **MS2D is first introduced in this paper**, which is carefully designed for Mamba in vision tasks. Unlike traditional multi-scale strategies that primarily enhance hierarchical feature learning, our **MS2D module is motivated by the need to mitigate the long-range issue** prevalent in Mamba models. This is a critical challenge that has not been adequately addressed by existing methods. As detailed in lines 151 to 161 of our manuscript, our analysis demonstrates that the contribution of tokens significantly decays with increased scanning distance. To address this, we innovatively downsample the feature map to effectively shorten the sequence length. This adaptation not only introduces a multi-scale design within a single layer but also specifically targets the reduction of computational complexity and improves the efficiency of long-range interactions. Figure 4 and Table 4 in our paper provide compelling evidence of the effectiveness of our approach. The multi-scale design significantly alleviates the long-range problem, as visually demonstrated in Figure 4. Furthermore, the quantitative ablations presented in Table 4 underscore the substantial improvements our model achieves over existing techniques. Thus, **the introduced multi-scale method, although similar to previous works, originates from a different motivation and addresses a distinct problem**. We hope this explanation helps to clarify the innovative aspects of our work and the specific challenges it addresses. \n\n### **Q2: Efficiency Comparison.**\n\nThanks for the suggestion. We have complemented the efficiency comparison as suggested, including training/inference FPS and memory usage, with our baseline VMamba and widely-used SwinTransformer and ConvNeXt in **Table1 and 2** of the uploaded PDF  for your reference. \nWe acknowledge that at the time of submission, the efficiency of our models at a 224x224 image resolution did not match that of well-established architectures such as Swin Transformer and ConvNeXt. However, it is important to highlight that the Mamba-based models still serve as a crucial backbone that achieves a competitive trade-off across various settings and is currently under continuous optimization. For instance, when the image resolution is increased, our model achieves comparable efficiency to the Swin Transformer, which can be found in Table 2 of the uploaded PDF. This is further supported by related experiments in ViM[1], which demonstrate the efficiency of the Mamba block in downstream tasks like detection. \nOur model design is specifically tailored for the Mamba architecture, allowing any subsequent optimizations related to efficiency to be directly inherited. In this work, our focus was primarily on comparing Mamba-based baselines. Our proposed model achieves significant improvements, with nearly **1.5x speedup in inference** and **2.0x speedup in training** compared to VMamba. \nWe understand the importance of systematic speed measurements across all tasks and scales. While our current manuscript may not cover all possible configurations, we are committed to extending our evaluations and will consider including more comprehensive speed comparisons in future revisions or subsequent works.\n\n\n\n\n### **Q3: Scalability**\n\nThanks for the suggestion! We have updated the results of the proposed model when scaling up to small and base size on ImageNet-1K. Please check further details in Table 1 of the uploaded PDF. MSVMamba-S and MSVMamba-B consistently outperform the VMamba baseline by 0.6% and 0.6% top-1 acc respectively.\nWhen it comes to the model size that exceeds 300M, we acknowledge the potential benefits of testing larger models with parameters exceeding 300M to compare against current state-of-the-art (SOTA) models. However, it is important to note that most existing works involving vision mamba models [1,2,3,4,5] operate under 100M parameters. In this work, we strictly follow the setting of [1,2,3] to conduct evaluation and comparison. For example, the largest model in VMamba exhibits 76M parameters. Thanks for your valuable suggestion. We will include the evaluation of Mamba models with much larger sizes in future works.\n\nWe hope these revisions could satisfactorily address your concerns and **thank you once again for your feedback**!\n\n**References:**\n\n[1] Zhu, Lianghui, et al. \"Vision mamba: Efficient visual representation learning with bidirectional state space model.\" arXiv preprint arXiv:2401.09417 (2024).\n\n[2] Liu, Yue et al. “VMamba: Visual State Space Model.”  arXiv preprint arXiv:2401.10166v1 (2024).\n\n[3] Huang, Tao, et al. \"Localmamba: Visual state space model with windowed selective scan.\" arXiv preprint arXiv:2403.09338 (2024).\n\n[4] Pei, Xiaohuan, Tao Huang, and Chang Xu. \"Efficientvmamba: Atrous selective scan for light weight visual mamba.\" arXiv preprint arXiv:2403.09977 (2024).\n\n[5] Yang, Chenhongyi, et al. \"Plainmamba: Improving non-hierarchical mamba in visual recognition.\" arXiv preprint arXiv:2403.17695 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffLaHrjxJG", "reviewer": "Reviewer_XVcS", "summary": "This paper presents a multi-scale vision mamba aimed at improving the performance of state space models (SSMs) in vision tasks while maintaining efficiency. The motivation stems from analyzing the multi-scan strategy in vision mamba, where the authors link its success to alleviating the long-range forgetting issue of SSMs in vision tasks. To address this problem effectively, the authors propose a multi-scale 2D scanning technique on both original and downsampled feature maps, which reduces the number of tokens in the multi-scan strategy. This method enhances long-range dependency learning and cuts down computational costs. Additionally, a ConvFFN is incorporated to overcome channel mixing limitations. Experimental results across various benchmarks validate the proposed multi-scale vision mamba's effectiveness.", "questions": "Could you provide more comparisons with simple baselines?\nCould you clarify the selection of different branches concerning the scanning direction and discuss its impact?", "limitations": "Reflecting on the weaknesses mentioned.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The analysis of the multi-scan strategy's success in vision mamba is intriguing and could be valuable to the research community.\nThe proposed approach is straightforward yet effective, addressing the long-range forgetting problem while significantly reducing computational costs. The approach strikes a better balance between performance and FLOPs, as demonstrated in the experiments.\nThe paper includes comprehensive experiments on widely-used datasets and tasks, with comparisons to leading neural architectures showing the proposed networks' superiority.", "weaknesses": "The authors should consider including some simple baselines in additional ablation studies. For instance, besides the half-resolution branches in the proposed MSVMamba, reducing the scanning number in vision mamba would be a useful baseline. Incorporating these simple baselines could further highlight MSVMamba's effectiveness.\nThe paper outlines that the proposed MSVMamba uses 3 half-resolution branches and 1 full-resolution branch (Equations 8-11). However, the scanning direction for these branches is not clearly described. There is a lack of discussion on how the scanning direction for the full-resolution branch is chosen and how this choice impacts performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and for the time you have dedicated to reviewing our manuscript. We greatly appreciate your feedback, which helps enhance the quality of our paper. Below, we address your concerns regarding the **baseline involving the reduction of scanning numbers(Q1)** and the **scanning direction for the full-resolution branch(Q2)**.\n\nFor clarity, we define the scanning directions as follows: Scan1 refers to the horizontal scan from the top-left corner to the bottom-right corner, and Scan2 refers to the vertical scan from the top-left corner to the bottom-right corner. Conversely, the reverse directions of Scan1 and Scan2 are denoted as Scan3 and Scan4, respectively.\n\n### **Q1: Reducing the scanning number as additional baselines.**\n\nThanks for the valuable suggestion. In response to this suggestion, we have added new baselines that involve only Scan1 (Uni-directional Scan) and a combination of Scan1 and Scan3 (Bi-directional Scan). These are now presented in **Table 3** of the uploaded PDF. Concretely, our MS2D further outperform **Uni-directional Scan** and **Bi-directional Scan** baselines by **3.0%** and **2.4%** top-1 acc respectively. We will include these results in the revised paper.\n\n### **Q2: Ablation of scanning direction for full-resolution branch.**\n\nThanks for your suggestion.  First of all, we apologize for the initial lack of clarity regarding the scanning direction in the full-resolution branch. In the original experiments, Scan1 was used as the full-resolution branch. To thoroughly explore the impact of different scanning directions, we conducted additional ablation studies as **table below**. The results indicate that while different scans yield similar accuracy, Scan1 was selected for its marginally superior performance consistency.\n\n| Full-res Scan | Scan1 | Scan2 | Scan3 | Scan4 |\n|---------------|-------|-------|-------|-------|\n| Top-1 Acc(%)  | 71.9  | 71.8  | 71.8  | 71.9  |\n\nThese findings and the rationale for our choice of scanning direction will be updated in the revised paper to ensure clarity.\n\nWe hope that these revisions adequately address your comments. **We thank you once again for your constructive feedback, which has significantly contributed to the improvement of our work**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mmvslzNwmE", "reviewer": "Reviewer_QVsi", "summary": "This paper introduces a Multi-Scale Vision Mamba (MSVMamba) for computer vision tasks. It uses a multi-scale 2D scan operation on both the original and sub-sampled features to preserve long-range information and reduce computational costs. In addition, they address the problem of channel mixing in Mamba-based models by introducing a Convolutional Feed-Forward Network (ConvFFN) module. The resultant model achieves favorable performance on image classification and a variety of downstream tasks such as detection and segmentation.", "questions": "1. How does the model compare in terms of image throughput to other Mamba-based vision models as well as CNN-based and ViT variants on a GPU ? \n\n2. Is the 2D selective scan approach faster than the naive selective scan that is originally introduced in Mamba ? \n\n3. Did authors try to scale up the model size to observe if performance is comparable to other models ?", "limitations": "Yes", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper is well-written and easy to follow. \n\n2. It attempts to address an important issue in making Mamba-based vision models more suitable for computer tasks. In order to achieve this, it introduces a hierarchical architecture, although not novel, but also integrates various modules such Convolutional Feed-Forward Network (ConvFFN) to further enhance the performance.  \n\n3. The analysis of effectiveness in 2D selective scan approach, initially introduced in VMamba, seems to be interesting and provide further insights on how to improve scan for vision tasks.", "weaknesses": "1. The major issue with this work is lack of novelty. Hierarchical Mamba-based vision models have been already introduced in VMamba and also its other variants. The MSVSS block seems to be a minor improvement over the existing VMamba block. In addition, the role of ConvFFN seems to be quite marginal. This is due to the fact that MLP blocks themselves can inherently perform channel mixing to a great extent. \n\n2. Experiments are insufficient. This work only presents three small variants of MSVMamba-N, MSVMamba-M and MSVMamba-T with the biggest model having only 33M parameters. Hence, it is not really clear how the proposed approach scales for mid to larger sized models which have better accuracies. It even can't be compared to the small variants of many models (e.g. Swin-S) due to its contrived setting. \n\n3. The paper only focuses on number of FLOPs as a representative for efficiency. However, a more practical scenario involved measuring throughput (or latency) on different devices (GPU, TPU, etc.). In particular, it is important to understand if the 2D selective scan approach introduces any significant overhead.\n\n\nPost-rebuttal: \n\nThe following issues and weaknesses were revealed during rebuttal after interactions with the authors: \n\n1. The proposed MSVMamba is slower than models such as ConvNeXt in both smaller and higher resolutions in terms of throughput (see Table 1 and Table 2 in rebuttal). This limits the practical usage of this work due to its lower throughput and can present significant challenges. \n\n2. The authors deliberately presented results from the first version of VMamba which is not optimal. Although the second version (https://arxiv.org/abs/2401.10166v2) was released 42 days before the submission deadline, the authors claim that it should be considered as concurrent. This argument is not well-founded since we need to fairly evaluate the contribution of this work against VMamba and other methods. \n\n3. The issue of limited novelty presents itself when comparing against VMamba (both first or second iterations). As expected, MSVMamba does not significantly improve the results -- authors claimed that MSVMamba addressed the long-distance forgetting issue in VMamba which is not backed up by these results.  \n\nConsidering these issues, I lower my score to strong reject (2). I encourage the authors to revise their manuscript, include best results from VMamba and try to evaluate the contributions of their work quantitatively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and for the time you have dedicated to reviewing our manuscript. We appreciate your feedback, which has helped us improve the quality of our paper.  We address your concerns in a few points as described below:\n\n### **Q1: Concerns About Novelty.**\n\nThank you for this nice concern. We agree that the hierarchical architecture is not something new, as similar structures have been utilized in VMamba and various Vision Transformers. However, our contribution lies in the introduction of **an additional hierarchy within a single layer**, which we refer to as **\"Hierarchy in Hierarchy\"** in our title and is distinct from the hierarchical designs previously explored. \n\nSpecifically, traditional hierarchical models that primarily focus on creating a feature pyramid between different stages. Our method makes the scanning process conducted on full-resolution and downsampled feature maps simultaneously, which introduces **a hierarchy inside one layer** or one stage. This **\"Hierarchy in Hierarchy\"** has not been explored in previous Vision Mamba-based works.\n\n Besides, unlike **traditional multi-scale strategies** that primarily **enhance hierarchical feature learning**, our **MS2D** module is motivated by the need to **mitigate the long-range issue** prevalent in Mamba models. This novel approach is not merely structural but is specifically designed to tackle the long-range problem in selective scanning, a critical issue for Mamba-based vision models in computational tasks.\n \nThe multi-scale 2D scan, though a straightforward strategy, effectively addresses this issue and achieves notable improvements. To further demonstrate its efficacy, we have conducted additional experiments on different scanning strategies and fine-grained tasks, the results of which are detailed in **Tables 3 and 4** of the uploaded PDF. Please check it for other details. Furthermore, ablations on tiny-size models with a 100-epoch training schedule are also reported **in table below**: \n\n| Model    | Param(M) | GFLOPs | Top-1 Acc(%) | Thru. (imgs/sec) | Train Thru. (imgs/sec) | Memory (MB) |\n|----------|----------|--------|--------------|------------------|------------------------|-------------|\n| VMamba   | 23       | 5.6    | 80.3         | 603              | 151                    | 6639        |\n| +MS2D    | 24       | 4.8    | 80.9         | 866              | 205                    | 4780        |\n| +Others  | 33       | 4.6    | 81.4         | 1092             | 331                    | 4532        |\n\nOur findings indicate that the proposed MS2D module contributes to an improvement of 0.6% in Top-1 accuracy for the tiny-size model. Other components of our model collectively contribute an additional 0.5% increase in accuracy. This ablation reveals that MS2D is more important in accuracy gain compared to other components on tiny-size model. Furthermore, the MS2D module not only enhances performance but also contributes to further speed gains and reductions in memory usage.\nIn terms of the ConvFFN in our model, it is intended to maintain consistency with established methodologies, as detailed in lines 250 to 253 of our manuscript. We acknowledge that using an MLP is also a viable alternative. We apologize for any confusion this may have caused and will ensure to clarify this point more explicitly in the revised version.\n\nWe hope this explanation helps to clarify the innovative aspects of our work and the specific challenges it addresses. \n\n\n### **Q2: Insufficient experiments.**\n\nThank you for this suggestion! It’s important to validate the scalability of the proposed model. To compensate for this, we have updated the results of the proposed model when scaling up to small and base size on ImageNet-1K. Please check further details in Table 1 of the uploaded PDF. Concretely, the MSVMamba-S and MSVMamba-B outperform VMamba by 0.6% and 0.6% top-1 acc. More experiments on downstream tasks are still undergoing due to the time limit. We will include more results of downstream tasks in the revised version. \n\n\n\n### **Q3: Efficiency Comparison.**\n\nThanks for this valuable suggestion! We apologize for the initial omission of a detailed efficiency comparison in our manuscript. To address this, we have complemented the efficiency comparison, including training/inference FPS and memory usage, with our baseline VMamba and widely-used SwinTransformer and ConvNeXt in **Table1 and 2** of the uploaded PDF for your reference.  Compared to our baseline, our proposed model achieves nearly **1.5x speedup in inference** and **2.0x speedup in training**. Additionally, it requires approximately **30% less memory**. The efficiency of our models at a 224x224 image resolution did not match that of well-established architectures such as Swin Transformer. However, when the image resolution is increased, our model achieves comparable efficiency to the Swin Transformer, which can be found in Table 2 of the uploaded PDF.\n\nIn addition, we also complemented the efficiency comparison of tiny-size model between different scanning strategies in 2D selective scan and naive selective scan in Mamba with the same setting as Table1 of the uploaded PDF:\n| Model       | Param | Thru. (imgs/sec) | Train Thru. (imgs/sec) | Memory (MB) |\n|-------------|-------|------------------|------------------------|-------------|\n| Vallina Scan| 22.9  | 602              | 151                    | 6623       |\n| Cross Scan  | 22.9  | 603              | 151                    | 6639        |\n| MS2D        | 24.2  | 866              | 205                    | 4780        |\n\nIt's worth noting that **no** FFN or ConvFFN is introduced in this comparison. As we can see, the cross scan in VMamba yields the same efficiency as the vallina scan in Mamba, while our multi-scan 2D scan further improves the efficiency .\n\n\nWe hope these revisions could address your concerns and **thank you once again for your constructive feedback**!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8rgfwjMNrB", "reviewer": "Reviewer_HsBE", "summary": "This paper presents a multi-scale Vmamba model, which incorporates multi-scale information into the design of the Vmamba architecture. Additionally, the authors analyze how the attenuation coefficient between tokens increases as the modeled distance in Vmamba grows, whereas MSVmamba alleviates this attenuation issue by reducing the sequence length. The authors have conducted extensive experiments to thoroughly validate the effectiveness of MSVmamba. Moreover, the authors have also employed SE modules and ConvFFN to further enhance the model's performance.", "questions": "Please refer to the weakness", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author's writing is clear and easy to follow.\n2. The authors analyzed the issue of long-distance forgetting and proposed an effective method to address this problem.\n3. The authors conducted comprehensive ablation experiments to validate the role of each module (MS2D, SE, ConvFFN) in MSVmamba for classification tasks.", "weaknesses": "1. Although MSVmamba has achieved better results compared to other Vmamba models, its scalability has not yet been validated. Could the authors provide training results of MSVmamba on larger models, such as those with 50M and 90M parameters, to demonstrate the model's scalability?\n\n2. The authors' ablation experiments only validated the performance of the modules in classification tasks. It would be better to further verify the roles of each module in more fine-grained tasks such as detection and segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and for the time you have dedicated to reviewing our manuscript. We greatly appreciate your feedback, which has been instrumental in enhancing the quality of our paper. Your concerns about the **scalability (Q1)** and the **ablation of our model in more fine-grained tasks(Q2)** are addressed as below.\n\n### **Q1: Scalability.**\n\nThanks for this valuable suggestion!  It’s truly important to validate the scalability of the proposed model. To compensate for this,  we have updated the results of the proposed model when scaling up to small and base size on ImageNet-1K. Please check further details in **Table 1** of the uploaded PDF. MSVMamba-S and MSVMamba-B consistently outperform the VMamba baseline by 0.6% and 0.6% top-1 acc respectively. Additionally, we are conducting further experiments on downstream tasks for small and base size models, including COCO detection and ADE20K segmentation. Due to time constraints, these results are still pending, but we commit to including them in the revised paper.\n\n### **Q2: Ablation on more fine-grained tasks.**\n\nThanks for this suggestion. As suggested, we have conducted an ablation study to evaluate the impact of each module within the detection and instance segmentation tasks on the COCO dataset, utilizing the Mask R-CNN framework. Detailed results of this study can be found in **Table 4** of the uploaded PDF. All experiments employed an ImageNet-1K pretrained backbone with a 100-epoch training schedule for initialization. The MS2D module alone brings improvements of 1.0% in box AP and 0.7% in mask AP compared to the VMamba baseline. Integrating additional components has further enhanced performance. The results on ADE20K segmentation are still undergoing due to the time limit. We will update all of the ablation details on fine-grained tasks in our revised paper.\n\n\nWe hope that these revisions could address your concerns. **We thank you once again for your constructive feedback, which has significantly contributed to the improvement of our work.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and ACs:

First and foremost, we wish to express our sincere gratitude for the time and effort you have dedicated to reviewing our manuscript. Your insightful suggestion and comments could further enhance the quality of this paper.

We have conducted additional experiments to address the key points raised by the reviewers, as detailed in the uploaded PDF. Specifically:
- Table 1 addresses scalability concerns.
- Tables 1 and 2 complement the detailed efficiency comparison.
- Table 3 compares our approach with additional baselines.
- Table 4 explores ablations on fine-grained tasks.

Please refer to these tables for more detailed information. Thank you once again for your constructive feedback!

Meta-review: The paper proposes Multi-Scale VMamba, a hierarchical visual state space model that aims to address the limitations of existing models by improving efficiency and long-range dependency learning through a multi-scale 2D scanning technique and a Convolutional Feed-Forward Network (ConvFFN). The reviewers generally appreciated the problem’s importance and the proposed solutions but raised concerns regarding the novelty of the approach, given the concurrent work on VMamba, as well as scalability and efficiency in practical applications and more detailed ablations.

In their rebuttal, the authors provided new results on scalability with small and base versions, further results on efficiency, additional ablations to clarify the usefulness of proposed components. They also argued that VMamba-v2 should be considered concurrent work, citing NeurIPS guidelines that suggest work appearing less than two months before the submission deadline should not be mandatory for comparison. The authors made a case that their contributions are orthogonal to the improvements in VMamba to which most of the reviewers agree. 

After carefully considering the reviews, the authors’ rebuttal, and the discussion, the AC acknowledges that the proposed approach has merit and presented ideas will be useful to the efforts to improve efficiency and performance of Mamba architectures. Specifically, not including VMamba-v2 in comparisons is understandable given the work is continuously improving and the specific changes were close to the submission dates (however, a more direct comparison with SoTA implementations will improve this work's quality - kindly see the comments below). The AC also recognizes the efforts made by the authors to address the reviewers’ concerns, particularly with the additional experiments and comparisons. While some concerns remain, particularly regarding scalability and efficiency, the paper presents a meaningful contribution to the field by tackling the long-range dependency issue in state space models with an interesting approach.

Considering all aspects, the AC recommends an accept decision. The authors are encouraged to further clarify their contributions and comparisons with VMamba in the final version and include all suggested changes. In line with the author's comment, "our contribution to efficiency is orthogonal to theirs, which could be integrated to achieve a better efficiency," it will be a great addition to the final version if the authors can include their proposed MS2D and ConvFFN on top of the recent version of VMamba to demonstrate its generalization and ability to provide gains over more competitive baselines. Similarly, AC will strongly suggest authors to update Fig. 1 comparison with actual GPU runtimes on x-axis, since the FLOPs comparison can give a misleading picture.
Decision: Accept (poster)