Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model

Authors: yuheng  shi
minjing  dong
chang  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 14:50:47

Abstract: Despite the significant achievements of Vision Transformers (ViTs) in various vision tasks, they are constrained by the quadratic complexity. Recently, State Space Models (SSMs) have garnered widespread attention due to their global receptive field and linear complexity with respect to the input length, demonstrating substantial potential across fields including natural language processing and computer vision. To improve the performance of SSMs in vision tasks, a multi-scan strategy is widely adopted, which leads to significant redundancy of SSMs. For a better trade-off between efficiency and performance, we analyze the underlying reasons behind the success of the multi-scan strategy, where long-range dependency plays an important role. Based on the analysis, we introduce Multi-Scale Vision Mamba (MSVMamba) to preserve the superiority of SSMs in vision tasks with limited parameters. It employs a multi-scale 2D scanning technique on both original and downsampled feature maps, which not only benefits long-range dependency learning but also reduces computational costs. Additionally, we integrate a Convolutional Feed-Forward Network (ConvFFN) to address the lack of channel mixing. Our experiments demonstrate that MSVMamba is highly competitive, with the MSVMamba-Tiny model achieving 83.0% top-1 accuracy on ImageNet, 46.9% box mAP, and 42.5% instance mAP with the Mask R-CNN framework, 1x training schedule on COCO, and 47.9% mIoU with single-scale testing on ADE20K.  Code is available at https://github.com/YuHengsss/MSVMamba.

Main paper: 
 introduction: 
 In the domain of computer vision, the extraction of features plays a pivotal role in the performance of various tasks, ranging from image classification to more complex applications like detection and segmentation. Traditionally, Convolutional Neural Networks (CNNs) [25, 40, 18, 21, 33] have been the backbone of feature extraction methodologies, prized for their linear scaling complexity and proficiency in capturing local patterns. However, CNNs often fall short in encapsulating global context, a limitation that becomes increasingly apparent in tasks requiring a comprehensive understanding of the entire visual field. In contrast, Vision Transformers (ViTs) [6, 32, 50, 43] have emerged as a compelling alternative, boasting an inherent global receptive field that allows for the direct capture of long-range dependencies within an image. Despite their advantages, ViTs are hampered by their quadratic scaling complexity concerning the input size, which significantly constrains their applicability to downstream tasks such as object detection and segmentation, where efficiency is paramount. Recently, State Space Model (SSM)-based approaches [11, 41, 8] have garnered attention for their ability to combine the best of both worlds: a global receptive field and linear scaling complexity. Notably, Mamba [9] introduces a hardware-aware and input-dependent algorithm that significantly enhances the performance and efficiency of SSMs. Inspired by Mamba's success, a burgeoning body of work has sought to leverage its advantages for vision tasks, pioneering efforts such as ViM [60] and VMamba [30] . The S6 block, developed by Mamba [9] , was originally designed for NLP tasks. To adapt S6 for vision tasks, images are first divided into patches and then flattened into a patch sequence along the scanning path. To accommodate the non-causal nature of image data, the multi-scan strategy is widely adopted for vision tasks, such as ViM [60] which enhances the sequence by summing it in both forward and reverse directions, and VMamba [30] which integrates horizontal and vertical scans. However, unlike NLP models, which can have up to billions of parameters, current vision backbones always take computational costs into consideration, i.e., the trade-off between accuracy and efficiency. This constraint on model size inherently limits the long-range modeling capabilities of SSMs in vision tasks. Taking ViM-Tiny [60] as an example, placing the cls token in the middle of the sequence yields markedly better results than positioning it at the ends. This suggests that central placement compensates for the model's limited ability to integrate distant information, highlighting the difficulties of handling long-range dependencies in parameter-constrained vision models. We refer to this as the long-range forgetting problem. In this work, we analyze how the multi-scan strategy in [60, 30] helps to alleviate this problem. Compared to the single-scan strategy, the multi-scan one allows long-range decay to manifest in various directions within 2D images. However, the increased scanning routes bring multiples of computations, significantly increasing redundancy and limiting efficiency. Thus, we aim to pursue a better trade-off between the performance and efficiency of Mamba in vision tasks.The most direct and effective method to address the long-range forgetting problem is to shorten the sequence length, which can be achieved by downsampling the feature map. However, placing all scanning routes on a downsampled feature map could result in the loss of fine-grained features and the downstream task performance. Through the visualization of different scans, we show that the decay rates could vary for different scanning routes, which motivates us to develop a hierarchical design of multi-scan. In this work, we propose a Multi-Scale 2D (MS2D) scanning strategy to alleviate the long-range forgetting problem with limited computational costs. Specifically, we divide the scanning directions of SS2D [30] into two groups: one retains the original resolution and is processed by the S6 block, while the others are downsampled, processed by the S6 block, and then upsampled, which not only shortens the sequence length for long-range dependencies learning but also alleviates redundancy. Building on the VMamba with its hierarchical architecture, we incorporate another hierarchical design within the block, creating a hierarchy within a hierarchy. Furthermore, we introduce a Convolutional Feed-Forward Network (ConvFFN) within each block to bolster the model's capability for channel-wise information exchange and local feature capture.We conduct extensive experiments to validate the effectiveness of MSVmamba across a spectrum of tasks, including image classification, object detection, and semantic segmentation. Detailed comparisons on the ImageNet-1K [3] dataset are illustrated in Fig. 1 . MSVMamba achieves a notable improvement over VMamba across different model sizes. 
generic vision backbone: 
 The evolution of generic vision backbones has significantly shaped the landscape of computer vision, transitioning from CNNs [25, 40, 18, 54, 21, 19, 42, 33] to ViTs [6, 32, 31, 50, 51, 5, 43, 57] . CNNs have been the cornerstone of vision-based models, dominating vision tasks in the early era of deep learning. The classic CNNs such as AlexNet [25] , VGG [40] , and ResNet [18] , have paved the way for numerous subsequent innovations [54, 21, 19, 42, 20, 33, 4, 49] . These designs have significantly improved performance on a wide range of vision tasks by enhancing the network's ability to capture complex patterns and features from visual data. The Vision Transformer [6] , drawing inspiration from the success of transformers [47] in natural language processing, has emerged as a formidable contender to conventional CNNs for vision-related tasks. ViT reimagines image processing by segmenting an image into patches and employing self-attention mechanisms to process these segments. This innovative approach enables the model to discern global dependencies across the entire image, a significant leap forward in understanding complex visual data. However, the ViT architecture demands considerable computational resources and extensive datasets for effective training. Moreover, its performance is intricately tied to the input sequence length, exhibiting a quadratic complexity that can escalate processing costs. In response, subsequent research has focused on developing more efficient training strategies [43, 45, 24] , hierarchical network structures [32, 31, 5, 50, 51, 15] , refined spatial attention mechanisms [57, 59, 52, 46, 16, 5] and convolution-based design [2, 29, 14, 36] to address these issues.State Space Models (SSMs) [10, 12, 11, 41, 8] have garnered increasing attention from researchers due to their computational complexity, which grows linearly with the length of the input sequence, and their inherent global awareness properties. To reduce the computational resource consumption of SSMs, S4 [11] introduced a diagonal structure and combined it with a diagonal plus low-rank approach to construct structured SSMs. Subsequently, S5 [41] and H3 [8] further enhanced the efficiency of SSM-based models by introducing parallel scanning techniques and improving hardware utilization. Mamba [9] then introduced the S6 block, incorporating data-dependent parameters to amend the Linear Time Invariant(LTI) characteristics of previous SSM models, demonstrating superior performance over transformers on large-scale datasets. In the realm of vision tasks, S4ND [35] pioneered the application of SSM models in vision tasks by treating visual data as 1D, 2D, and 3D signals. U-Mamba [34] combined CNNs with SSMs for medical image segmentation. ViM [60] and VMamba [30] integrated the S6 block into vision backbone design, employing multiple scanning directions to accommodate the non-casual nature of image data, achieving competitive results against ViTs and CNNs. Motivated by the success of ViM and VMamba, a plethora of Mamba-based works [37, 23, 7, 55, 1, 26, 56, 39] have emerged across various vision tasks, including vision backbone design [37, 23, 55] , medical image segmentation [39, 27] , and video classification [26] , showcasing the potential of SSM-based approaches in advancing the field of computer vision.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper discusses the limitations of the work in Section 5 of the main paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we first summarize the state space model in Section 3.1. Subsequently, in Section 3.2, we provide an in-depth analysis of the multi-scan strategy in existing vision Mamba models. Following the analysis, Section 3.3 tackles the redundancy and long-range dependency issue by introducing a Multi-Scale 2D (MS2D) scanning strategy. Finally, Section 3.4 details the integration of the Multi-Scale State Space (MS3) block, which incorporates the MS2D technique alongside a ConvFFN. Building upon the MS3 block, various model configurations are developed across different scales, illustrating the adaptability and scalability of our proposed approach.State Space Models. Classical State Space Models (SSMs) represent a continuous system that maps an input sequence x(t) ∈ R L to a latent space representation h(t) ∈ R N and subsequently predicts an output sequence y(t) ∈ R L based on this representation. Mathematically, an SSM can be described as follows:hEQUATIONwhere the discretized versions of A and B (denoted as A and B, respectively) can be derived, with which, Eq. 1 can be reformulated into a discretized manner as:A ∈ R N ×N , B ∈ R N ×1EQUATIONwhere I denotes the identity matrix. Afterward, the process of Eq. 2 could be implemented in a global convolution manner as:EQUATIONwhere K ∈ R L is the convolution kernel.Selective State Space Models. The Selective State Space (S6) mechanism, introduced by Mamba [9] , renders the parameters B, C, and ∆ input-dependent, thereby enhancing the performance of SSMbased models. After making B, C, and ∆ input-dependent, the global convolution kernel in Eq. 3 could be rewritten as:EQUATIONWhen processing image data using the S6 block, the 2D feature map Z ∈ R H×W ×D is flattened into a 1D sequence of image tokens, denoted as X ∈ R L×D . According to Eq. 4 and Eq. 2, the contribution of the m th token to the construction of the n th token (m < n) in S6 can be expressed as:EQUATIONTypically, the learned ∆ i A is negative, which biases the model towards prioritizing recent tokens' information. Consequently, as the sequence length increases, the exponential term e n i=m ∆iA in Eq. 5 decays significantly, resulting in minimal contributions from distant tokens. We refer to it as the long-range forgetting issue, which has also been observed in recent studies applying S6 to vision tasks [13] . Although this problem can be mitigated by increasing the number of parameters and the depth of the model, such adjustments introduce additional computational costs. Furthermore, the causal property of the S6 block ensures that information can only propagate in a unidirectional manner between tokens, preventing earlier tokens from accessing information from subsequent tokens.The inherent non-causal nature of images renders the direct application of the S6 block to visionrelated tasks less than optimal, as identified by ViM [60] . To mitigate this limitation, ViM [60] and VMamba [30] have introduced methodologies that entail scanning image features across various directions and then integrating these features. Generally, the updated token along one of the scanning routes, denoted as Scan(Z (p,q) ), where (p, q) indicates the coordinate, could be obtained by:EQUATIONIn Eq. 6, σ represents the transformation that converts a 2D feature map into a 1D sequence, and α denotes the corresponding index of Z(p, q) in the transformed 1D sequence. Afterward, results from multi-scan routes are added together to produce enhanced feature Z ′ (p, q), which can be denoted as:EQUATION)This multi-scan strategy allows tokens to access information from each other. In ViM [60] , two distinct scanning routes correspond to two different transformations in Eq. 6, specifically, the flatten and the flatten with flip transformations. Similarly, VMamba [30] extends the basic bidirectional scanning by incorporating both horizontal and vertical scanning directions, yielding four distinct scanning routes. Besides, the multi-scan strategy also alleviates the long-range forgetting problem by minimizing the effective distance between tokens. For tokens at coordinates (p 1 , q 1 ) and (p 2 , q 2 ), the strategy employs multiple scanning routes, each potentially altering their relative positions. The minimum distance across these routes is given by min k d k ((p 1 , q 1 ) → (p 2 , q 2 )), where d k represents the distance between the tokens in the sequence generated by the k-th scan. By reducing this distance, the multi-scan strategy reduces the decay of influence between distant tokens, thereby enhancing the model's ability to maintain and utilize long-range information.To more intuitively demonstrate the relationship between the multi-scan strategy and long-range decay, we visualize the exponential term e α i=1 ∆iA along the horizontal and vertical scanning directions in VMamba-Tiny with respect to the central token in Fig. 2a and Fig. 2b . Specifically, we randomly select 50 images from the ImageNet [3] validation set and compute the average decay along scanning routes at the last layer of the final stage across these images and feature dimensions. We use a higher input resolution to enhance the quality of the visualization.According to these observations, the success of the multi-scan strategy in VMamba can be attributed to its mitigation of the non-causal properties of image data and alleviation of the long-range forgetting problem. However, as the number of scanning routes increases, the computational cost also rises linearly, introducing computational redundancy. In Fig. 2c , we illustrate the maximum ratio of Fig. 2a to Fig. 2b and vice versa. While in Fig. 2d , we present a binarized version of Fig. 2c , applying a threshold of 10, which covers more than 40% of the entire figure. This phenomenon indicates that the varying decay rates across different scanning routes lead to certain routes dominating the decay dynamics, which can also be attributed to the long-range forgetting problem. The existence of dominant scanning routes implies that some scans contribute significantly more to information retention than others, leading to computational redundancy in the multi-scan strategy.As discussed in the last subsection, the contribution of tokens decays with increasing scanning distance. The most effective and direct way to alleviate the long-range forgetting problem is to reduce the number of tokens. Simultaneously, since the computational complexity of the S6 block is linearly dependent on the number of tokens, reducing the token count also enhances efficiency. Thus, an alternative approach to address the aforementioned issue is to apply the multi-scan strategy on a downsampled feature map. However, setting all scans on a downsampled feature map will ignore fine-grained features and result in unavoidable information loss. Thus, scanning along the full-resolution feature map is also essential.Motivated by these considerations, we introduce a simple yet effective Multi-Scale 2D scanning(MS2D) strategy, as depicted in Fig. 3 . Our approach commences with the generation of EQUATIONEQUATION)where σ is transformation that convert 2D feature maps into 1D sequences used in SS2D, and Y is the processed sequence. These processed sequences are converted back into 2D feature maps, and the downsampled feature maps are interpolated for merging:EQUATIONEQUATIONwhere γ is the inverse transformation of σ and Z ′ is the feature map enhanced by MS2D. The downsampling operation reduces the sequence length by a factor of s 2 , which also shortens the distance between tokens in Eq. 5 by a factor of s 2 , thereby alleviating the long-range forgetting problem. As the computational complexity of a single S6 block is O(9LDN ) [9] , where N denotes the SSM dimension, replacing SS2D with MS2D reduces the total sequence length across four scans from 4L to (1+3/s 2 )L, thereby improves the efficiency. Practically, the downsampling rate is set to 2. It is worth noting that sequences from Z 2 are processed by the same S6 block. This approach maintains the same accuracy as using multiple S6 blocks for different scanning routes while effectively reducing the number of parameters.To better illustrate the alleviation of the long-range forgetting problem, we also provide empirical evidence, as shown in Fig. 4 . We compare the decay along scanning routes in the SS2D of VMamba and our MS2D, focusing on the last token with the same configuration as Fig. 2 . The decay maps in downsampled features are interpolated back. As observed, the decay rate along scanning routes in downsampled maps is significantly alleviated, enhancing the capability to capture global information.In this study, we extend the capabilities of the VMamba framework by substituting its VSS block with our Multi-Scale State Space (MS3) block. The architectural framework of the MS3 block is delineated in Fig. 5 , comprising a Multi-Scale Vision Space State(MSVSS) component and a Convolutional Feed-Forward Network (ConvFFN). The MSVSS component is devised by adapting the vision state space framework in VMamba, substituting the SS2D with an MS2D to further introduce a hierarchy design in a single layer. Additionally, a Squeeze-Excitation (SE) [20] block is integrated subsequent to the multi-scale 2D scanning, as informed by related literature [23, 37] . Diverging from the conventional focus on token mixing in prior vision Mamba architectures [60, 30, 23] , our design introduces a channel mixer to augment the flow of information across different channels, aligning with the structural paradigms of typical vision transformers. In concordance with preceding studies [52, 48, 14, 51] , the ConvFFN which consists of a depth-wise convolution and two fully connected layers is employed as the channel mixer. Upon the amalgamation of MSVSS and ConvFFN within the MS3 block, meticulous adjustments are made to the number of blocks to ensure a comparable computational budget, facilitating a fair comparison.To empirically validate the efficacy of our proposed modifications, we introduce model variants across different scales. These variants, namely Nano, Micro, Tiny, Small and Base, are characterized by their parameter counts of 7M, 12M, 32M, 50M and 88M respectively. In terms of computational expenditure, these models require 0.9, 1.5, 5.1, 8.8 and 15.5 GFLOPs correspondingly, demonstrating a scalable approach to model design that accommodates varying computational constraints. For models above tiny size, the multiplicative branch of the MSVSS block is removed as informed by the VMambav3 [30] for better performance. Detailed architectures are shown in Appendix A. Settings. Our models are trained and tested on the ImageNet-1K dataset [3] . In alignment with previous works [32, 30, 23] , all models undergo training for 300 epochs, with the initial 20 epochs dedicated to warming up. The training utilizes a batch size of 1024 across 8 GPUs.We employ the AdamW optimizer, setting the betas to (0.9, 0.999) and momentum to 0.9. The learning rate is managed through a cosine decay scheduler, starting from an initial rate of 0.001, coupled with a weight decay of 0.05. Additionally, we leverage the exponential moving average (EMA) and implement label smoothing with a factor of 0.1 to enhance model performance and generalization. During testing, images are center cropped with the size of 224×224. When dealing with the routes for multi-scale scanning, we select top-left to the bottom-right for dealing with the full-resolution feature map while the other three scans are responsible for scanning the downsampled feature map.Tab. 1 showcases our MSV-Mamba models against established CNNs, ViTs, and SSM-based models on ImageNet-1K. MSVMamba-T, with 32M parameters and 5.1G FLOPs, achieves 83.0% top-1 accuracy, outperforming similar-cost SSM-based LocalVMamba-T. The MSVMamba-B model attains 84.3% accuracy with 88M parameters and 15.5G FLOPs, exceeding VMambav3-B by 0.4%. These results highlight MSVMamba's efficiency and scalability, offering a robust option for highaccuracy, resource-efficient model design.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Code, models and logs are provided in the github repo.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
object detection: 
 Setup. We evaluate our MSVMamba on the MSCOCO [28] dataset using the Mask R-CNN [17] framework for object detection and instance segmentation tasks. Following previous works [32, 30] , Table 3 : We present the results of semantic segmentation on the ADE20K dataset [58] using the UperNet framework [53] . The computational complexity, measured in FLOPs, is calculated for input dimensions of 512 × 2048. The abbreviations "SS" and "MS" refer to single-scale and multi-scale testing, respectively. Setup. Consistent with the methodologies used in Swin [32] and VMamba [30] , we utilize the UperHead [53] framework atop an ImageNet pre-trained MSVMamba backbone. The training process is conducted over 160K iterations with a batch size of 16. We employ the AdamW optimizer with a learning rate set at 6 × 10 -5 . Our experiments are primarily conducted using a default input resolution of 512 × 512. Additionally, we also incorporate Multi-Scale (MS) testing to assess performance variations.Results. We present the detailed results of our model and other competitors in Tab. 3, which includes both single-scale and multi-scale testing.Our MSVMamba consistently outperforms the Swin and ConNeXt models in the tiny variant by margins of +2.2 and +1.6 mIoU, respectively.To validate the effectiveness of the proposed modules, we conducted a comprehensive ablation study. Specifically, we scaled the VMamba-Tiny model by setting its embedding dimension d to 48, the state space dimension N to 8, and the number of blocks in the four different stages to [1, 2, 4, 2] . The scaled model, referred to as VMamba-Nano, has parameters and computational costs of 4.4M and 0.87GFLOPs, respectively. This model and the standard tiny-sized models serve as the baseline for our ablation experiments. Models in ablation study are conducted over a training schedule of 100 epochs on ImageNet-1K to reduce training time. Besides, the AP b and AP m under Mask R-CNN with 1x schedule on COCO dataset are also reported for nano-size models. 4 . Furthermore, we conducted an ablation on the number of scans in the multi-scale scan, considering both full-resolution and half-resolution branches on nano-size models. The results are shown in Tab. 6. Placing all scans in the half-resolution branch led to a significant loss of finegrained features, resulting in a substantial decrease in model accuracy. Positioning two or three scans in the full resolution branch, compared to just one, resulted in accuracy improvements of 0.1% and 0.6%, respectively, but introduced an additional computational cost of approximately 12% and 25%. Allocating four scans to the full resolution branch, effectively reverting to the SS2D method, increased the computational cost by 34% while only improving accuracy by 0.4%. For an optimal trade-off between computational cost and accuracy, we select one scan in the full-resolution branch as the default setting. Building upon the MS2D foundation, we introduce an SE block following EfficientV-Mamba [37] , which further enhanced accuracy by 0.5% with minimal additional computational cost.Experiments related to tiny-size model are reported in Tab. 5. Our findings indicate that the proposed MS2D module contributes to an improvement of 0.6% in Top-1 accuracy for the tiny-size model while other components of our model collectively contribute an additional 0.5% increase in accuracy. The MS2D module not only enhances performance but also contributes to further speed gains and reductions in memory usage. Since MS2D is orthogonal to the updates in VMambav3, they can be combined for further enhancements.In the last row of Tab. 5, we present the results after adopting optimization used in VMambav3, which include reducing the ssm ratio and eliminating the entire multiplicative branch. These optimizations are preserved across the tiny, small, and base size models.On ConvFFN. Upon replacing SS2D with MS2D and incorporating a SE block, we constructed a model that utilizes ConvFFN as a channel mixer. When only using SSM, the model exhibited insufficient information exchange between channels. The integration of ConvFFN as a channel mixer significantly enhanced the model's capability for inter-channel information interaction. As indicated in Tab. 4, the addition of ConvFFN resulted in an additional accuracy improvement of 2.0%. Besides, we set the state space dimension N = 1 and stacked one more block to further enhance the capability of capturing long-range information while maintaining a roughly constant computational cost. This operation resulted in an additional accuracy improvement of 0.7%, as shown in Tab. 4. To maintain a roughly equivalent computational cost, we adjusted the number of blocks within the model. When integrating the ConvFFN and setting the state space dimension N = 1, we meticulously calibrate the quantity of blocks to maintain a nearly constant computational cost, measured in GFLOPs. 
limitations: 
 The design of multi-scale VMamba aims to tackle the long-range forgetting problem of Mamba models with limited parameters on vision tasks. Although the proposed model has proven to be effective, its scalability remains to be explored since this issue can also be alleviated by increasing model sizes. 
conclusion: 
 In this paper, we introduced Multi-Scale VMamba (MSVMamba), an SSM-based vision backbone that leverages the advantages of linear complexity and global receptive field. We developed the Multi-Scale 2D (MS2D) scanning technique to minimize computational redundancy and alleviate the long-range forgetting problem in parameter-limited vision models. Additionally, we incorporated the Convolutional Feed-Forward Network (ConvFFN) to enhance the exchange of information between channels, thereby significantly improving the performance of our model. Our experiments demonstrate that MSVMamba consistently outperforms popular models from various architectures, including ConvNeXt, Swin Transformer, and VMamba, in image classification and downstream tasks. A Network Architecture Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: All the training and test details are properly stated.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The paper does not report error bars or statistical significance information because the results of multiple experiments are consistent and show minimal variation. Therefore, the precision of the results is deemed sufficient without the need for additional statistical measures.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All assets used in the paper, including code, data, and models, are properly credited to their original creators. The licenses and terms of use for these assets are explicitly mentioned in the paper, and their usage complies with the specified terms. Proper citations and acknowledgments are provided to ensure that the contributions of the original creators are recognized and respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To further demonstrate the effectiveness of our MS2D, we add more baselines that involve only Scan1 (Uni-directional Scan) and a combination of Scan1 and Scan3 (Bi-directional Scan). These results are presented in Tab. 8. Concretely, our MS2D further outperform Uni-directional Scan and Bi-directional Scan baselines by 3.0% and 2.4% top-1 acc respectively.In Tab. 9, we explore the impact of full-resolution scanning directions. The results indicate that while different scans yield similar accuracy, Scan1 was selected for its marginally superior performance consistency. We report detailed throughput comparison of Swin [32] , ConvNeXt [33] , and VMamba [30] in Tab. 10. All models are tested on a RTX 4090 GPU with a batch size of 128 and FP32 precision at an image resolution of 224.We provide visualizations in Fig. 6 comparing the proposed MS2D and SS2D configurations in VMamba. These visualizations are generated by converting the S6 layer into an attention format, as demonstrated by VMamba [30] . The results clearly show that the full-resolution in MS2D scan captures more detailed features, whereas the scans at half resolution primarily focus on broader architectural details, compared to SS2D. The proposed hierarchical scanning pattern facilitates the current layer's ability to discern and amalgamate features across various levels of abstraction.Table 10 : Efficiency comparison with our baseline VMamba [30] and widely-used Swin Transformer [32] and ConvNeXt [33] . Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper proposes Multi-Scale VMamba, a hierarchical visual state space model that aims to address the limitations of existing models by improving efficiency and long-range dependency learning through a multi-scale 2D scanning technique and a Convolutional Feed-Forward Network (ConvFFN). The reviewers generally appreciated the problem’s importance and the proposed solutions but raised concerns regarding the novelty of the approach, given the concurrent work on VMamba, as well as scalability and efficiency in practical applications and more detailed ablations.

In their rebuttal, the authors provided new results on scalability with small and base versions, further results on efficiency, additional ablations to clarify the usefulness of proposed components. They also argued that VMamba-v2 should be considered concurrent work, citing NeurIPS guidelines that suggest work appearing less than two months before the submission deadline should not be mandatory for comparison. The authors made a case that their contributions are orthogonal to the improvements in VMamba to which most of the reviewers agree. 

After carefully considering the reviews, the authors’ rebuttal, and the discussion, the AC acknowledges that the proposed approach has merit and presented ideas will be useful to the efforts to improve efficiency and performance of Mamba architectures. Specifically, not including VMamba-v2 in comparisons is understandable given the work is continuously improving and the specific changes were close to the submission dates (however, a more direct comparison with SoTA implementations will improve this work's quality - kindly see the comments below). The AC also recognizes the efforts made by the authors to address the reviewers’ concerns, particularly with the additional experiments and comparisons. While some concerns remain, particularly regarding scalability and efficiency, the paper presents a meaningful contribution to the field by tackling the long-range dependency issue in state space models with an interesting approach.

Considering all aspects, the AC recommends an accept decision. The authors are encouraged to further clarify their contributions and comparisons with VMamba in the final version and include all suggested changes. In line with the author's comment, "our contribution to efficiency is orthogonal to theirs, which could be integrated to achieve a better efficiency," it will be a great addition to the final version if the authors can include their proposed MS2D and ConvFFN on top of the recent version of VMamba to demonstrate its generalization and ability to provide gains over more competitive baselines. Similarly, AC will strongly suggest authors to update Fig. 1 comparison with actual GPU runtimes on x-axis, since the FLOPs comparison can give a misleading picture.

Reviews: review: paper_summary: paper_summary: The paper introduces Multi-Scale VMamba (MSVMamba), a novel vision backbone model that leverages State Space Models (SSMs) to address the challenges of quadratic complexity in Vision Transformers (ViTs). The proposed Multi-Scale 2D Scanning (MS2D) and Convolutional Feed-Forward Network (ConvFFN) contributes to the final performance.
strength_weakness: strength_weakness: - This paper lacks the efficiency comparison among training/inference FPS and memory usage on real GPU devices.
- The proposed multi-resolution processing seems to contain too many inductive biases. I wonder if the gains would be reduced when applying to a standard-ablation-size model, i.e., VMamba-T.
questions: questions: Please refer to the weakness
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces a novel vision backbone model, MSVMamba, which incorporates State Space Models (SSMs) to address limitations in computational efficiency and long-range dependency capture in vision tasks. The model utilizes a multi-scale 2D scanning technique and a Convolutional Feed-Forward Network (ConvFFN) to improve performance with limited parameters.
strength_weakness: strength_weakness: 1. Lack of novelty. 

The paper propose MSVMamba, the main contribution is shown in table.4. MS2D, SE, ConvFFN have already been proposed in previous papers. The main improvements come from existing knowledge. 

2. Speed

The paper does not systematically measure the speed of their model on all the tasks and scales. The training and inference of MSVMamba could be slow compared with current models, like CAFormer, Conv2Former, CSwinTransformer.

3. Scalability

The model in the experiments is relatively small. Model with more than 300M or 600M parameters could show whether MSVMamba does perform better than current sota model. Now the experiments just show that the model converges quickly w.r.t. model size.
questions: questions: Please refer to the weakness.
limitations: limitations: Please refer to the weakness.
paper_summary: paper_summary: This paper presents a multi-scale vision mamba aimed at improving the performance of state space models (SSMs) in vision tasks while maintaining efficiency. The motivation stems from analyzing the multi-scan strategy in vision mamba, where the authors link its success to alleviating the long-range forgetting issue of SSMs in vision tasks. To address this problem effectively, the authors propose a multi-scale 2D scanning technique on both original and downsampled feature maps, which reduces the number of tokens in the multi-scan strategy. This method enhances long-range dependency learning and cuts down computational costs. Additionally, a ConvFFN is incorporated to overcome channel mixing limitations. Experimental results across various benchmarks validate the proposed multi-scale vision mamba's effectiveness.
strength_weakness: strength_weakness: The authors should consider including some simple baselines in additional ablation studies. For instance, besides the half-resolution branches in the proposed MSVMamba, reducing the scanning number in vision mamba would be a useful baseline. Incorporating these simple baselines could further highlight MSVMamba's effectiveness.
The paper outlines that the proposed MSVMamba uses 3 half-resolution branches and 1 full-resolution branch (Equations 8-11). However, the scanning direction for these branches is not clearly described. There is a lack of discussion on how the scanning direction for the full-resolution branch is chosen and how this choice impacts performance.
questions: questions: Could you provide more comparisons with simple baselines?
Could you clarify the selection of different branches concerning the scanning direction and discuss its impact?
limitations: limitations: Reflecting on the weaknesses mentioned.
paper_summary: paper_summary: This paper introduces a Multi-Scale Vision Mamba (MSVMamba) for computer vision tasks. It uses a multi-scale 2D scan operation on both the original and sub-sampled features to preserve long-range information and reduce computational costs. In addition, they address the problem of channel mixing in Mamba-based models by introducing a Convolutional Feed-Forward Network (ConvFFN) module. The resultant model achieves favorable performance on image classification and a variety of downstream tasks such as detection and segmentation.
strength_weakness: strength_weakness: 1. The major issue with this work is lack of novelty. Hierarchical Mamba-based vision models have been already introduced in VMamba and also its other variants. The MSVSS block seems to be a minor improvement over the existing VMamba block. In addition, the role of ConvFFN seems to be quite marginal. This is due to the fact that MLP blocks themselves can inherently perform channel mixing to a great extent. 

2. Experiments are insufficient. This work only presents three small variants of MSVMamba-N, MSVMamba-M and MSVMamba-T with the biggest model having only 33M parameters. Hence, it is not really clear how the proposed approach scales for mid to larger sized models which have better accuracies. It even can't be compared to the small variants of many models (e.g. Swin-S) due to its contrived setting. 

3. The paper only focuses on number of FLOPs as a representative for efficiency. However, a more practical scenario involved measuring throughput (or latency) on different devices (GPU, TPU, etc.). In particular, it is important to understand if the 2D selective scan approach introduces any significant overhead.


Post-rebuttal: 

The following issues and weaknesses were revealed during rebuttal after interactions with the authors: 

1. The proposed MSVMamba is slower than models such as ConvNeXt in both smaller and higher resolutions in terms of throughput (see Table 1 and Table 2 in rebuttal). This limits the practical usage of this work due to its lower throughput and can present significant challenges. 

2. The authors deliberately presented results from the first version of VMamba which is not optimal. Although the second version (https://arxiv.org/abs/2401.10166v2) was released 42 days before the submission deadline, the authors claim that it should be considered as concurrent. This argument is not well-founded since we need to fairly evaluate the contribution of this work against VMamba and other methods. 

3. The issue of limited novelty presents itself when comparing against VMamba (both first or second iterations). As expected, MSVMamba does not significantly improve the results -- authors claimed that MSVMamba addressed the long-distance forgetting issue in VMamba which is not backed up by these results.  

Considering these issues, I lower my score to strong reject (2). I encourage the authors to revise their manuscript, include best results from VMamba and try to evaluate the contributions of their work quantitatively.
questions: questions: 1. How does the model compare in terms of image throughput to other Mamba-based vision models as well as CNN-based and ViT variants on a GPU ? 

2. Is the 2D selective scan approach faster than the naive selective scan that is originally introduced in Mamba ? 

3. Did authors try to scale up the model size to observe if performance is comparable to other models ?
limitations: limitations: Yes
paper_summary: paper_summary: This paper presents a multi-scale Vmamba model, which incorporates multi-scale information into the design of the Vmamba architecture. Additionally, the authors analyze how the attenuation coefficient between tokens increases as the modeled distance in Vmamba grows, whereas MSVmamba alleviates this attenuation issue by reducing the sequence length. The authors have conducted extensive experiments to thoroughly validate the effectiveness of MSVmamba. Moreover, the authors have also employed SE modules and ConvFFN to further enhance the model's performance.
strength_weakness: strength_weakness: 1. Although MSVmamba has achieved better results compared to other Vmamba models, its scalability has not yet been validated. Could the authors provide training results of MSVmamba on larger models, such as those with 50M and 90M parameters, to demonstrate the model's scalability?

2. The authors' ablation experiments only validated the performance of the modules in classification tasks. It would be better to further verify the roles of each module in more fine-grained tasks such as detection and segmentation.
questions: questions: Please refer to the weakness
limitations: limitations: N/A

Comments: title: Please allow me to step forward and share my thoughts on this discussion thread
Thanks for addressing my concerns
final reviewer
Post-Rebuttal Score Update
Comment by Reviewer
Further Comment by Reviewer
Comment by Reviewer
Reviewer's Response to Rebuttal
Comment by Reviewer
Discrepancies in VMamba Benchmarks
comment: Thanks for your response. Your responses solve my concern to some extent. I maintain my score.
VMamba is a new and promising backbone network in computer vision that warrants more research efforts. However, from what I understand, VMamba-based models currently face challenges in achieving better efficiency than CNNs and ViTs. This is why numerous recent works aim to further enhance Mamba to make it viable for real-world applications. In my opinion, the authors’ claim of achieving a 1.5x speedup in inference and a 2.0x speedup in training is commendable and meets expectations.

I reviewed the NeurIPS 2024 FAQ for Authors on the official website, and it appears that the second version of VMamba, which appeared on arXiv this April, can indeed be considered concurrent with this submission. Therefore, I have no concerns about the current comparative experiments, and it’s great to see the authors include new comparison results with that second version of VMamba.

Overall, I am inclined to accept the paper and am open to further discussions with other reviewers.
I'd like to thank the authors' response to my question. I think the response is clear enough to address my confusion.
Thanks the author's efforts. My concerns are well addressed. I will keep my initial rates (7).
We thank Reviewer QVsi for the detailed summary in the post-rebuttal, where they pointed out three issues during the discussion. We would like to clarify each issue in details.

Q1: The proposed MSVMamba is slower than models such as ConvNeXt.

Note that our model design is specifically tailored for the Mamba architecture, allowing any subsequent optimizations related to efficiency to be directly inherited. Thus, we focused on comparing the VMamba baseline. It’s important to note that while our baseline VMamba is indeed much slower than ConvNeXt, it remains highly valuable and has inspired many subsequent works. For example, the first version of VMamba was released on 2024.01.18 and its citation is 300+ currently, which highlights its importance. Given the observation that our baseline VMamba is much slower than existing CNNs, we are motivated to improve its efficiency. Our proposed model achieves significant improvements, with nearly 1.5x speedup in inference and 2.0x speedup in training compared to our baseline VMamba. Taking other versions of VMamba into consideration, our contribution to efficiency is orthogonal to theirs, which could be integrated to achieve a better efficiency.  Concretely, our core contribution, the MS2D, focuses on the optimization of CrossScan in our baseline VMamba. In all three versions of VMamba, CrossScan inherently exists, and the speedup mainly comes from implementational optimizations and hyper-parameter adjustments in the Mamba block. Thus, our improvement is orthogonal to the subsequent techniques used in VMamba and can be integrated with them seamlessly.


Q2: The authors deliberately presented results from the first version of VMamba which is not optimal.

As clarified in our previous comments, the second version of VMamba is released 42 days before the submission deadline and should be considered as concurrent work. This argument is well-founded by the NeurIPS 2024 official instructions (https://neurips.cc/Conferences/2024/PaperInformation/NeurIPS-FAQ). You can find the question “**What is the policy on comparisons to recent work?**” in the Section Submission format and content, and the corresponding answer is “**Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions.  Authors are not expected to compare to work that appeared only a month or two before the deadline.**” Since 42 days is obviously less than 2 months, we do not include the second version of VMamba in our comparison and focus on the comparison with the first version.

Although the second version of VMamba is defined as concurrent work based on the NeurIPS 2024 official instructions, we provide a detailed comparison with the second version as a reference during the rebuttal discussion because it is strongly recommended by Reviewer QVsi. Concretely, the Top-1 accuracy in the second version of VMamba on ImageNet still lagged behind our model by 0.3%, 0.5% and 0.4% for the tiny, small and base model respectively. This comparison highlights the superiority of our method over the concurrent work.


Q3: The issue of limited novelty presents itself when comparing against VMamba (both first or second iterations).

In comparison with VMamba (the first version of VMamba), our models not only exhibit a 0.6% improvement in Top-1 accuracy on ImageNet across different model sizes but also show nearly 1.5x speedup in inference and 2.0x speedup in inference and training FPS. 
The Multi-Scale 2D (MS2D) module, as the core contribution of our work, is well motivated by the need to mitigate the long-range issue prevalent in Mamba models. This is a critical challenge that has not been adequately addressed by existing methods. The subsequent optimization in VMamba focuses on the implementational optimizations and hyper-parameter adjustments in the Mamba block. Thus, the long-range issue inherently exists. VMamba utilizes multi-scan strategy with redundant FLOPs to alleviate this issue, while MS2D utilizes multi-scale strategy to efficiently tackle this issue. As our contribution in accuracy and efficiency gain is also orthogonal to the subsequent versions of VMamba, these techniques could be integrated for further improvement.

We hope this clarifies our position and we appreciate your understanding.
I thank the authors for providing the rebuttal and their engagement during this period. 

Considering all aspects, I have decided to lower my score to strong reject (2). Here's the key reasons behind this decision: 

1. The proposed MSVMamba is slower than models such as ConvNeXt in both smaller and higher resolutions in terms of throughput (see Table 1 and Table 2 in rebuttal). This limits the practical usage of this work due to its lower throughput and can present significant challenges.

2. The authors deliberately presented results from the first version of VMamba which is not optimal. Although the second version (https://arxiv.org/abs/2401.10166v2) was released 42 days before the submission deadline, the authors claim that it should be considered as concurrent. This argument is not well-founded since we need to fairly evaluate the contribution of this work against VMamba and other methods.

3. The issue of limited novelty presents itself when comparing against VMamba (both first or second iterations). As expected, MSVMamba does not significantly improve the results -- authors claimed that MSVMamba addressed the long-distance forgetting issue in VMamba which is not backed up by these results.

I encourage the authors to revise their manuscript, include best results from VMamba and try to evaluate the contributions of their work quantitatively.
Thank you for your response. In this case, as shown in Table 1, there exists a substantial gap between the throughput of well-established models such as Swin and ConvNext and the proposed model. 

Even for larger resolutions, models such as ConvNeXt-T are significantly faster than the proposed MSVMamba-T. This may complicate the usage of this model where throughput is important (which is almost all use-cases at this time). And to add to this, ConvNeXt itself is not a very fast model in terms of throughput. 

Regarding the comparison between different versions of VMamba (released in less than two months after each other) and MSVMamba, the authors initially claimed that the 2nd iteration (available 42 days before submission deadline) should be disregarded. However, a later comment provided the performance for both models. Upon closer examination, we observe that MSVMamba does not significantly improve the Top-1 performance of the VMamba model. 

I appreciate the authors' efforts in this stage of the rebuttal. I believe I have all information I need to make a final decision.
Dear Reviewer QVsi,

Thank you for your feedback. We apologize for any confusion regarding Table 2.

To clarify, an input resolution of 224x224 and batch size of 128 has already been reported in Table 1 of the submitted PDF. If Table 2 were to report results for the same resolution and batch size, it would indeed be redundant with Table 1. Instead, Table 2 is intended to provide additional efficiency details across different resolutions.

We hope this clarification addresses your concerns and helps to clear up any misunderstandings.
I thank the authors for providing responses to my feedback. 

Regarding the throughput comparison, the authors mention:
" With the input resolution increase, the memory cost for models also increase significantly". 

However, my request was to use an input resolution of **224x224** and batch size of **128** which is very standard and has been presented in previous works before. This setup should be achievable on any consumer GPUs with moderate VRAMs. 

Can the authors only provide a comparison between MSVMamba, ConvNeXt and Swin Transformers in terms of throughput using this setup ? I would like to understand how your model compares to well-established models in the literature.
Table: Comparsion of different versions of VMamba and our models.

| Model       | Param(M) | GFLOPs | Top-1 Acc(%) | Thru. (imgs/sec) | Train Thru. (imgs/sec) | Memory (MB) |
|-------------|----------|--------|--------------|------------------|------------------------|-------------|
| VMamba-T [01-18]  | 23       | 5.6    | 82.2         | 603             | 151                    | 6639        |
| VMambav9-T [04-10]  | 31       | 4.9    | 82.5         | 1135             | 313                    | 5707        |
| MSVMamba-T  | 33       | 4.6    | 82.8         | 1092             | 331                    | 4532        |
| VMamba-S [01-18]  | 44       | 11.2    | 83.5         | 425             | 106                    | 6882        |
| VMambav9-S [04-10]  | 50       | 8.7    | 83.6         | 749              | 207                    | 5785        |
| MSVMamba-S  | 51       | 9.2    | 84.1         | 665              | 232                    | 4910        |
| VMamba-B [01-18]  | 76       | 18.0    | 83.7         | 314             | 77                    | 8853        |
| VMambav9-B [04-10]  | 89       | 15.4   | 83.9         | 542              | 151                    | 7918        |
| MSVMamba-B  | 91       | 16.3   | 84.3         | 476              | 127                    | 6347        |
Dear Reviewer QVsi,

We respectfully agree with your opinion that we must include the best results from VMamba.

To address this concern and provide further clarification, we have uploaded a detailed comparison of different versions of VMamba and our models in the rebuttal comment box for your reference. This table offers a comprehensive overview of the performance metrics across various model versions.

We hope this additional information helps to clear any misunderstandings. Furthermore, we commit to including the best results from VMamba in our revised paper to ensure a thorough and fair comparison.

We appreciate your diligence in reviewing our work and look forward to your feedback on this updated information.
Dear Reviewer QVsi,

We sincerely appreciate your prompt feedback and the opportunity to clarify our position. We apologize for any confusion regarding the evidence we presented earlier. For further clarification, we update the concrete link in our previous comment here. The official NeurIPS 2024 FAQ (https://neurips.cc/Conferences/2024/PaperInformation/NeurIPS-FAQ) states:

"**Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions**"

This definition of concurrent work is provided by the official NeurIPS instructions, not by us. We adhered to these guidelines in our comparisons and citations.

We are genuinely grateful for your strong sense of responsibility in reviewing our work. We hope this explanation resolves any misunderstandings.

Thank you again for your thorough review and commitment to maintaining high standards in academic research.

Sincerely,
Authors of Paper 3968
I would like to thank the authors for providing their response. 

However, the argument that the "the second iteration of the VMamba paper (released 42 days before deadline) is concurrent and should not be considered while the first iteration is not concurrent" is indeed not well-founded and adds more confusion for other reviewers and readers. 

Even if we accept this work to Neurips, the authors MUST include the best results from VMamba. Hence, as a reviewer, it is our duty to evaluate the effectiveness of this work in a fair manner which includes reasonable comparisons to previous efforts.
Thanks for your feedback and we are very appreciated for this opportunity for further clarification. We will do our best to **achieve the goal of comparison to previous models to further understand the contributions of the proposed effort**.


### Q1: Why not include the results of the second version results in VMamba for comparison?
We thank the reviewer for proposing this reasonable concern. As you mentioned, the second version of VMamba (VMambav9) was available since **Apr 10**, and the DDL of submission is **May 22**. According to NeurIPS 2024 official instruction, we argue that the second version of VMamba is not expected to be included in our comparison.

The detailed evidence is provided by the NeurIPS 2024 official instruction (in NeurIPS-FAQ page). In Section **Submission format and content**, one question is “What is the policy on comparisons to recent work?”, and the given answer is “**Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions.  Authors are not expected to compare to work that appeared only a month or two before the deadline.**” Thus, given the timeline of VMamba, we only include the first version of VMamba in our comparison.

Besides, even if we take into consideration the contribution of VMambav9, our core contribution MS2D discusses and tackles a general long-range forgetting problem of VMamba, which is orthogonal to the contributions of other versions of VMamba. Thus, the appearance of existing VMamba variants does not weaken our contributions.




### Q2: Updated version of Table 1 for VMamba and MSVMamba models alone.
As suggested, we add the updated version of Table 1 as below for your reference, which updated the results of the VMamba with **VMambav9** in the v2 version of VMamba paper. 

| Model       | Param(M) | GFLOPs | Top-1 Acc(%) | Thru. (imgs/sec) | Train Thru. (imgs/sec) | Memory (MB) |
|-------------|----------|--------|--------------|------------------|------------------------|-------------|
| VMambav9-T  | 31       | 4.9    | 82.5         | 1135             | 313                    | 5707        |
| MSVMamba-T  | 33       | 4.6    | 82.8         | 1092             | 331                    | 4532        |
| VMambav9-S  | 50       | 8.7    | 83.6         | 749              | 207                    | 5785        |
| MSVMamba-S  | 51       | 9.2    | 84.1         | 665              | 232                    | 4910        |
| VMambav9-B  | 89       | 15.4   | 83.9         | 542              | 151                    | 7918        |
| MSVMamba-B  | 91       | 16.3   | 84.3         | 476              | 127                    | 6347        |


As we can see, **VMambav9** exhibits **higher inference speed**  and **similar training speed** compared to our model.  However, its **Top-1 accuracy** on ImageNet still **lagged behind our model by 0.3%, 0.5% and 0.4%** for the tiny, small and base model respectively. 

### Q3: Why Table 2 adopt a batch of 32 instead of a standard 128 for comparison?

We apologize for this confusion as this explanation should be contained in the caption of Table 2. When testing the throughput, we follow the code provided in the official VMamba repo ( L580 in utils.py of analysis) and adjusted the batch size downwards to fit the GPU’s memory constraints. With the input resolution increase, the memory cost for models also increase significantly. Take Swin-tiny as an example, it takes more than 20000 MB memory with an input resolution of 768 and a batch size of 32. Adopting a batch size of 128 or 64 will cause Out-Of-Memory error in some model. To make all comparisons under the same configuration, we take a batch size of  32 in the Table 2 of the uploaded PDF as a compromise.


We hope that these revisions could address your concern and we sincerely look forward to your feedback.
We thank the reviewer for proposing this reasonable concern. As you mentioned, the second version of VMamba (VMambav9) was available since **Apr 10**, and the DDL of submission is **May 22**. According to NeurIPS 2024 official instruction, we argue that the second version of VMamba is not expected to be included in our comparison.

The detailed evidence is provided by the NeurIPS 2024 official instruction (in NeurIPS-FAQ page). In Section **Submission format and content**, one question is “What is the policy on comparisons to recent work?”, and the given answer is “**Papers appearing less than two months before the submission deadline are generally considered concurrent to NeurIPS submissions.  Authors are not expected to compare to work that appeared only a month or two before the deadline.**” Thus, given the timeline of VMamba, we only include the first version of VMamba in our comparison.
I would like to thank the authors for providing responses to my feedback as well as uploading the rebuttal.

I have the following concerns:

1. As mentioned in my other comment, the reported Top-1 accuracy from VMamba seem to be lower than their existing benchmarks. The authors claim that they have used the results from the first iteration of the VMamba arXiv submission. However, the benchmarks I refer to can also be found in the 2nd iteration which was available since Apr 10:

https://arxiv.org/pdf/2401.10166v2

Can the authors provide an updated version of Table 1 for VMamba and MSVMamba models alone (Tiny, Small and Base) with the updated Top-1 accuracy as reported in the above arXiv submission ?  it should suffice to just reply to my comment with this table.

2. The reported numbers in Table 2 for efficiency comparison are for batch size 32. Why not report numbers for standard 224x224 resolution with batch size 128 which matches the same setup in Table 1 ?

Our goal here is to have a fair comparison to previous models to further understand the contributions of the proposed effort.
I appreciate the authors' response, but why not using the best results/models when comparing to them ? 

The first version in Jan 18 was a preliminary effort which may not be optimized. And our goal is not to just show better performance, but also provide a meaningful comparison to previous efforts.

The results I refer to can also be found in the second iteration of their arXiv paper which has been released since 10 Apr 2024:

https://arxiv.org/pdf/2401.10166v2
Thank you for your reply. VMamba now has three versions on arxiv. The one you see is the third version released on May 26, which is latter than the submission ddl. The one compared in table 1 is the first version released on January 18, whose details could be found in 2401.10166v1. We hope this reply can clarify your doubts. And we sincerely look forward to your feedback.
I'd like to thank the authors for providing the rebuttal. However, upon reading Table 1, VMamba Top-1 accuracy benchmarks seem to be  lower than the official release (and their paper):
https://github.com/MzeroMiko/VMamba

Any reason for this discrepancy ?