Title: QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model
Venue: NeurIPS 2024
Accepted: yes
Keywords: Visual Mamba

Visual Recognition

Vision Backbone
Abstract: Recent advancements in State Space Models, notably Mamba, have demonstrated superior performance over the dominant Transformer models, particularly in reducing the computational complexity from quadratic to linear. Yet, difficulties in adapting Mamba from language to vision tasks arise due to the distinct characteristics of visual data, such as the spatial locality and adjacency within images and large variations in information granularity across visual tokens. Existing vision Mamba approaches either flatten tokens into sequences in a raster scan fashion, which breaks the local adjacency of images, or manually partition tokens into windows, which limits their long-range modeling and generalization capabilities. To address these limitations, we present a new vision Mamba model, coined QuadMamba, that effectively captures local dependencies of varying granularities via quadtree-based image partition and scan. Concretely, our lightweight quadtree-based scan module learns to preserve the 2D locality of spatial regions within learned window quadrants. The module estimates the locality score of each token from their features, before adaptively partitioning tokens into window quadrants. An omnidirectional window shifting scheme is also introduced to capture more intact and informative features across different local regions. To make the discretized quadtree partition end-to-end trainable, we further devise a sequence masking strategy based on Gumbel-Softmax and its straight-through gradient estimator. Extensive experiments demonstrate that QuadMamba achieves state-of-the-art performance in various vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Our code and models will be released.

QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model

Abstract

Recent advancements in State Space Models, notably Mamba, have demonstrated superior performance over the dominant Transformer models, particularly in reducing the computational complexity from quadratic to linear. Yet, difficulties in adapting Mamba from language to vision tasks arise due to the distinct characteristics of visual data, such as the spatial locality and adjacency within images and large variations in information granularity across visual tokens. Existing vision Mamba approaches either flatten tokens into sequences in a raster scan fashion, which breaks the local adjacency of images, or manually partition tokens into windows, which limits their long-range modeling and generalization capabilities. To address these limitations, we present a new vision Mamba model, coined QuadMamba, that effectively captures local dependencies of varying granularities via quadtree-based image partition and scan. Concretely, our lightweight quadtree-based scan module learns to preserve the 2D locality of spatial regions within learned window quadrants. The module estimates the locality score of each token from their features, before adaptively partitioning tokens into window quadrants. An omnidirectional window shifting scheme is also introduced to capture more intact and informative features across different local regions. To make the discretized quadtree partition end-to-end trainable, we further devise a sequence masking strategy based on Gumbel-Softmax and its straight-through gradient estimator. Extensive experiments demonstrate that QuadMamba achieves state-of-the-art performance in various vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation. The code is in https://github.com/VISION-SJTU/QuadMamba .

1 Introduction

The architecture of Structured State Space Models (SSMs) has gained significant popularity in recent times. SSMs offer a versatile approach to sequence modeling that balances computational efficiency with model flexibility. Inspired by the success of Mamba [13] in language tasks, there has been a rise in using SSMs for various vision tasks. These applications range from designing generic backbone models [80, 41, 26, 66, 49] to advancing fields such as image segmentation [51, 39, 65, 46] and synthesis [17]. These advancements highlight the adaptability and potential of Mamba in the visual domain.

Despite their appealing linear complexity in long sequence modeling, applying SSMs directly to vision tasks results in only marginal improvements over prevalent CNNs and vision Transformer

: Informative tokens

Spatial locality

far apart in 1D

sequence

Coarse-level locality

Learnable

Fine-level locality

Coarse-level locality

Learnable

(a) Naive flattening

(b) Fixed window

(c) Learnable window

QuadMamba

VMamba

(d) Effective Receptive Field

1.0

0.8

0.6

0.4

0.2

×

一

models. In this paper, we seek to expand the applicability of the Mamba model for computer vision. We observe that the differences between the language and visual domains can pose significant obstacles in adapting Mamba to the latter. The challenges come from two natural characteristics of image data: 1) Visual data has rigorous 2D spatial dependencies, which means flattening image patches into a sequence may destroy the high-level understanding. 2) Natural visual signals have heavy spatial redundancy-e.g., an irrelevant patch does not influence the representation of objects. To address these two issues, we develop a vision-specific scanning method to construct 1D token sequences for Vision Mamba. Typically, vision Mamba models need to transform 2D images into 1D sequences for processing. As illustrated in Fig. 1(a), the straightforward method, e.g., Vim [80], that flattens spatial data into 1D tokens directly disrupts the natural local 2D dependencies. LocalMamba improves local representation by partitioning the image into multiple windows, as shown in Fig. 1(b). Each window is scanned individually before conducting a traversal across windows, ensuring that tokens within the same 2D semantic region are processed closely together. However, the handcrafted window partition lacks the flexibility to handle various object scales and is unable to ignore the less informative regions.

In this work, we introduce a novel Mamba architecture that learns to improve local representation by focusing on more informative regions for locality-aware sequence modeling. As shown in Fig. 1(c), the gist of QuadMamba lies in the learnable window partition that adaptively learns to model local dependencies in a coarse-to-fine manner. We propose to employ a lightweight prediction module to multiple layers in the vision Mamba model, which evaluates the local adjacency of each spatial token. The quadrant with the highest score is further partitioned into sub-quadrants in a recursive fashion for fine-grained scan, while others, likely comprising less informative tokens, are kept in a coarse granularity. This process results in window quadrants of varying granularities partitioned from the 2D image feature.

It is noteworthy that direct sampling from the 2D windowed image feature based on the index is non-differentiable, which renders the learning of window selection intractable. To handle this, we adopt Gumbel-Softmax to generate a sequence mask from the partition score maps. We then employ fully differentiable operators, i.e., Hadamard product and element-wise summation, to construct the 1D token sequences from the sequence mask and local windows. These lead to an end-to-end trainable pipeline with negligible computational overhead. For the informative tokens that cross two adjacent quarter windows, we apply an omnidirectional shifting scheme in successive blocks. Shifting 2D image features in two directions allows the quarter-window partition to be flexible in modeling objects appearing in arbitrary locations.

Extensive experiments on ImageNet-1k and COCO2017 demonstrate that QuadMamba excels at image classification, object detection, and segmentation tasks, with considerable advantages over existing CNN, Tranformer, and Mamba models. For instance, QuadMamba achieves a Top-1 accuracy of 78.2% on ImageNet-1k with a similar model size as PVT-Tiny (75.1%) and LocalViM (76.2%).

2 Related Work

2.1 Generic Vision Backbones

Convolutional Neural Networks (CNNs) [10, 30, 31] and Vision Transformer (ViT) [7] are two categories of dominant backbone networks in computer vision. They have proven successful as a generic vision backbone across a broad range of computer vision tasks, including but not limited to image classification [29, 53, 55, 20, 23, 21, 74, 4, 12], segmentation [44, 19], object detection [36, 79], video understanding [28, 76], and generation [11]. In contrast to the constrained receptive field in CNNs, vision Transformers [7, 42, 61], borrowed from the language tasks [59], are superior in global context modelling. Later, numerous vision-specific modifications are proposed to adapt the Transformer better to the visual domain, such as the introduction of hierarchical features [42, 61], optimized training [58], and integration of CNN elements [5, 54]. Thus, vision Transformers demonstrate leading performance on various vision applications [63, 1, 8, 34, 48]. This, however, comes at a cost of attention operations' quadratic time and memory complexity. which hinders their scalability despite remedies proposed [42, 61, 72, 57].

More recently, State Space Models (SSMs) emerged as a powerful paradigm for modeling sequential data in language tasks. Advanced SSM models [13] have reported even superior performance compared to state-of-the-art ViT architectures while having a linear complexity. Their initial success on vision tasks [80, 41] and, more importantly, remarkable computational efficiency hint at the potential of SSM as a promising general-purpose backbone alternative to CNNs and Transformers.

2.2 State Space Models

State Space Models (SSMs) [16, 15, 18, 35] are a family of fully recurrent architectures for sequence modeling. Recent advancements [15, 14, 47, 13] have gained SSMs Transformer-level performance, yet with its complexity scaling linearly. As a major milestone, Mamba [13] revamped the conventional SSM with input-dependent parameterization and scalable, hardware-optmized computation, performing on par with or better than advanced Transformer models on different tasks involving sequential 1D data.

Following the success of Mamba, ViM [80] and VMamba [41] reframe Mamba's 1D scan into bidirectional and four-directional 2D cross-scan for processing images. Subsequently, SSMs have been quickly applied to vision tasks (semantic segmentation [51, 65, 46], object detection [26, 3], image restoration [17, 52], image generation [9], etc.) and to data of other modalities (e.g., videos [67, 32], point clouds [40, 73], graphs [2], and cross-modality learning [60, 6]).

A fundamental consideration in adapting Mamba to non-1D data concerns the design of a path that scans through and maps all image patches into a SSM-friendly 1D sequence. Along this direction, preliminary efforts include the bi-directional zigzag-style scan in ViM [80], the 4-direction cross-scan in VMamba [41], and the serpentine scan in PlainMamba [66] and ZigMa [22], all conducted in the spatial domain spanned by the height and width axes. Other works [52, 75, 33] extend the scanning to an additional channel [52, 33] or temporal [75, 33] dimension. Yet, by naively traversing the patches, these scan strategies overlooked the importance of spatial locality preservation. This inherent weakness has been partially mitigated by LocalMamba [26], which partitions patches into windows and performs traversal within each window.

However, due to a monolithic locality granularity throughout the entire image domain, as controlled by the arbitrary window size, it is hard to decide on an optimal granularity. LocalMamba opts for DARTS [38] to differentially search for the optimal window size alongside the optimal scan direction for each layer, which adds to the complexity of the method. On the other hand, all existing methods involve hard-coded scan strategies, which can be suboptimal. Unlike all these methods, this paper introduces a learnable quadtree structure to scan image patches with varying locality granularity.

3 Preliminaries

State Space Models (S4) . SSMs [16, 15, 18, 35] are in essence linear time-invariant systems that map a one-dimensional input sequence x ( t ) ∈ R L to output response sequence y ( t ) ∈ R L recurrently through hidden state h ( t ) ∈ R N (with sequence length L and state size N ). Mathematically, such

Images

QuadVSS

Block

VSS Block

VSS Block

Stage 1

Stage 2

Stage 3

Stage 4

ൈʹ

ൈʹ

ൈ͸

ൈʹ

ு

ସ

ସ

ൈGLYPH<c=6,font=/BGPNLE+CambriaMath>

ு

଼

଼

ൈʹGLYPH<c=6,font=/BGPNLE+CambriaMath>

ு

ଵ଺

ଵ଺

ൈͶGLYPH<c=6,font=/BGPNLE+CambriaMath>

ு

ଷଶ

ଷଶ

ൈͺGLYPH<c=6,font=/BGPNLE+CambriaMath>

Mamba layer

Norm

FFN

Norm

Quad-Scan

QuadVSS

Block

(b) QuadVSS Block

(a) Overall architecture

Shift

systems can be formulated as the following Ordinary Differential Equations (ODEs):

where matrix A ∈ R N × N contains the evolution parameters; B ∈ R N × 1 and C ∈ R 1 × N are projection matrices.

The continuous-time ODEs in Eqn. 1 are, however, difficult to solve using deep learning. In practice, Eqn. 1 is usually transformed into a discretized form [16] via the zero-order hold (ZOH) rule, where the value of x is held constant over a sample interval ∆ [18]. The discretized ODEs are given by:

where A and B are the discrete version of A and B : A = exp (∆ A ) , and B = (∆ A ) -1 ( A -I ) · ∆ B . For efficient implementation, the iterative computations in Eqn. 2 can be performed in parallel with a global convolution operation:

where /circleasterisk denotes the convolution operator and K ∈ R L the SSM kernel.

Selective State Space Models (S6) . Traditional SSMs have input-agnostic parameters. To improve this, Selective State Space Models [13], also referred to as 'S6' or 'Mamba', are proposed with input-dependent parameters, such that A , B , and ∆ become learnable. To make up for parallelism difficulties, hardware-aware optimization is also performed. In this work, we particularly investigate the effective adaptation of the Mamba architecture for vision tasks. Early works such as ViM [80] and VMamba [41] have explored intuitive adaptations by transforming 2D images to 1D sequences in a raster scan fashion. We argue that simple raster scan is not an optimal design as it breaks the local adjacency of images. In our work, a novel learnable scanning scheme based on quadtree is proposed.

4 Method

4.1 General Architecture

QuadMamba shares a similar multi-scale backbone design to many CNNs [20, 64, 23] and vision Transformers [61, 71, 42, 25]. As shown in Fig. 2, an image I ∈ R H im × W im × 3 is first partitioned into patches of size 4 × 4 , resulting in N = H × W = /floorleft H im 4 /floorright ×/floorleft W im 4 /floorright visual tokens. A linear layer maps these visual tokens to hidden embeddings with dimension d , which are subsequently fed into our proposed Quadtree-based Visual State Space (QuadVSS) blocks. Unlike the Mamba structure used in language modeling [13], the QuadVSS blocks follow the popular structure of the

Quad window

partition

(

ு

ଶ

,

ௐ

ଶ

)

Quad window

partition

(

ு

ସ

,

ௐ

ସ

)

Prediction

Softmax

Windowed flatten

(

ு

ଶ

,

ௐ

ଶ

)

Windowed flatten

(

ு

ସ

,

ௐ

ସ

)

Up-sampling

& flatten

1- Mask

Mask

濇

Reverse &

reshape

Coarse-level

sequence

Fine-level

sequence

Quad-based sequence

濇

: Hadamard product

: Element-wise summation

Transformer block [7, 68], as illustrated in Fig 2(b). QuadMamba consists of a cascade of QuadVSS blocks organized in four stages, with stage i ( i ∈ { 1 , 2 , 3 , 4 } ) having S i QuadVSS blocks. In each stage, a downsampling layer halves the spatial size of feature maps while doubling their channel dimension. Thanks to the linear complexity of Mamba, we are free to stack more QuadVSS blocks within the first two stages, which enables their local feature preserving and modeling capabilities to be fully exploited with minimal computational overheads introduced.

4.2 Quadtree-based Visual State Space Block

As shown in Fig 2, our QuadVSS block adopts the meta-architecture [68] in vision Transformer, formulated by a token operator, a feedforward network (FFN), and two residual connections. The token operator consists of a shift module, a partition map predictor, a quadtree-based scanner, and a Mamba Layer. Inside the token operator, a lightweight prediction module first predicts a partition map over feature tokens. The quadtree-based strategy then partitions the 2D image space by recursively subdividing it into four quadrants or windows. According to the scores of the partition map at the coarse level, fine-level sub-windows within less informative coarse-level windows are skipped. Thus, a multi-scale, multi-granularity 1D token sequence is constructed, capturing more locality in more informative regions while retaining global context modeling elsewhere. The key components of the QuadVSS block are detailed as follows:

Partition map prediction. The image feature x ∈ R H × W × C , containing a total of N = HW embedding tokens, is first projected into score embeddings x s :

where φ s is a lightweight projector with a Norm-Linear-GELU layer. To better assess each token's locality, we leverage both the local embedding and the context information within each quadrant. Specifically, we first split x s in the channel dimension to obtain the local feature x local s and the context feature x global :

s

Then, we obtain 2 × 2 context vectors through an adaptive pooling layer and broadcast each context vector v local into the local embedding along the channel dimension:

s

where Interpolate( · ) is the bilinear interpolation operator that upsamples the context vector to a spatial size of H × W . Thus, we obtain the aggregated score embedding x agg s and feed it to a Linear-GELU layer φ p for partition score prediction:

where s ( i, j ) denotes the partition score of the token at spatial coordinate ( i, j ) .

Quadtree-based window partition. With each feature token's partition score s ( i, j ) predicted, we apply a quick quadtree-based window partition strategy with negligible computation cost. We construct bi-level window quadrants to capture spatial locality from coarse to fine. The quadtree-based strategy partitions the image feature into N 1 = 4 sub-windows at the coarse level and N 2 = 16 subwindows at the fine level. Different from the Transformer scheme, which maintains the spatial shape of image features with arbitrary Key/Value features simply by setting the Query feature as the same size, the token sequence generated by the learned scan for QuadMamba should be merged into the feature with the original spatial size. Thus, we select the top K = 1 quadrant with the highest averaged local adjacency score at the coarse level and further partition it into four sub-windows:

H

W

Shift step  =

௅

ଶ

h

ܮ

Shifted QuadVSS block  i

H

W

Shift step =

௅

ଶ

ܮ

Shifted QuadVSS block  i+2

/negationslash where coarse-level windows w i N 1 and fine-level sub-windows w j N 1 have spatial sizes of { H 2 , W 2 } and { H 4 , W 4 } , respectively. Afterward, we construct a 1D token sequence from the coarse-level windows { w i N 1 | i = k } and fine-level sub-windows { w (k , j) N 2 | j = 0 , 1 , 2 , 3 } based on their relative spatial indices.

Differentiable 1D sequence construction. Although our target is to conduct adaptive learned window quadrant partition, it is non-trivial to construct the 1D token sequence from the 2D spatial windowed features. Directly sampling from the 2D feature according to the learned index for each sequence token is non-differentiable, which impedes end-to-end training. To overcome this, we apply a sequence masking strategy to formulate the token sequence, implemented by the Gumbel-Softmax technique [27]:

where value '1' in M N 1 represents the selected coarse-level windows, which will be divided into subwindows further; '0' denotes those non-selected. A differentiable operator, Gumbel-Softmax effectively gets around the non-differential index selection in Eqn. 8. Then, we first obtain two token sequences in the coarse and fine levels separately:

where QuadWindowArrange ( ·| (h , w)) flattens the 2D feature into a 1D sequence based on the fixed size (h , w) window partition order. Restore ( · ) is to convert the 1D sequence back to the quadtreewindow partitioned 2D feature. Next, we upsample the mask M N 1 to obtain the full sequence mask M ∈ R H × W × 1 . We then use Hadamard product /circledot and element-wise summation ⊕ to construct the 1D token sequence with negligible computation cost:

where sequence L contains tokens in both coarse- and fine-level windows and is sent to the SS2D block for sequence modeling.

Omnidirectional window shifting. Considering the case that the most informative tokens are crossing two adjacent window quadrants, we borrow the idea of a shifted window scheme in Swin Transformer [42]. The difference is that the Transformer ignores the spatial locality for each token inside the window, while the token sequence inside the window in Mamba is still directional. Thus, we add additional shifted directions in the subsequent VSS blocks as shown in Fig 4, compared to only one direction shifting in Swin Transformer.

4.3 Model Configuration

It is noteworthy that QuadMamba's model capacity can be customized by tweaking the input feature dimension d and the number of (Q)VSS layers { S i } 4 i =1 . In this work, we build four variants of the QuadMamba architecture, QuadMamba-Li/T/S/B, with varying capacities:

Lite - block: { 2 , 2 , 2 , 2 } , QuadVSS stages: { 1 , 2 } , #Params: 5.4M, FLOPs: 0.82G.

Small - block: { 2 , 2 , 5 , 2 } , QuadVSS stages: { 1 , 2 } , #Params: 31.2M, FLOPs: 5.5G.

Tiny - block: { 2 , 6 , 2 , 2 } , QuadVSS stages: { 1 , 2 } , #Params: 10.3M, FLOPs: 2.0G.

Base - block: { 2 , 2 , 15 , 2 } , QuadVSS stages: { 1 , 2 } , #Params: 50.6M, FLOPs: 9.3G.

In all these variants, QuadVSS blocks are placed in the specified QuadVSS model stages to bring into full play their locality preservation capabilities on higher-resolution features, Omnidirectional shifting layers are applied in every other QuadVSS blocks. More details are found in the Appendix.

5 Experiment

We conduct experiments on commonly used benchmarks, including ImageNet-1k [29] for image classification, MS COCO2017 [37] for object detection and instance segmentation, and ADE20K [78] for semantic segmentation. Our implementations follow prior works [42, 80, 41]. Detailed descriptions of the datasets and configurations are found in the Appendix. In what follows, we compare the proposed QuadMamba with mainstream vision backbones and conduct extensive ablation studies to back the motivation behind QuadMamba's designs.

5.1 Image Classification on ImageNet-1k

Tab. 1 demonstrates the superiority of QuadMamba in terms of accuracy and efficiency. Specifically, QuadMamba-S surpasses RegNetT-8G [50] by 2.5%, DeiT-S [58] by 2.6%, and Swin-T [42] by 1.1% Top-1 accuracy, with comparable or reduced FLOPs. This advantage holds when comparing other model variants of similar numbers of parameters or FLOPs. Compared to other Mamba-based vision backbones, QuadMamba also yields favorable performance under comparable network complexity. For instance, QuadMamba-B ( 83 . 8% ) performs on par with or better than VMamba-B ( 83 . 7% ), LocalVim-S ( 81 . 2% ), and PlainMamba-L3 ( 82 . 3% ), while having significantly less parameters and FLOPs. These results manifest the performance and complexity superiority of QuadMamba and its potential as a powerful yet highly efficient vision backbone. Furthermore, QuadMamba achieves similar or higher performance than LocalMamba while being completely free of the latter's expensive architecture and scan policy search, which makes it a more practical and versatile choice of vision backbone.

5.2 Object Detection and Instance Segmentation on COCO

On object detection and instance segmentation tasks, QuadMamba stands out as a highly efficient backbone among models and architectures of similar complexity, as measured by number of networks parameters and FLOPs. QuadMamba finds very few competitors under the category of tiny backbones with less than or around 30M parameters. As shown in Tab. 2, in addition to dramatically outperforming ResNet18 [20] and PVT-T [61], QuadMamba-T also leads EfficientVMamba-S [49] by considerable margins of 3 . 0% mAP on object detection and 2 . 1% on instance segmentation. Among larger backbones, QuadMamba once again surpasses all ConvNet-, Transformer-, and Mamba-based

rivals. Notably, QuadMamba-S outperforms EfficientVMamba-B, a Mamba-based backbone characterised by its higher efficiency, by 3 . 0% mAP on object detection and 2 . 2% on instance segmentation, using comparable parameters. Furthermore, QuadMamba-S is able to keep up with and even surpass the performance of LocalVMamba-T [26] while circumventing a whole load of architecture and scan search hassles not reflected by Tab. 2's complexity measurements. These results show how QuadMamba can serve as a strong and versatile vision backbone with a pragmatic trade-off among computational complexity, design costs, and performance.

5.3 Semantic Segmentation on ADE20K

As shown in Tab. 3, under comparable network complexity and efficiency, QuadMamba achieves significantly higher segmentation precision than ConvNet-based ResNet-50/101 [20] and ConvNeXt [43], Transformer-based DeiT [58] and Swin Transformer [42], as well as most Mamba-based architectures [80, 49, 66]. For instance, QuadMamba-S with 47 . 2% mIoU reports superior segmentation precision to Vim-S ( 44 . 9% ), LocalVim-S ( 46 . 4% ), EfficientVMamba-B ( 46 . 5% ), and PlainMambaL2 ( 46 . 8% ), and competitive results to VMamba-T ( 47 . 3% ). Compared with LocalMamba-S/B, QuadMamba-S/B trails by small margins, yet enjoying the advantage of not incurring extra network searching costs. It is worth noting that the LocalMamba is designed by the Neural Architecture Search (NAS) technology, which is data-dependent and lacks flexibility with other data modalities and data sources.

5.4 Ablation Studies

We conduct ablation studies to validate QuadMamba's design choices from various perspectives. QuadMamba-T is used for all experiments unless otherwise stated.

Effect of locality in Mamba. We factorise the effect of coarse- and fine-grain locality modeling in building 1D token sequences on model performance. Specifically, we compare the naive window-free flattening strategy in [80, 41] that overlooks 2D locality against window partitions in three scales (i.e., 28 × 28 , 14 × 14 , 2 × 2 ) that represent three granularity levels of feature locality. In practice, we replace QuadVSS blocks of a QuadMamba-T model with the plain VSS blocks in [41]. To exclude the negative effects of padding operations, we only partition the features with a spatial size of 56 × 56 in the first model stage. As illustrated in Tab. 4, the naive scan strategy leads to significantly degraded object detection and instance segmentation performance compared to when windowed scan is adopted. The scale of local windows is also shown to considerably influence the model performance, which suggests that too large or too small a window given the image resolution can be suboptimal.

Quadtree-based partition resolutions. We examine the choice of partition resolutions in the bi-level quadtree-based partition strategy. The configured resolutions in Tab. 5 are applied in the first two

×

LP1

LP3

73.5

73.7

73.8

74

73

73.5

74

74.5

:S-QuadVSS

:QuadVSS

:VSS

Stage1

Stage2

Stage3

Stage4

LP1

LP3

+ Comp. Direction

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

Layer 6

Layer 7

Layer 8

Score map

model stages with feature resolutions of { 56 × 56 , 28 × 28 , 14 × 14 } . Experimentally we deduce the optimal resolutions to be { 1 / 2 , 1 / 4 } for the coarse- and fine-level window partition, respectively. This handcrafted configuration may be replaced by more flexible and learnable ones in future work.

Layer patterns in the hierarchical model stage. We investigate different design choices of the layer pattern within our hierarchical model pipeline. From Fig. 5, layer pattern LP2 outperforms LP1 with less QuadVSS blocks by 0 . 2% accuracy. This is potentially due to the effect of locality modeling being more pronounced in shallower stages than in deeper stages as well as the adverse influence of padding operations in stage 3. LP3, which places the QuadVSS blocks in the first two stages and in an interleaved manner, achieves the best performance, and is adopted as our model design.

Necessity of multi-directional window shift. Different from the unidirectional shift in Swin Transformer [42], Fig. 5 shows a 0 . 2% gain in accuracy as complementary shifting directions are added. This is expected since attention in Transformer is non-causal, whereas 1D sequences in Mamba, being causal in nature, are highly sensitive to relative position. A multi-directional shifting operation is also imperative for handling cases where the informative region spans across adjacent windows. Fig. 6 further visualizes the shifts in the learned fine-grained quadrants across hierarchical stages, which adaptively attend to different spatial details at different layers.

Numbers of (Quad)VSS blocks per stage. We conduct experiments to evaluate the impact of different numbers of (Quad)VSS blocks in each stage. Tab. 6 presents four configurations, following design rule LP3 in Fig. 5, with a fixed channel dimension of 96. We find that a bulky 2nd or 4th stage leads to diminished performance compared to a heavy 3rd stage design, whereas dealing out the (Quad)VSS blocks more evenly between stages 2 and 3 yields comparable if not better performance with favorable complexities. This evidence can serve as a rule of thumb for model design in future work, especially as the model scales up.

6 Conclusion

In this paper, we propose QuadMamba, a vision Mamba architecture that serves as a versatile and efficient backbone for visual tasks, such as image classification and dense predictions. QuadMamba effectively captures local dependencies of different granularities by learnable quadtree-based scanning, which adaptively preserves the inherent locality within image data with negligible computational overheads. The QuadMamba's effectiveness has been proven through extensive experiments and ablation studies, outperforming popular CNNs and vision transformers. However, one limitation of QuadMamba is that window partition with more than two levels is yet to be explored, which may be particularly relevant for handling dense prediction visual tasks and higher-resolution data, such as remote sensing images. The fine-grained partition regions are rigid and lack flexibility in attending to regions of arbitrary shapes and sizes, which is left for our future work. We hope that our approach will motivate further research in applying Mamba to more diverse and complex visual tasks.

Acknowledgments. This work was supported by NSFC (62322113, 62376156), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), and the Fundamental Research Funds for the Central Universities.

References

Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Luˇ ci´ c, and Cordelia Schmid. Vivit: A video vision transformer. In CVPR , 2021.

Ali Behrouz and Farnoosh Hashemi. Graph mamba: Towards learning on graphs with state space model. arXiv preprint arXiv:2402.08678 , 2024.

Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint arXiv:2403.19888 , 2024.

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In CVPR , 2017.

Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In Neurips , 2021.

Wenhao Dong, Haodong Zhu, Shaohui Lin, Xiaoyan Luo, Yunhang Shen, Xuhui Liu, Juan Zhang, Guodong Guo, and Baochang Zhang. Fusion-mamba for cross-modality object detection, 2024.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2020.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR , 2021.

Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Scalable diffusion models with state space backbone, 2024.

Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position . 1980.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In CVPR , 2014.

Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR , 2018.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023.

Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Neurips , 2022.

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021.

Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Neurips , 2021.

Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: A simple baseline for image restoration with state-space model. arXiv preprint arXiv:2402.15648 , 2024.

Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. In Neurips , 2022.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Björn Ommer. Zigma: A dit-style zigzag mamba diffusion model, 2024.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR , 2017.

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV , 2016.

Tao Huang, Lang Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Lightvit: Towards light-weight convolution-free vision transformers. arXiv preprint arXiv:2207.05557 , 2022.

Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. In WACV , 2024.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR , 2017.

Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR , 2014.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. 2012.

Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks , 3361(10):1995, 1995.

Yann LeCun, Lottou Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 1998.

Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Limin Wang, Yali Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In ECCV , 2024.

Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprint arXiv:2402.05892 , 2024.

Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV , 2022.

Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolutional models great on long sequence modeling? arXiv preprint arXiv:2210.09298 , 2022.

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR , 2017.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR , 2019.

Jiarun Liu, Hao Yang, Hong-Yu Zhou, Yan Xi, Lequan Yu, Yizhou Yu, Yong Liang, Guangming Shi, Shaoting Zhang, Hairong Zheng, et al. Swin-umamba: Mamba-based unet with imagenetbased pretraining. arXiv preprint arXiv:2402.03302 , 2024.

Jiuming Liu, Ruiji Yu, Yian Wang, Yu Zheng, Tianchen Deng, Weicai Ye, and Hesheng Wang. Point mamba: A novel point cloud backbone based on state space model with octree-based ordering strategy. arXiv preprint arXiv:2403.06467 , 2024.

Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. In Neurips , 2024.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In CVPR , 2021.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. 2022.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , 2015.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2019.

Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722 , 2024.

Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. 2022.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In CVPR , 2023.

Xiaohuan Pei, Tao Huang, and Chang Xu. Efficientvmamba: Atrous selective scan for light weight visual mamba. arXiv preprint arXiv:2403.09977 , 2024.

Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR , 2020.

Jiacheng Ruan and Suncheng Xiang. Vm-unet: Vision mamba unet for medical image segmentation. arXiv preprint arXiv:2402.02491 , 2024.

Yuan Shi, Bin Xia, Xiaoyu Jin, Xing Wang, Tianyu Zhao, Xin Xia, Xuefeng Xiao, and Wenming Yang. Vmambair: Visual state space model for image restoration. arXiv preprint arXiv:2402.15648 , 2024.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition, 2021.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR , 2015.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR , 2016.

Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In ICLR , 2022.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neurips , 2017.

Zifu Wan, Yuhao Wang, Silong Yong, Pingping Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Sigma: Siamese mamba network for multi-modal semantic segmentation, 2024.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In CVPR , 2021.

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV , 2018.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers, 2021.

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR , 2017.

Zhaohu Xing, Tian Ye, Yijun Yang, Guang Liu, and Lei Zhu. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint arXiv:2401.13560 , 2024.

Chenhongyi Yang, Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot J. Crowley. Plainmamba: Improving non-hierarchical mamba in visual recognition. arXiv preprint arXiv:2403.17695 , 2024.

Yijun Yang, Zhaohu Xing, Chunwang Huang, and Lei Zhu. Vivim: a video vision mamba for medical video object segmentation, 2024.

Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR , 2022.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV , 2019.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In CVPR , 2021.

Qinglong Zhang and Yu-Bin Yang. Rest: An efficient transformer for visual recognition. In Neurips , 2021.

Tao Zhang, Xiangtai Li, Haobo Yuan, Shunping Ji, and Shuicheng Yan. Point cloud mamba: Point cloud learning via state space model. arXiv preprint arXiv:2403.00762 , 2024.

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR , 2018.

Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective ssm. arXiv preprint arXiv:2403.07487 , 2024.

Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In ICCV , 2017.

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI , 2020.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR , 2017.

Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR , 2018.

Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417 , 2024.

A Appendix

In the supplementary materials, we provide more details and analysis of QuadMamba in Sec. A.1, implementation details in Sec. A.2, and more information on QuadMamba's model variants in Sec. A.3. In Sec.A.4, we also provide the pseudo-code to help understand the key operations within the QuadVSS block. Next, we present the complexity analysis and the throughput measurement of our proposed QuadMamba variants in Sec.A.5. Finally, we provide more visualization results of QuadMamba in Sec.A.6.

A.1 Schematic Illustration

To better assess the impact of the Mamba model and our proposed learnable scanning method, we integrated our key token operator into the meta-architecture of the Vision Transformer, as illustrated in Fig. 7.

Norm

FFN

Norm

(b)

QuadVSS Block

SS2D

Linear

Linear

Linear

ߪ

Conv

SS2D

ߪ

(a)

Vision Mamba Block

(Gated Network)

Relationship to RNNs . The concept of recurrence in a hidden state is closely linked to Recurrent Neural Networks (RNNs) and State Space Models (SSMs), as suggested by [13]. Some RNN variants employ forms of gated RNNs and remove the time-wise nonlinearities. These can be considered as a combination of the gating mechanisms and selection mechanisms.

A.2 Implementation Details

Following the VMamba [41], we conducted a benchmark to assess the image classification performance of QuadMamba on the ImageNet-1k [29] dataset. ImageNet [29] is widely recognized as the standard for image classification benchmarks, consisting of around 1.3 million training images and 50,000 validation images spread across 1,000 classes.

The training scheme is based on DeiT [58]. The data augmentation techniques used include random resized crop (input image size of 2242), horizontal flip, RandAugment [77], Mixup [70], CutMix [69], Random Erasing [77], and color jitter. Additionally, regularization techniques such as weight decay, stochastic depth [24], and label smoothing [56] are applied. All models are trained using AdamW [45]. The learning rate scaling rule is calculated as BatchSize 1024 × 10 -3 . Our models are implemented with PyTorch and Timm libraries and trained on A800 GPUs.

For object detection and instance segmentation, we assess QuadMamba using the MS COCO2017 dataset [37]. The MS COCO2017 dataset is widely used for object detection and instance segmentation and consists of 118,000 training images, 5,000 validation images, and 20,000 test images of common objects.

Name

Output

Lite

Small

Base

Large

stem

56

×

56

patch_embed: Conv-3_C/2_2, Conv-3_C/2_1, Conv-3_C_2

stage1

56

×

56

[

QuadVSS_1_48

MLP_48

]

×

2

[

QuadVSS_1_64

MLP_64

]

×

2

[

QuadVSS_2_96

MLP_96

]

×

2

[

QuadVSS_2_96

MLP_96

]

×

2

stage2

28

×

28

patch_embed: Conv-3_2C_2

[

QuadVSS_1_96

MLP_96

]

×

2

[

QuadVSS_1_128

MLP_128

]

×

6

[

QuadVSS_2_192

MLP_192

]

×

2

[

QuadVSS_2_192

MLP_192

]

×

2

stage3

14

×

14

patch_embed: Conv-3_4C_2

[

VSS_1_192

MLP_192

]

×

2

[

VSS_1_256

MLP_256

]

×

5

[

VSS_2_384

MLP_384

]

×

5

[

VSS_2_384

MLP_384

]

×

15

stage4

7

×

7

patch_embed: Conv-3_8C_2

[

VSS_1_384

MLP_384

]

×

2

[

VSS_1_512

MLP_512

]

×

2

[

VSS_2_768

MLP_768

]

×

2

[

VSS_2_768

MLP_768

]

×

2

Classifier

1

×

1

average pool, 1000d fully-connected

GFLOPs

0.82G

2.07G

5.51G

9.30G

Params

5.47M

10.32M

31.25M

50.6M

For semantic segmentation, we evaluate QuadMamba on the ADE20K [78] dataset. ADE20K is a popular benchmark for semantic segmentation with 150 categories. It consists of 20,000 training images and 2,000 validation images. Following Swin Transformer [42], we construct UperHead [62] with the pretrained model. The learning rate is set as 6 × 10 -5 . The fine-tuning process consists of a total of 160,000 iterations with a batch size of 16. The default input resolution is 512x512, and the experimental results are using 640x640 inputs and multi-scale (MS) testing.

A.3 Model Variants

We present the detailed architectural configurations of QuadMamba variants in Tab. 7. Each building unit of the model variants and the corresponding hyper-parameters are illustrated in detail. We mostly place the proposed QuadVSS block in the first two stages and place the plain VSS block from [41] in the latter two stages. The reason is that the features in the shallow stage have larger spatial resolution and need more locality modeling.

A.4 Key Operators in QuadVSS Block

In this section, we present a detailed illustration of the QuadVSS block. The overall structure of the QuadVSS block is presented in Alg. 1. The process to partition feature tokens via the bi-level quadtree-based strategy is outlined in Alg. 2, and that to restore them back into a 2D spatial feature is described in Alg. 3. Moreover, in Alg. 4, we provide the pseudo-code of the proposed sequence masking strategy to construct a 1D token sequence in a differential manner. In Fig. 8, we illustrate the detailed pipeline of the ominidirectional shifting scheme in two QuadVSS blocks. Moreover, in

H

W

Shift step =

௅

ଶ

S

ܮ

Shifted QuadVSS block  i

H

W

Shift step =

௅

ଶ

S

ܮ

Shifted QuadVSS block  i+2

reverse

reverse

Fig. 9, we provide the details of three different quadtree-based partition resolution configurations at coarse and fine levels, which are mentioned in Tab. 5.

A.5 Complexity & Throughput Analysis

Our learnable scanning method does not alter the total sequence length L = H × W . Thus, the QuadVSS block retains the linear computational complexity of O ( L ) of mainstream vision Mamba architectures [80, 41, 26, 66]. Other additional model costs are negligible. In Tab. 9, we show the throughput of the proposed QuadMamba variants, which are measured in the V800 GPU platform.

The computation complexity of a standard Transformer is as follows: Assuming its input X ∈ R N × D has a total input token number of N = H × W and channel dimensions of D , the FLOPs for the transformer attention can be calculated as:

It shows a quadratic complexity with the input size of the transformer attention scheme, as compared to the linear complexity of QuadMamba.

A.6 Visualization Results

In this section, we provide more visualization results. In Fig. 10, we visualize the partition maps that focus on different regions from shallow to deep blocks, using QuadMamba-T as an example. In Fig. 11, we showcase more visualizations of the Effective Receptive Field (ERF) of CNN, Transformer, and Mamba variants. As can be observed, QuadMamba exhibits not only a global ERF but also a more locality-sensitive response compared to VMamba [41] with a naive flattening strategy.

Coarse level

Fine level

ሺ

ܪ

ǡ

ܹ

)

ሺ

ு

ଶ

ǡ ௐ

ଶ

)

ሺ

ு

ଶ

ǡ ௐ

ଶ

)

ሺ

ு

ସ

ǡ ௐ

ସ

)

(a)

(b)

ሺ

ு

ସ

ǡ ௐ

ସ

)

ሺ

ு

଼

ǡ ௐ

଼

)

(c)

Algorithm 1 PyTorch code of QuadVSS block

Algorithm 2 PyTorch code of Quadtree window partition at two levels

Algorithm 3 PyTorch code of Quadtree window restoration at two levels

Algorithm 4 PyTorch code of differentiable sequence masking

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

Layer 6

Layer 7

Layer 8

Score map

in Layer1

1

！

QuadMamba

VMamba

DeiT

ResNet

After

training

Before

training

1.0

0.8

0.6

0.4

0.2

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Sec. 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Sec. 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Sec.3 and Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Sec.4 and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See Abstract and Appendix.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Sec.5 and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Sec.5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Sec.5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: See Sec.5.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: No potential social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No risk for misusing data and models. All data and models are publicly available for academic usage.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See reference part.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new asset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing experiments and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No potential risks incurred by study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents QuadMamba, a novel Mamba architecture for visual tasks such as image classification and dense predictions. Unlike the classic Vision Mamba, which splits 2D visual data using fixed windows, the authors introduce learnable windows by using a lightweight module that predicts 2D locality more informatively, allowing for the ignoring of irrelevant windows and further splitting of the most informative windows into sub-windows, capturing more fine-grained information. This coarse-to-fine fashion is made possible by the new implementation of a splitting operator with Hadamard product and element-wise summation, making the pipeline fully differentiable. The authors experiment on the classic benchmarks for image classification (ImageNet), object detection (COCO), and semantic segmentation (ADE20K), showing that the method achieves state-of-the-art results.\n\nSTRENGTHS:\n- First, the paper is very well written and most of the figures are done very well so that it is easy to fully understand the story of paper.\n\n- I also like the idea coarse-to-fine scanning, to capture more relevant information in each layer in QuadMamba. It is also worth noting that as the authors mentioned, direct sampling from 2D visual data based on index is not differentiable, but the authors has proposed a solution to overcome it which I consider is a strong contribution.\n\n- The experiment shows that QuadMamba always outperforms the classic CNN, Vision Transformer on popular tasks (image classification, object detection, image segmentation).\n\nWEAKNESSES:\nI do not see any major weakness for this paper but only have a few questions for better understanding the paper and improve its clarify:\n\n- The lightweight module to predict 2D locality, informative windows, is it shared across layers or specific to each layer? It seems this module is shared across layers but I feel it needs to be different as each layer capture different information.\n\n- As the authors mentioned in limitation, in remote sensing images which might be helpful to have more than two levels partition in QuadMamba. However, I feel this experiment can still be experimented with the current architecture. In this case, what can happen is that the lightweight module will split into sub-quadrants for all windows ( since all of them are mostly capturing relevant for predicting the output). Did the authors observe this  extreme case to subplot all quadrants into sub-quadrants can happen with the current architecture?\n\nQUESTIONS:\nAs mentioned in weakness section, I have only two questions related to lightweight module, extreme case of splitting all quadrants to sub-quadrants in each layer. Besides, it would be great if the authors can provide some experiments with more than two levels partition as mentioned in conclusion, for example, by using the lite model on semantic segmentation tasks should be enough. I would be very helpful to get some insights for possible future works.\n\nLIMITATIONS:\nYes, the authors explicit mentioned the limitation which is that the partition with more than two levels is yet to be explored.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a vision mamba backbone for various vision tasks. It aims to adapt the recent popular mamba model originated from the language domain to vision tasks. The authors propose a learnable quad-tree partition strategy which can adaptively generate multi-scale visual sequences with spatial prior for mamba model. The authors also make contributions on differentiable training and shifted token interactions. Experimental results show the effectiveness of the proposed vision mamba, including image classification, object detection and segmentation. It shows the potential to be applied in various vision mamba models. Ablation studies are also extensively conducted.\n\nSTRENGTHS:\n- Strong Motivation. This paper is well-motivated and important. Effectively adapting the Mamba model to vision tasks has gained wide attention recently. Multi-grained image information and 2D priors are essential to building a vision backbone, which has led to a strong motivation for this paper. Treating the 2D image as token sequences in the language domain needs exploration and adaptation. \n\n- Novel \\& applicable method.  The quadtree-based window partition is novel in vision Mamba. The learnable module part is lightweight and conceptually simple. The quadtree search technique is seen in other tasks, such as the point cloud domain. This paper only learns to partition the semantic-rich content with a handcrafted partition strategy, which avoids developing a search algorithm that is too complex. The other two points, the differentiable trick and shifted scheme, are also interesting. It has the potential to be applied to constructing token sequences for other vision mamba models.  \n\n- Good experimental results. The method proposed is tested in image classification, detection, and segmentation tasks, which proves its effectiveness over other vision mamba models. The ablation studies and visualization results also give more explanations about the quadtree-based sequence construction. \n\n- Well-written presentation.  The writing is clear and precise. In general, the presentation of this work is easy to follow.\n\nThe experiments are well performed and the presented method compares favorably to the considered baselines.\n\nWEAKNESSES:\nI have a number of suggestions and questions that may help to further improve the paper:\n\n- It would be helpful to demonstrate the impact of the hyper-parameters in the model design. For example, is it helpful to partition more window levels?\n\n- More illustrations and analysis are needed for the learnable parameters. It can show the relationship between model cost and performance gain to other researchers/readers. \n\n- It would be useful to give the details about the training settings and compare them to other vision mamba works.\n\nQUESTIONS:\nI have listed some questions in the previous section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
Authors propose a technique to adapt Mamba to vision tasks. They propose a novel Quad-tree based approach instead of flattening tokens for images in a raster scan mechanism to avoid losing local dependencies. They evaluate on three vision tasks of object recognition, objet detection and instance & semantic segmentation.

STRENGTHS:
- Interesting novelty and contribution designed specifically for Mamba type of models adapted to vision tasks.
- Practically beneficial algorithm on the efficiency side where the show lower FLOPs and parameters with better or on par performance to SOA

WEAKNESSES:
- It provides a tradeoff though and it's not clear which is more favourable when compared to VMamba with corresponding T/S/B in Table 1. Meaning you can see when comparing VMamba-T (88.2) with QuadMamba-T (78.2) it is much lower, it is only at the base variant they become on-par but with reduced FLOPS/parameters on QuadMamba side. The fact it doesn't seem that on the lighter weight variants the performance is on par in fact it is much lower with around 10%.

- What’s the reported inference time for Table 1 in addition to the FLOPs.

QUESTIONS:
- Table 2 not clear what is the result for LocalVMamba and VMamba with S for fair comparison to QuadMammba-S

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces QuadMamba, an enhancement of vision State Space Model (SSM) architectures. At its core is a learnable QuadVSS network block that processes the image input patch at two different resolutions. For every 2x2 coarse window with 4 image patches, the method adaptively learns to process one of the 4 patches at the finer 2x resolution using the differentiable Gumbel-softmax formulation. These block are hierarchically stacked similar to SwinTransformer, but using a window shifting scheme in two different directions which is a better fit for SSM. \n\nThe method is demonstrated to obtain good results relative to comparably sized transformer and CNN architectures on Imagenet classification, COCO object detection / instance segmentation, and ADE20k semantic segmentation.\n\nSTRENGTHS:\n- Intuitive idea to learn how to pick an area that requires higher resolution processing and pack it into SSM via Gumbel Softmax. Solid although not particularly novel stacking of QuadVSS blocks into a SwinTransformer-like network architecture. \n- Competitive results relative to ViT and CNN, and seems to improve a little bit over other Mamba methods, although the gains there seem quite incremental. \n- Ablation over multiple network parameter decisions.\n\nWEAKNESSES:\n# Significance\nThe contribution seems a bit incremental. The overall idea of image traversals into several windows to enforce better locality was already explored before in LocalMamba. The gains in the experimental section over some of the other Mamba network variants (EfficientVMamba, VMamba, Swin-S etc) are not too large. The idea that we do not have to process all areas in high resolution but only 1/4 of them does seem useful, although does not seem strictly limited to SSMs -- e.g. would ViT methods also benefit? \n\n# Experimental results\n- It seems that the natural baseline to this method is to compare a network made of VSS modules, as opposed to QuadVSS (the main novelty). Such a comparison was done in Fig 5, however it is not too clear how rigorous it is. It would be helpful to compare more directly VSS / QuadVSS equivalents with same FLOPS, for several different FLOPS thresholds. This does not seem to have been done - it would validate more strongly the fact that allocating parameters selectively to higher resolutions actually helps (as opposed to using the baseline VSS module pyramid). \n- Unclear what the training overhead of a QuadVSS block is compared to a VSS block (in terms of memory, compute)? \n- As opposed to just a table, it would be helpful to have plots with flops vs quality as axes, with a separate curve in that graph for each model family. Such a plot can make the comparison more obvious, given that different methods have somewhat different amounts of flops. \n- In Sec 4.3, it's unclear why for the Tiny model, we stack more blocks in the second stage, while for Small and Base we stack more in the third stage. Any particular reason? Third stage is more standard VSS modules, as opposed to your QuadVSS innovation. Also later in Table 6 yet different stackings are best (2,4,6,2). It's hard to discern the logic in all these choices. \n\n# Clarity and questions\nSome details of the approach were not particularly clear to me. \n- L175 Do you always pick the same fine-grained patch for all 2x2 windows, or do they vary by window? Not particularly clear from the exposition/notation. \n- Also, it appears that 7 patches are picked for each region as per L185-187. Don't you want to have 8 patches (powers of 2 are usually more efficient hardware-wise?) \n- The softmax in Eq 7, what labels do you train it on? Is it the final labels? Right now it appears that this part is trained first, and kept fixed. \n- (minor) In Eq 6, it seems 'v_local' is aggregated across the whole image, shouldn't it be called 'global' instead? \n- (minor) L186: where is 'local adjacency score' defined? This is the first mention of the term. \n- (minor) Fig 5: S-QuadVSS is not defined, I assume it's the opposite direction shift, but helps to be explicit. \n\n# Minor Nits and Typos \n159: infOrmative\nTable 1: VMamaba\n606: Uperhead [62] (I believe it should be UperNet)\n\nQUESTIONS:\n- Is your quad idea limited to SSM or would it also work for Vit? Can one learn to downsample specific patches in a Vit? \n- Do you have comparisons between networks made of QuadVSS/VSS or only VSS for several same FLOPS budgets? Do you have comparison of training overhead (compute/memory) for networks made of QuadVSS/VSS or only VSS baseline? \n- L175 Do you always pick the same fine-grained patch for all 2x2 windows, or do they vary by window? Not particularly clear from the exposition/notation. \n- Also, it appears that 7 patches are picked for each region as per L185-187. Don't you want to have 8 patches (powers of 2 are usually more efficient hardware-wise?) \n- The softmax in Eq 7, what labels do you train it on? Is it the final labels? Right now it appears that this part is trained first, and kept fixed.\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}