QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model
Authors: Fei Xie
Weijia Zhang
Zhongdao Wang
Chao Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual Mamba
Visual Recognition
Vision Backbone
Primary area: deep_learning_architectures

Abstract: Recent advancements in State Space Models, notably Mamba, have demonstrated superior performance over the dominant Transformer models, particularly in reducing the computational complexity from quadratic to linear. Yet, difficulties in adapting Mamba from language to vision tasks arise due to the distinct characteristics of visual data, such as the spatial locality and adjacency within images and large variations in information granularity across visual tokens. Existing vision Mamba approaches either flatten tokens into sequences in a raster scan fashion, which breaks the local adjacency of images, or manually partition tokens into windows, which limits their long-range modeling and generalization capabilities. To address these limitations, we present a new vision Mamba model, coined QuadMamba, that effectively captures local dependencies of varying granularities via quadtree-based image partition and scan. Concretely, our lightweight quadtree-based scan module learns to preserve the 2D locality of spatial regions within learned window quadrants. The module estimates the locality score of each token from their features, before adaptively partitioning tokens into window quadrants. An omnidirectional window shifting scheme is also introduced to capture more intact and informative features across different local regions. To make the discretized quadtree partition end-to-end trainable, we further devise a sequence masking strategy based on Gumbel-Softmax and its straight-through gradient estimator. Extensive experiments demonstrate that QuadMamba achieves state-of-the-art performance in various vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Our code and models will be released.

QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model
Fei Xie Weijia Zhang'
Zhongdao Wang² Chao Ma
1 MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University
2 Huawei Noah's Ark Lab (jaffe031, we jiajia.s zhang, chaoma @ jsjtu. edu.cn
wangzhongdaahhawee.... com
Abstract
Recent advancements in State Space Models, notably Mamba, have demonstrated superior performance over the dominant Transformer models, particularly re- ducing the computational complexity from quadratic to linear. Yer, difficulties in adapting Mamba from language to vision tasks arise due to the distinct charae- teristics of visual data, such as the spatial locality and adjacency within images and large variations in information granularity across visual tokens. Existing vision Mamba approaches either flatten tokens into sequences in raster scan fashion, which breaks the local adjacency of images, or manually partition to- kens into windows, which limits their long-range modeling and generalization capabilities. To address these limitations, we present a new vision Mamba model, coined QuadMamba, that effectively captures local dependencies of varying granu- larities via quadtree- image partition and scan. Concretely, our lightweight quadtree-based
module learns to preserve the 2D locality of spatial regions within learned window quadrants. The module estimates the locality score each token from their features, before adaptively partitioning tokens into win- dow quadrants. An omnidirectional window shifting scheme is also introduced capture more intact and informative features across different local regions. To make the discretized quadtree partition end-to-end trainable, we further devise a sequence masking strategy based on Humbel-Softmat and its trraighttthrouge gradient estimator. Extensive experiments demonstrate that Maaba achieves state-of-the-art performance various vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation. The code is in https: / / thub. com com/ ///./VSSIONN Stt.///ITTTOOSSST///Aaa/a-.
Introduction
The architecture of Structured State Space Models (SSMs) has gained significant popularity recent times. SSMs offer versatile approach to sequence modeling that balances computational efficiency with model flexibility, Inspired by the success of Mamba [13] language tasks, there has been a rise in using SSMs for various vision tasks. These applications range from designing generic backbone models [80, 41, 26, 66, 49] to advancing fields such as image segmentation [51, 39, 65, 46) and synthesis [17]. These advancements highlight the adaptability and potential of Mamba in the visual domain.
Despite their appealing linear complexity in long sequence modeling, applying SSMs directly to vision tasks results only marginal improvements over prevalent CNNs and vision Transformer
CCorresponiing author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Informative sokens
p-g
Leamable
Leumo
spurt
32
sequence
Coorse level locality
Spetial locality
Sevel locality
Fine-level locality
VMamba
QuadiMamba
(3) flattening
Fixed window
Learnable window
(4) Effective Receptive
Figure 1: Illustaatoon of scan strategies for transforming 2D visual data into ID sequences. (a) naive raster scan [80, 41, 66] ignores the 2D locality; (b) fixed window scan [26] lacks the flexibility to handle visual signals of varying granularities; (c) our learnable window partition and scan strategy adaptively preserves the 2D locality with a focus on the more informative window quadrant; (d) the effective receptive field of our QuadMamba demonstrates more locality than the plain Vision Mamba.
models. In this paper. we seek to expand the applicability of the Mamba model for computer vision. We observe that the differences between the language and visual domains can pose significant obstacles in adapting Mamba to the latter. The challenges come from two natural characteristics of image data: 1) Visual data has rigorous 2D spatial dependencies, which means flattening image patches into a sequence may destroy the high-level understanding. 2) Natural visual signals have heavy spatial redundancy-ess e.g., an irrelevant patch does not influence the representation of objects. To address these two issues, we develop a vision-specific scanning method to construct ID token sequences for Vision Mamba. Typically, vision Mamba models need to transform 2D images into ID sequences for processing. As illustrated in Fig. (a), the straightforward method, e.g., Vim [80], that flattens spatial data into ID tokens directly disrupts the natural local 2D dependencies. LocalMamba improves local representation by partitioning the image into multiple windows, shown in Fig. ((b). Each window is scanned individually before conducting a traversal across windows, ensuring that tokens within the same 2D semantic region are processed closely together. However, the handcrafted window partition lacks the flexibility to handle various object scales and is unable to ignore the less informative regions.
In this work, we introduce novel Mamba architecture that learns to improve local representation by focusing on more informative regions for locality-aware sequence modeling. As shown in Fig. (cc), the gist of QuadMamba lies the learnable window partition that adaptively learns to model local dependencies in coarse-to to-fine manner. We propose employ lightweight prediction module to multiple layers in the vision Mamba model, which evaluates the local adjacency each spatial token. The quadrant with the highest score is further partitioned into sub-quadrants in recursive fashion for fine- rreigraieed scan, while others, likely comprising less informative tokens, are kept in a coarse granularity. This process results in window quadrants of varying granularities partitioned from the 2D image feature.
It is noteworthy that direct sampling from the 2D windowed image feature based the index is non-difierentiable, which renders the learning of window selection intractable. To handle this, we adopt Gumbel-Softmal fully differentiable operators, i.c., Hadamard product and element-wise summation, to construct the ID token sequences from the sequence mask and local windows. These lead to end-to-end trainable pipeline with negligible computational overhead. For the informative tokens that cross two adjacent quarter windows, Shifting 2D image features in two directions allows the quarter-wing window partition to be flexible in modeling objects appearing in arbitrary locations.
generate a sequence mask from the partition score maps. We then employ
apply an omnidirectional shifting scheme in successive blocks.
Extensive experiments on ImageNet-Ik 1k and COC02017 demonstrate that QuadMamba excels at image classification, object detection, and segmentation tasks, with considerable advantages over existing CNN, Tranformer, and Mamba models. For instance, QuadMamba achieves a Top-1 accuracy of 78.2% ImageNet-Il Ik with similar model size PVT-Tiny (75.1% (6) and LocalViM (76.2%).

---

2 Related Work
2.1 Generic Vision Backbones
Convolutional Neural Networks (CNNs) [10, 30, 31] and Vision Transformer (ViT) [7] are two categories of dominant backbone networks in computer vision. They have proven successful generic vision backbone across a broad range of computer vision tasks, including but not limited to image classification [29, 53, 55, 20, 23, 21, 74, 4, 12], segmentation [44, 19], object detection [36, 79], video understanding [28, 76]. and generation [11]. In contrast to the constrained receptive field in CNNs, vision Transformers [7, 42, 61], borrowed from the language tasks [59], are superior in global context modelling. Later, numerous vision-specific modifications are proposed to adapt the Transformer better to the visual domain, such the introduction of hierarchical features [42, 61], optimized training [58], and integration of CNN elements [5, 54]. Thus, vision Transformers demonstrate leading performance on various vision applications [63, 1, 8, 34, 48]. This, however, comes at a cost of attention operations' quadratic time and memory complexity. which hinders their
scalability despite remedies proposed [42, 61, 72, 72, 57].
More recently, State Space Models (SSMs) emerged as a powerful paradigm for modeling sequential data in language tasks. Advanced SSM models [13] have reported compared to state-of-the-en VIT architectures while having a linear complexity. Their initial success on vision tasks [80, 41] and, more importantly, remarkable computational efficiency hint at the potential of SSM as promising general general-purpose purpose backbone alternative to CNNs and Transformers.
superior performance
2.2 State Space Models
State Space Models (SSMs) [16, 13, 18, 35] are family of fully recurrent architectures for sequence modeling. Recent advancements [15, 14, 47, 13] have gained SSMs Transformer-leeel performance, yet with its complexity scaling linearly. As major milestone, Mamba [13] revamped the conven- tional SSM with input-dependent parameterization and scalable, hardware-optmized computation, performing on par with or better than advanced Transformer models on different tasks involving
sequential ID data.
Following the success of Mamba, ViM [80] and VMamba [41] reframe Mamba's ID scan into bi- directional and four-directional 2D cross-scar for processing images. Subsequently, SSMs have been quickly applied to vision tasks (semantic segmentation [51, 65, 46], object detection [26, 3], image restoration [17, 52], image generation [9], etc.) and to data of other modalities (e.g., videos [67, 32], point clouds [40, 73], graphs [2], and cross- coos-mmdality learning [60, 6)).
A fundamental consideration in adapting Mamba to non- 1D data concerns the design of path that scans through and maps all image patches into a SSM-friendly 1D sequence. Along this direction, preliminary efforts include the bi-directional zigzag-style scan in ViM [80], the 4-direction cross-scan in VMambba [41], and the serpentine scan in Plain [66] and ZigMa [22], all conducted the
spatial domain spanned by the height and width axes. Other works [52. 75, 33] extend the scanning to an additional channel [52, 33] or temporal [75, 33] dimension. Yet, by naively traversing the patches, these scan strategies overlooked the importance of spatial locality preservation. This inherent weakness has been partially mitigated by MocalMamba [26], which partitions patches into windows and performs traversal within each window.
However, due to a monolithic locality granularity throughout the entire image domain, as controlled by the arbitrary window size, it is hard decide DARTS [38] to differentially search for the optimal window size alongside the optimal scan direction for each layer, which adds to the complexity of the method. On the other hand, all existing methods involve hard-coded scan strategies, which can be suboptimal. Unlike all these methods, this paper introduces learnable quadtree structure scan image patches with varying locality granularity.
optimal granularity. Local opts for
3 Preliminaries
State Space Models (S4). SSMs [16, 15, 18, 35) are in essence linear time-invariant systems that map a one-dimensional input sequence sr() R4 to output response sequence y(1) E R recurrently through hidden state h(t) _ (with sequence length L and state size N). Mathematically, such
3

---

FEN
Stage XC
Stage
X2C
Stage
Stage
Norm
Patch Imbeodin
Dova Sumpling
Down Sampling
Down Sampley
QuiVVSS
Mamba layer
Qual-Soan
Stun
Norn
(0) Overaal architecture
(b) QuaVVSS Block
Figure 2: The pipeline of the proposed QuadMamba (a) and its building block: QuadVSS block (b). Similar to the hierarchical vision Transformer, QuadMamba builds stages with multiple blocks, making it flexible
as the backbone for vision tasks.
systems can be formulated as the following Ordinary Differential Equations (ODEs):
h' (t) Ah(t) + Bx(() v() = Ch(()
where matrix A E N contains the evolution parameters; B E Nx1 and C E RIxN are projection matrices.
The continuous-time ODEs Eqn. are, however, difficult to solve using deep learning. In practice, Eqn. 1 is usually transformed into discretized form [16] via the zero-order hold (ZOH) rule, where the value of æ is held constant over a sample interval [18]. The discretized ODEs are given by:
h(t) Ah(t 1) + Br(t),
y(t) Ch(t).
where A and B are the discrete version of A and B: A exp(AA), and (A-))-- For efficient implementation, the iterative computations in Eqn. can be performed in parallel with global convolution operation:
y == K with = (CB,CAA CA" I-1 CAA B),
where @ denotes the convolution operator and K E R the SSM kernel.
Selective State Space Models (S6). Traditional SSMs have input-agnostic parameters. To improve this, Selective State Space Models [13], also referred to as "S6" or "Mamba", are proposed with input-dependent parameters, such that A. B. and become learnable. To make up for parallelism difficulties, bardware-aware optimization is also performed. In this work, we particularly investigate the effective adaptation of the Mamba architecture for vision tasks. Early works such as ViM [80] and VMmmba [41] have explored intuitive adaptations by transforming 2D images to 1D sequences in raster scan fashion. We argue that simple raster scan is not an optimal design breaks the local adjacency of images. In our work, novel learnable scanning scheme based quadtree is proposed.
4 Method
4.1
General Architecture
QuadMamba shares similar multi-scale backbone design many CNNs [20, 64, 23] and vision le Hum.&WWWn Transformers [61, 71, 42, 25). As shown in Fig. 2, an image / E into patches of size 4 x 4, resulting in
is first partitioned visual tokens. linear layer maps these visual tokens to hidden embeddings with dimension d, which are subsequently fed into our proposed Quadtree-based Visual State Space (QuadVSS) blocks. Unlike the Mamba structure used in language modeling [13], the VSS blocks follow the popular structure of the
x W = 1 Hum x
W.m. 1

---

Hadamard product
Prediction
Softmax
summation
Up-sampling flatten
Quad window partition W.
I- Mask
Mask
Quad-bused sequence
Reverse & reshape
Windowed flatten
# W
Coarse-level
soquence
Quad window partition
Windowed flatten
Fine-leve
M
W
sequence
Figure 3: Quadtree- based selective scan with prediction modules. Image tokens are partitioned into bi-level window quadrants from coarse to fine. A fully differentiable partition mask is then applied to generate the ID sequence with negligible computational overhead.
Transformer block [7, 68), as illustrated in Fig 2(b). QuadMamba consists of a cascade of QuadVSS blocks organized in four stages, with stage (i E {1, 2, 3, 4}) having S; QuadVSS blocks. In each stage, a downsampling layer halves the spatial size of feature maps while doubling their channel dimension. Thanks to the linear complexity of Mamba, we are free to stack more QuadVSS blocks within the first two stages, which enables their local feature preserving and modeling capabilities to be fully exploited with minimal computational overheads introduced.
4.2 Quadtree-based Visual State Space Block
As shown in Fig our QuadVSS block adopts the meta aethitechiee [68] vision Transformer, formulated by token operator, feedforward network (FFN), and two residual connections. The token operator consists of a shift module, a partition map predictor, quadtree-based scanner, and Mamba Layer. Inside the token operator, a lightweight prediction module first predicts partition map over feature tokens. The quadtree-based strategy then partitions the 2D image space by recursively subdividing it into four quadrants windows. According the scores of the partition map at the coarse level, fine-level sub-windows within less informative coarse-level windows are skipped. Thus, multi-scale, multi-granularity ID token sequence is constructed, capturing more locality in more informative regions while retaining global context modeling elsewhere. The key components of the QuadVSS block are detailed follows:
Partition map prediction. The image feature E "xxCC containing total of N HW embedding tokens, is first projected into score embeddings ng:
(x)),
I NxC
where di is a lightweight projector with a Norm-Linear-GGil layer. To better assess each token's locality, leverage both the local embedding and the context information within each quadrant. Specifically, we first split 2, in the channel dimension to obtain the local feature x. and the context feature x global.
local
global
x.(0 G. 2z! f
: C1.
xx globall
BRRR #xxxx
(5)
Then, obtain 2 2 context vectors through adaptive pooling layer and broadcast each context vector v local into the local embedding along the channel dimension:
D local AdaptiveAvgPool22( local))
a local
2222x
(6)
x Concat (agloal Interpolate ( local)), xARR Hx
5

---

where Interpolate()) is the bilinear interpolation operator that upsamples the context vector to spatial size of H W. Thus, we obtain the aggregated score embedding x EE and feed Linear-GELU layer Up for partition score prediction:
s (i, ("
E
Hxxxx2
where j( denotes the partition score of the token at spatial coordinate (i.
Quadtree-based window partition. With each feature token's 's partition score s(i. j) predicted, we apply quick quadtree-basee window partition strategy with negligible computation cost. We construct bi-level window quad- rants to capture spatial locality from coarse fine. The quadtree-basee baadd strategy partitions the image feature into N1 sub-windowd windows at the fine level. Different from the Transformer scheme, which maintains the spatial shape image fea- tures with arbitrary Key/Value features simply by setting the Query feature as the same size, the token sequence generated by the learned scan for QuadMamba should be merged into the feature with the original spatial size. Thus, we select the top K 1 quadrant with the highest aver- aged local adjacency score at the coarse level and further partition it into four sub-windows:
W
the coarse level and N₂ = 16 sub-
Shift step
É Shift step
Shifted QuadVSS black
Shifted QuadVSS block
Figure 4: Omnidirectional window shifting scheme.
(wk) i 0,1,2,3) = QuadWinndowPPrtttoon(na)))
{skx i 2,1,2 A
(8)
k index
TopK( skk K =)=1;; i 10,1,2,3)
(k.)) ( | 0, 1, [uurtiindow
where coarse-level windows wk, and fine-level sub-windows w have spatial sizes of { H H W.
2 2 and
I 1 T. respectively. Afterward, construct ID token sequence from the coarse-level windows (k.j) j 0, 1, 3, based on their relative spatial (wk, * k} and fine-level sub-windows ( indices.
Differentiable 1D sequence construction. Although our target is to conduct adaptive learned window quadrant partition, it is non-trivial to construct the 1D token sequence from the 2D spatial windowed features. Directly sampling from the 2D feature according to the learned index for each sequence token is non difieifferenti,ble, which impedes end-to-end training. To overcome this, we apply a sequence masking strategy to formulate the token sequence, implemented by the Gumbel-Softmax technique [27]:
Mx. Gumbel-SSofmax(((s)) i- 0, 1,2,3)) 10,1),M, E 2
where value "1" in My, represents the selected coarse-level windows, which will be divided into subwindows further; "0" denotes those non-selected. differentiable operator, Gumbel-Softmall effectively gets around the non-differential index selection Eqn. 8. Then, we first obtain two token sequences the coarse and fine levels separately:
H W
LN1 = QuadWindowAArraneeea
)),
2
(10)
H W 4 )),
LN, = QuadWindooAArangg(eesteeee (L,, )((
w)) flattens the 2D feature into 1D sequence based on the fixed size (h, w) window partition order. Restore(-) is to convert the ID sequence back to the quadtree- window partitioned 2D feature. Next, we upsample the mask MN, to obtain the full sequence mask M E R xxx1 We then use Hadamard product @ and element-wise summation to construct the 1D token sequence with negligible computation cost:
where QuadWindowAArangef/
L = (M & LN,) @ ((1 M) ④ IN,)),
(1I)

---

where sequence L contains tokens in both coarse- and fine-level windows and is sent to the SS2D block for sequence modeling.
Omnidirectional window shifting. Considering the case that the most informative tokens are crossing two adjacent window quadrants, we borrow the idea of shifted window scheme in Swin Transformer [42]. The difference is that the Transformer ignores the spatial locality for each token inside the window, while the token sequence inside the window in Mamba is still directional. Thus, we add additional shifted directions in the subsequent VSS blocks as shown in Fig 4, compared to only one direction shifting Swin Transformer.
4.3 Model Configuration
It is noteworthy that QuadMamba's 's model capacity can be customized by tweaking the input feature dimension d and the number of (Q)VSS layers {(S))== In this work, build four variants of the QuadMamba architecture, BandMambaaLLi//SS with varying capacities:
Lite block: (2, 2, QuadVSS stages:(1, 2). #Params: 5.4M, FLOPs: 0.82G.
Tiny block: 6, QuadVSS stages: [1, 2). #Params: 10.3M, FLOPs: 2.0G.
Small block: b 5,,
QuadVSS stages: (1, 2), #Params: 31.2M, FLOPs: 5.5G.
Base - [lock: 2, 15, QuadVSS stages: (1, 2). #Params: 50.6M, FLOPs: 9.3G.
In all these variants, QuadVSS blocks are placed in the specified QuadVSS model stages to bring into full play their locality preservation capabilities on higher-resolution features, Omnidirectional shifting layers are applied in every other QuadVSS blocks. More details are found in the Appendix.
5
Experiment
We conduct experiments on commonly used benchmarks, including ImageNet- [29] for image clas- sification, MS COC02017 [37] for object detection and instance segmentation, and ADE20K [78] for semantic segmentation. Our implementations follow prior works [42, 80, 41]. Detailed descriptions of the datasets and configurations found in the Appendix. In what follows, compare the proposed QuadMamba with mainstream vision backbones and conduct extensive ablation studies to back the motivation behind QuadMamba's designs.
5.1 Image Classification on ImageNet-1k
Tab. demonstrates the superiority of QuadMamba in terms accuracy and efficiency. Specifically, QuadMamba-S surpasses RegNerT-8G 8- [50] by 2.5%. DeiT-S [58] by 2.6%, and T by 1.1% Top-1 accuracy, with comparable or reduced FLOPs. This advantage holds when comparing other model variants of similar numbers of parameters FLOPS. Compared other Mamba-based vision backbones, QuadMamba also yields favorable performance under comparable network complexity. For instance, Maadaam (83.8) performs par with or better than VMamba-B (83.7%), LocalVim-S (81.2% 2%), and PlainMamba-L3 (82.3%), while having significantly less parameters and FLOPs. These results manifest the performance and complexity superiority of QuadMamba and its potential as a powerful yet highly efficient vision backbone. Furthermore, QuadMamba achieves similar or higher performance than LocalMamba while being completely free of the latter's expensive architecture and scan policy search, which makes it a more practical and versatile choice of vision backbone.
5.2 Object Detection and Instance Segmentation on COCO
On object detection and instance segmentation tasks, QuadMamba stands out as a highly efficient backbone among models and architectures of similar complexity, as measured by number of net- works parameters and FLOPs. QuadMamba finds very few competitors under the category of tiny backbones with less than or around 30M parameters. As shown in Tab. 2, in addition to dramatically outperforming ResNet18 8 [20] and PVT-T [61]. QuadMamba-T also leads Efficient Mamba-S [49] by considerable margins of 3. 3.0% mAP on object detection and 2.1% on instance segmentation. Among larger backbones, Mamda once again surpasses all ConyNet- Transformer-, and Mamba-based
7

---

Table 1: Image classification results on ImageNet-1k, Throughput (images 8) is measured on
single V100 GPU. All models are trained and evaluated 224 resolution.
Model
#Params (M)
FLOPs (G)
Top- (%)
Top-5 (%)
ResNet-18 [20]
11.7
1.8
69.7
89.1
ResNet-50 [20]
25.6
4.1
79.0
94.4
ResNet-101 [20]
44.7
7.9
80.3
95.2
ConvNet
RegNerY-4G [50]
20.6
4.0
79.4
94.7
RegNetY-8G [50]
39.2
79.9
94.9
RegNerY-16G [50]
83.6
15.9
80.4
95.1
DeiT-S [58]
22.1
4.6
79.8
94.9
DeïT-B [58]
86.6
17.6
81.8
95.6
PVT-T [61]
13.2
1.9
75.1
92.4
PVT-S [61]
24.5
3.7
79.8
94.9
PVT-M [61]
44.2
6.4
81.2
95.6
Transformer
PVT-L. [61]
61.4
9.5
81.7
95.9
Swin-T [42]
28.3
4.5
81.3
95.5
Swin-S [42]
49.6
8.7
83.3
96.2
Swin-B [42]
87.8
15.4
83.5
96.5
Vim-Ti Ti [42]
76.1
93,0
Vim-$ [42]
26
80.5
95.1
VMamaba-1 [42]
22
4.5
82.2
VMamaba-S [42]
44
9.1
83.5
VMamaba-B [42]
15.2
83.7
LocalVim-Te [26]
8
1.5
76.2
LocalVim-S [26]
28
4.8
81.2
PlainMamba--LI [66]
3.0
77.9
Mamba
PlainMamba--12 [66]
25
8.1
81.6
PlainMamba-L3 [66]
50
14.4
82.3
QuadMamba-Li
5,4
0.8
74.2
92.1
QuadMamba-TT
10
2.0
78.2
94.3
Ours
QuadMamba-S Qaad-aa
31
5.5
82.4
95.6
83.8
96.7
rivals. Notably, QuadMamba-S outperforms EfficientV/Mambae a Mamba-based backbone charac- terised by its higher efficiency, by 3.0% mAP on object detection and 2.2% on instance segmentation, using comparable parameters. Furthermore, QuadMamba-S is able to keep up with and even surpass LocalVMamba- [26] while circumventing a whole load of architecture and scan search hassles not reflected by Tab. 2's complexity measurements. These results show how QuadMamba can serve as strong and versatile vision backbone with a pragmatic trade-off among computational complexity, design costs, and performance.
the performance
5.3 Semantic Segmentation on ADE20K
As shown in Tab. 3, under comparable network complexity and efficiency, QuadMamba achieves sig- nificantly higher segmentation precision than ConvNet-basee ResNet-50/101 [20] and ConvNeXt [43], Transformer-based DeiT [58] and Swin Transformer [42], well most Mamba-based architec- tures [80, 49, 66). For instance, QuadMamba-S with 47 47.2% mloU reports superior segmentation precision to Vim-S (44.9%), Lical (46.4%), Efficient
(46.5%), and PlainMamba- L2 (46.8%), and competitive results to VMamba-T (47.3%). Compared with LocalMamba-S// QuadMamba-S/I trails by small margins, yet enjoying the advantage of not incurring extra network searching costs. It is worth noting that the LocalMamba is designed by the Neural Architecture Search (NAS) technology, which is data-dependent and lacks flexibility with other data modalities and data sources.
5.4 Ablation Studies
We conduct ablation studies to validate QuadMamba's ss design choices from various perspectives. QuadMamba-T is used for all experiments unless otherwise stated.

---

Table 2: Object detection and instance segmentation results on the COCO val2017 split using the
Mask RCNN [19] framework.
Backbones
#Params (M)
FLOPs (G)
Ap
APhis
Apbux
Al prinkk
Aprusk
APaask
R18 [20]
31
207
34.0
54.0
36.7
31.2
51.0
32.7
PVT-T [61]
32
208
36.7
59.2
39.3
35.1
56.7
37.3
Vit.T [?1]
26
145
41.4
63.5
45.0
60.3
40.8
Efficient Mambba-S [49]
31
197
61.8
42.6
36.7
58.9
QuadMamb
25
186
39.3
61.7
42.4
36.9
58.8
39.4
QuadMamba-T
30
213
42.3
64.6
46.2
38.8
61.6
41.4
R50 [20]
44
260
38.6
59.5
42.1
35.2
56.3
37.5
PVT-S [61]
44
40.4
62.9
43.8
37.8
60.1
40.3
Swin-T [39]
48
267
42.7
65.2
46.8
39.3
62.2
42.2
ConvNeXI-T [43]
48
262
44.2
66.6
48.3
40.1
63.3
42.8
AificicntYMambee [49]
53
252
43.7
66.2
47.9
40.2
63.3
42.9
PlainMamha--.22 [66]
53
542
46.0
66.9
50.1
40.6
63.8
43.6
ViL-S [71]
45
218
44.9
67.1
49.3
41.0
64.2
44.1
VMamba-T [41]
42
262
46.5
68.5
50.7
42.1
65.5
LocalVMamba-TT [26)
45
291
46.7
68.7
50.8
42.2
65.7
45.5
QuadMamba-S
55
301
46.7
69.0
51.3
42.4
65.9
45.6
Table 3: Semantic segmentation results on ADE20K using UperNet [62]. mloUs are measured with
single-scale (SS) and multi-scale (MS) testings on the val set. FI.OPs are measured with an input size of 512 2048.
Backhone
Image size
#Params (M)
FLOPPs (G)
mloU (SS)
mloU (MS)
Efficient [49]
5122
230
38.9
39.3
DeîT-Ti [58]
5122
11
39.2
Vim-Ti [80]
5122
13
40.2
EfficientVMambee [49]
5122
29
505
41.5
42.1
Vom [26]
5122
36
181
43.4
44.4
PlainMamba-LL [66]
5122
35
174
44.1
QuadMamba-T
5122
40
886
44.3
45.1
ResNet-50 [20]
5122
67
953
42.1
42.8
DeiT-S + MI.N [58]
5122
1217
43.8
45.1
Swin-T [42]
5122
60
945
44.4
45.8
Vim-S [80]
5122
46
44.9
LicalVim-S [26]
5122
58
297
46.4
47.5
EfficientVMambaa-B [49]
5122
65
930
46.5
47.3
PlainMamba--L2 [66]
5122
55
285
46.8
VMamba-T
5122
55
964
47.3
48.3
LocalV [26]
5122
57
970
47.9
49.1
QuadMamba-S
5122
62
961
47.2
48.1
ResNet- 101 [20]
5122
85
1030
42.9
44.0
DeîT-B + MLN [58]
5122
144
2007
45.5
47.2
Swin-S [42]
5122
81
1039
47.6
49.5
PlainMamba-LL3 [66]
5122
81
419
49.1
VMamba-S [41]
5122
76
1081
49.5
50.5
MamalVMamba-S [26]
5122
81
1095
50.0
51.0
QuadMamba-B
5122
82
1042
49.7
50.8
Effeet of locality in Mamba. We factorise the effect of coarse- and fine-grain locality modeling in building 1D token sequences on model performance. Specifically, we compare the naive window-free flattening strategy in [80, 41] that overlooks 2D locality against window partitions in three scales (i.e., 28 x 28, 14 14, 2 2) that represent three granularity levels of feature locality. In practice, replace QuadVSS blocks of a QuadMamba-T model with the plain VSS blocks in [41]. To exclude the negative effects of padding operations, only partition the features with a spatial size of 56 56 the first model stage. As illustrated in Tab. 4, the naive sean strategy leads to significantly degraded object detection and instance segmentation performance compared to when windowed scan is adopted. The scale of local windows is also shown to considerably influence the model performance, which small window given the image resolution
suggests that too large
be suboptimal.
Quadtree-based partition resolutions. We examine the choice of partition resolutions in the bi-level quadtree-based partition strategy. The configured resolutions in Tab. 5 are applied in the first two

---

Table 4: Impact of using different local window sizes and the naive flattening strategy.
Table 6: Impact of different number of (Quad)VSS blocks per stage.
Table Impact of different local window partition reso- lutions.
Window Size Top-1 (%6 Aphix Apmask
Depth #Prrams FLOPs Top-I (%)
Coarse
Fine
Top-1
(%)
2-2-8-2
31
82.0
w/ windows
28 x 28
72.2
72.9
33.1
33.8
31.7
30.5
(H, W)
R A d
W
73.5
2-8-2-2
38
81.5
2 2
14
72.4
73,5
33.4
35.8
32.1
30.6
8 W
1
4
it
8
73.6
73.8
2-4-6-2
2-2-2:8
74
36
8.5
7.8
81.8
82.1
pM
Score
Logar
Loger 2
Liow 3
Layer 4
Togee
Lager
Laye1
Jaye X
Impact of different Figure 6: Visualization of partition maps that focus on different layer patterns and shift directions. regions from shallow to deep blocks.
Figure 5:
model stages with feature resolutions of (56 56, 28 28, 14 14). Experimentally deduce the optimal resolutions to be {1/2, / for the coarse- and fine-level window partition, respectively, This handcrafted configuration may be replaced by more flexible and learnable ones in future work.
Layer patterns in the hierarchical model stage. We investigate different design choices of the layer pattern within our hierarchical model pipeline. From Fig. 5, layer pattern LP2 outperforms LP1 with less QuadVSS blocks by 0.2% accuracy. This is potentially due to the effect of locality modeling being more pronounced in shallower stages than in deeper stages as well as the adverse influence of padding operations in stage 3. LP3, which places the QuadVSS blocks in the first two stages and in an interleaved manner, achieves the best performance, and is adopted our model design.
Necessity of multi-directional window shift. Different from the unidirectional shift in Swin Transformer [42], Fig. 5 shows a 0.2% gain in accuracy as complementary shifting directions are added. This is expected since attention in Transformer is non-causal, whereas ID sequences in Mamba, being causal in nature, are highly sensitive relative position. A multi-directional shifting operation is also imperative for handling cases where the informative region spans across adjacent windows. Fig. 6 further visualizes the shifts in the learned fine-graine quadrants across hierarchical stages, which adaptively attend to different spatial details at different layers.
Numbers of (Quad)VSS blocks per stage. We conduct experiments to evaluate the impact of different numbers of (QS blocks each stage. Tab. presents four configurations, following design rule LP3 in Fig. 5, with a fixed channel dimension of 96. We find that a bulky 2nd or 4th stage leads to diminished performance compared to a heavy 3rd stage design, whereas dealing out the (QS blocks more evenly between stages 2 and 3 yields comparable if not better performance with favorable complexities. This evidence can serve as a rule of thumb for model design in future work, especially the model scales up.
6 Conclusion
In this paper, we propose QuadMamba, a vision Mamba architecture that serves as a versatile and efficient backbone for visual tasks, such as image classification and dense predictions. QuadMamba effectively captures local dependencies of different granularities by learnable quadtree-bast based scanning. which adaptively preserves the inherent locality within image data with negligible computational overheads. The QuadMamba's 's effectiveness has been proven through extensive experiments and ablation studies, outperforming popular CNNs and vision transformers. However, one limitation of Mamba is that window partition with more than two levels is yet to be explored, which may be particularty larly relevant for handling dense prediction visual tasks and higher-resolution data, such as remote sensing images. The fine-grained partition regions are rigid and lack flexibility in attending to regions of arbitrary shapes and sizes, which is left for our future work. We hope that our approach will motivate further research in applying Mamba to more diverse and complex visual tasks.
10

---

This work was supported by NSFC (62322113, 13, 62376156), Shanghai Munic- ipal Science and Technology Major Project (2011SHDDZZ0102), and the Fundamental Research Funds for the Central Universities.
Acknowledgments.
References
[1] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: video vision transformer. In CVPR, 2021.
[2] Ali Behrouz and Farnoosh Hashemi. Graph mamba: Towards learning on graphs with state space model. arXiv preprini arXiv:2402.08678, 2024.
[3] Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint wwXiv:2403.19888 2024.
[4] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In 2017.
[5] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In Neurips, 2021.
[6] Wenhao Dong, Haodong Zhu, Shaohui Lin, Xiaoyan Luo, Yunhang Shen, Xuhui Liu, Juan Zhang. Guodong Guo, and Baochang Zhang. Fusion-mamba for cross-modality object detection, 2024.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
[8] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[9] Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Scalable diffusion models with state space backbone, 2024.
[10] Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. 1980.
[11] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In CVPR, 2014.
[12] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, 2018.
[13] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023.
[14] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Neurips, 2022.
[15] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00300 2021.
[16] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Neurips, 2021.
[17] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: A simple baseline for image restoration with state-space model. arXiv preprint arXiv:2402. 15648. 2024.
[18) Ankit Gupta, Albert Gu, and Jonathan Berant, Diagonal state spaces are as effective as structured state spaces. In Neurips, 2022.

---

[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
[22] Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Björn Ommer. Zigma: A dit-style zigzag mamba diffusion model, 2024.
[23] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In CVPR, 2017.
[24] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
[25] Tao Huang, Lang Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Lightvit: Towards light-weight convolution-free vision transformers. arXiv preprint arXiv:2207.05557, 2022.
[26] Tao Huang. Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. In WACV, 2024.
[27] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.
[28]
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
[29] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. 2012.
[30] Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
[31] Yann LeCun, Lottou Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
[32] Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Limin Wang, Yali Wang, and Yu Qiao. Video- mamba: State space model for efficient video understanding. In ECCV, 2024.
[33] Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprini arXiv:2402.0558 058892 2024.
[34] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022.
[35] Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolu- tional models great on long sequence modeling? arXiv preprint arXiv:2210.09290 2022.
[36] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[37] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
[38] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
[39] Jiarun Liu, Hao Yang. Hong- Yu Zhou, Yan Xi, Lequan Yu, Yizhou Yu, Yong Liang, Guangming Shi, Shaoting Zhang, Hairong Zheng, et al. Swin-umamba: Mamba-based unet with imagenet- based pretraining, arXiv preprint 43X3v22402.03302, 2024.
12

---

[40] Jiumiin Liu, Ruiji Yu, Yian Wang, Yu Zheng, Tianchen Deng, Weicai Ye, and Hesheng Wang. Point mamba: A novel point cloud backbone based on state space model with octree-based ordering strategy. arXiv preprint vrXiv:2403.0646) 2024.
[41] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu, Vmamba: Visual state space model. Neurips, 2024.
[42] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In CVPR, 2021.
[43] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. 2022.
[44] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
[45] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[46] Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.0472. 2024.
[47] Eric Nguyen, Karan Goel, Albert Gu. Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. 2022.
[48] William Peebles and Saining Xie. Scalable diffusion models with transformers. In CVPR, 2023.
[49] Xiaohuan Pei, Tao Huang, and Chang Xu. Efficientvmamba: Atrous selective scan for light weight visual mamba. arXiv preprint 24Xiv:2003309977,, 2024.
[50] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In CVPR, 2020.
[51] Jiacheng Ruan and Suncheng Xiang. Vm-unet: Vision mamba unet for medical image segmen- tation. arXiv preprint arXiv:2402.02491, 2024.
[52] Yuan Shi, Bin Xia, Xiaoyu Jin, Xing Wang. Tianyu Zhao, Xin Xia, Xuefeng Xiao, and Wen- ming Yang. Vmambair: Visual state space model for image restoration. arXiv preprint
208452240..15648, 2024.
[53] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556. 2014.
[54] Aravind Srinivas, Tsung-Y. Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition, 2021.
[55] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[56] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016.
[57] Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In ICLR, 2022.
[58] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint 22877 2020.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neurips, 2017.
[60] Zifu Wan, Yuhao Wang, Silong Yong, Pingping Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xic. Sigma: Siamese mamba network for multi-modal semantic segmentation, 2024.
13

---

[61] Wenhai Wang, Enze Xie, Xiang Li. Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In CVPR, 2021.
[62] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. ECCV, 2018.
[63] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers, 2021.
[64] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
[65] Zhaohu Xing, Tian Ye, Yijun Yang, Guang Liu, and Lei Zhu, Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint vrXiv:2401.13560 2024.
[66] Chenhongyi Yang, Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot J. Crowley. Plainmamba: Improving non- non-hierarchica. mamba in visual recognition. arXiv preprini arXiv:2403. 17695, 2024.
[67] Yijun Yang, Zhaohu Xing, Chunwang Huang, and Lei Zhu. Vivim: video vision mamba for medical video object segmentation, 2024.
[68] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022.
[69] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[70] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 17X1::1710.09412, 2017.
[71] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In CVPR, 2021.
[72] Qinglong Zhang and Yu-Bin Yang. Rest: An efficient transformer for visual recognition. In Neurips, 2021.
[73] Tao Zhang, Xiangtai Li, Haobo Yuan, Shunping Ji, and Shuicheng Yan. Point cloud mamba: Point cloud learning via state space model. arXiv preprint arXiv:2403.00760 2024.
[74] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
[75] Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective ssm. arXiv preprint arXiv:2403.07487, 2024.
[76] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin, Temporal action detection with structured segment networks. In ICCV, 2017.
[77] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, 2020.
[78] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[79] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018.
[80] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09411 2024.
14

---

A Appendix
In the supplementary materials, we provide more details and analysis of QuadMamba in Sec. A. implementation details in Sec. A.2, and more information on QuadMamba's ' model variants in Sec. A.3. In Sec.A.4, we also provide the pseudo-code to help understand the key operations within the QuadVSS block. Next, we present the complexity analysis and the throughput measurement of our proposed QuadMamba variants in Sec.A.5. Finally, we provide more visualization results of QuadMamba in Sec.A.6.
A.1 Schematic Illustration
To better assess the impact of the Mamba model and our proposed learnable scanning method, we integrated our key token operator into the meta-architecture of the Vision Transformer, as illustrated in Fig. 7.
FFN
SS2D
Norm
Conv
SS2D
Lincar
Lincar
Norm
(b) QuadVSS Block
Vision Mamba Block (Gated Network)
Figure 7: Architecture of (a) the vanilla VSS block, which is in the form of a gated network, and (b) our modified block, which adopts the meta-architecture of the transformer and comprises token operator, an FFN, and residual connections.
Relationship to RNNs. The concept of recurrence in a hidden state is closely linked Recurrent Neural Networks (RNNs) and State Space Models (SSMs), as suggested by [13]. Some RNN variants employ forms of gated RNNs and remove the time-wise nonlincarities. These can be considered as combination of the gating mechanisms and selection mechanisms.
A.2 Implementation Details
Following the VMamba [41], we conducted benchmark assess the image classification perfor- mance of QuadMamba on the ImageNet- 1 [29] dataset. ImageNet [29] is widely recognized as the standard for image classification benchmarks, consisting of around 1.3 million training images and 50,000 validation images spread across 1,000 classes.
The training scheme is based on DeiT [58]. The data augmentation techniques used include random resized crop (input image size of 2242), horizontal flip, RandAugment [77], Mixup [70], CutMix [69], Random Erasing [77], and color jitter. Additionally, regularization techniques such as weight decay, stochastic depth [24], and label smoothing [56] are applicd. All models are trained using Adam W [45]. The learning rate scaling rule is calculated as BatchSize x 10-3, Our models are implemented with 1024
PyTorch and Timm libraries and trained on A800 GPUs.
For object detection and instance segmentation, we assess QuadMamba using the MS COC02017
dataset [37]. The MS COCO2017 dataset is widely used for object detection and instance segmen- tation and consists of 118,000 training images, 5,000 validation images, and 20,000 test images of common objects.
15

---

Table 7: Our detailed model variants for ImageNet-lk, Here, The definitions follows: "Cony k_c_s" denotes convolution layers with kernel size k, output channel c and stride s. "MLP_ __" is the MLP structure with hidden channel 4c and output channel And "(Quad)VSS, ___r"" r" is the VSS operation with the dimension expansion ratio n and the channel dimension т. "C" is 48 for QuadMamba-Li and 64 for QuadMamba-S, and 96 for QuadMamba-B and QuadManba-L.
Name
Output
Lite
Small
Basc
Large
stem
x6
patch eencc__emed:: Conv-3 C/2,,, Conv-3 3_2_1, Conv-3_C_2
QuadVSS 1,48
QuadVSS 6_
QuadVSS 2_96
QuadVSS 2_96
stagel
56
MLP_46 _88
x2
MLP_64
MLP_99
MLP P_96
patch embed: Conv-3_20 22_2
stage2
x8x
QuadVSS __96 MLP_96
QuadVSS_ 1_18
QuadVSS_2 2_________________
QuadVSS_2 2_ 192 MLP_192
x2
MLP_12S 128
x
MLP_192
x2
x2
patch embed: Conv-3_4C.
stage3
14 14
VSS 1_192 MLP 192
VSS 226
2_384
VSS 2_384 MLP
x2
MLP 256
x5
MLP_384
x5
x
patch embed: Conv-3_8C__
stage4
7x7
VSS 1 L84 MLP 384
VSS 1 512
VSS 2 768 MLP 768
VSS 2 768
x-
MLP 512
MLP_512
x2
MLP 768
x2
Clussifier
1x1
average pool. 1000d fully-connected
GFLOPs
0.82G
2.07G 10.32M
5.51G 31,25M
9.30G 50.6M
Params
5.47M
Table 8: Increased model costs when Quad blocks are applied in the first two stages. The blocks four stages are 2,)
QuadBlock
Channel
#Params.
FLOPs
48
5,4
0.78
48
5.5 (+1.8%)
0.86 (+10.2%)
24.8
3.7
27.1 (+9.2%)
4.8 (+31.6%)
For semantic segmentation, we evaluate QuadMamba on the ADE20K [78] dataset. ADE20K is popular benchmark for semantic segmentation with 150 categories. consists of 20,000 training images and 2,000 validation images. Following Swin Transformer [42]. we construct UperHead [62] with the pretrained model. The learning rate is set as 6 x 10 10-5 5 The finettunigg process consists of a total of 160,000 iterations with batch size of 16. The default input resolution is 512x5 12, and the experimental results are using 640x640 inputs and multi-scale (MS) testing.
A.3 Model Variants
We present the detailed architectural configurations of QuadMamba variants in Tab. 7. Each building unit the model variants and the corresponding hyper-parameters are illustrated in detail. We mostly place the proposed QuadVSS block in the first two stages and place the plain VSS block from [41] in the latter two stages. The reason is that the features in the shallow stage have larger spatial resolution and need more locality modeling.
A.4 Key Operators in QuadVSS Block
In this section, we present a detailed illustration of the QuadVSS block. The overall structure of the QuadVSS block is presented in Alg. 1. The process to partition feature tokens via the bi-level quadtree-based strategy is outlined Alg. 2, and that to restore them back into a 2D spatial feature is described in Alg. 3. Moreover, in Alg. 4, we provide the pseudo-code of the proposed sequence masking strategy to construct 1D token sequence differential manner. In Fig. 8, we illustrate the detailed pipeline of the ominidirectional shifting scheme in two Qua blocks. Moreover, in
16

---

Table 9: Throughputs of QuadMamba variants. Measurements are taken with A800 GPU.
Model
#Params.
FLOPs
Throughput
QuadMamba-L Li
5.4
0.8
1754
QuadMamba-T
10.3
2.1
1586
QuadMamba-S
31.2
5.5
1252
QuadMamba-B
50.6
9.3
582
W
reverse
reversc
Shift step
Shift step
Shifted QuadVSS block
Shifted QuadVSS 1+2
Figure 8: Ominidirectional shifting in two successive QuadVSS blocks. Two directions are comple-
mentary to each other, which mitigates the issue of the informative region spanning across adjacent windows.
Fig. 9, we provide the details of three different quadtree-based partition resolution configurations at coarse and fine levels, which are mentioned in Tab. 5.
A.5
Complexity & Throughput Analysis
Our learnable scanning method does not alter the total sequence length I, = x W. Thus, the QuadVSS block retains the linear computational complexity of (L)) of mainstream vision Mamba architectures [80, 41, 26, 66]. Other additional model costs are negligible. In Tab. 9, we show the throughput of the proposed QuadMamba variants, which are measured in the V800 GPU platform.
The computation complexity of standard Transformer as follows: Assuming its input X N has total input token number of N H W and channel dimensions of D, the FLOPs for the transformer attention can be calculated as:
FLOPs 4HW
(12)
It shows quadratic complexity with the input size of the transformer attention scheme, compared to the linear complexity of QuadMamba.
A.6 Visualization Results
In this section, provide more visualization results. In Fig. 10, we visualize the partition maps that focus on different regions from shallow to deep blocks, using QuadMamba-T as an example. In Fig. 11, we showcase more visualizations of the Effective Receptive Field (ERF) of CNN, Transformer, and Mamba variants. As can be observed, QuadMamba exhibits not only a global ERF but also a more locality-senn sensitvvee response compared to VMamba [41] with naive flattening strategy.
17

---

Coarse level
(H,
7
W
Fine level
W
2'2
(a)
(b)
(c)
Figure Details of the three different local window partition resolution configurations.
Algorithm 1 PyTorch code of QuadVSS block
import torch
import torch.nn
class QuadvssBlock Module)
def (seit_ dim, expension (ippensio_rrtio08/3,, sesm_d_stated int=16, conv_ cenw_rationl.oo .0,
ssm_dt_ ssm__t_rrank:Ang ""auto", ssm_conv: conv:iit3,, sum ssm_conv bias-True, sum sum_ satp_rate::ll Elost =0, alp_ratio 4.0, ml a___a___aye nn.G, norm_ corm_layerspartis (nn. layanoom,,,,, eps-lc-6), act aay_ super init innitt))__iit_0)) self.norml norm norm_layerddim)) self. norm_ (din)
drop_path-0.11
hidden wetteeeienn__eeell
dim)
self.token_op = (dadEs2)(d_model hidden, datate sam_d_atate ssm_ratio
ssm_ratio, dt_r ssm_dt_ranl, lay_r ssm_act_ layer, donov sum ccnv, canv ssm csm__onn_biie dropout rat_
self fff.droo_paah D-opPath (drop_pattt self.mlp MLP (hidden, drop mig rra_ddr
def (self,
x imput
self.drop arop_path (self. norml (self zoker_op (x))
x self.drop_aath www..ttthhhh.tto.oeeeeee wsw.f.eel............. www.sellll........mmmm.mmmmmlmmm (x)1)
return x
18

---

Algorithm 2 PyTorch code of Quadtree window partition at two levels
import torch
import torch.nn nn as
def coarse_level shape
B, M, quad-2)
E, x.view e, E, X)
quadl quad2 quad
math.ceil - math.cail /quad)
quad)
x.view muadll, quad2, ppermue 2, reshapp (B, -1) return x
def fine_ beee__eer__uuatree_dddooooorrrrr
H, K, quad-2)
B, C, L- x. Shape
v.ew
quadl quad2 - quad3 quad4 quad
math.cail
quad) quad)
math.ceil / quad) / quad)
x.view quadi, quad3*h, quad2, quada.w) (B, c, quadi, quad3, h, quad2, quad4, permutee 1, 2, 5, 6, 4, 7) reshappp 6, -1)
return
Algorithm PyTorch code of Quadtree window restoration levels
def coarse_levell que suw.drro_wiinooo.oootttttiii
H, W, quad-2)
B, C, 1 shsaaee
quadl quad2 quad h mmath.ceill
quad)
math.cell(M /quad)
y- (B, quadl, quad, quad2, h, permute (0, 1, 2, 4, 3, 5) reesaapee (B, o, -1)
return
def Ltt.eel__garrreeewoooooror
(v, H, quad=2)
B,
shaae
quadi quad2 = quad3 quad4 = quad
math.coill
quad) quad)
math. ceil (K / quad) / quad)
y- y.view (B, quadi, quad3, quad2, quadd, h, w) we.germute 2, 3, 4, 5, 7)
reshape (B, -1)
return
Algorithm 4 PyTorch code of differentiable sequence masking
x.reehepe sesiiie__vg__ooo po (szore i 0:1, :), (2, 2) + b, 1, 2, perr_keep_decciine gr.gumbel__soee sortmax (score_Mo) (score_window, weweeewindwwwwwd... (Be 1, -1), dim=-1, hazd-True)
score score_ window F.adeptive_avg
unsqueeze unsqueeze (-1) [bb 1, 4, 1, 1]
B-int (8), M-int(()) (bb, x maskake nonseeect. x (I.0 hard_keep_decik deci_deeeciiiass.l
derc_keep_decssionnnee I,
window_expanslon leardd_keep_eeceeee
w_mmassed__elllee pelecct x_ss hard hard_keep_peeeeec wwwppeeecppsiieeeeeeeeeeeeeeeese dww.iiiicninaaaee
leeal scan quad region
masked select seleect_oceesssan
beaeed_ooneelect_oooalleo macll_scan_uuae guad ix massee В-H, K-9) tBCHK B,C C,L
x_maseed
local_scar_quall (x_maaked_nosseeerr 8-8, 4BCHN -> B,
window
3,C,L
19

---

Score map in Layer
Layer
Layer 2
Layer
Layer
Layer 5
Layer
Layer
Layer
T
Figure 10: Visualization of partition maps which focus different regions from shallow to deep blocks. The second column shows the partition score maps in the 1st block.
Before training
After training
02
ResNet
DeiT
Swin
VMMamba
QuadMamba
Figure 11: Visualization Effective Receptive Field (ERF) of CNN, Transformer, and Mamba variants. Our QuadMamba exhibits not only a global ERF but also more locality-sensitive response compared to VMamba with naive flattening strategy.
20

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Sec. 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Sec. 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
See Sec.3 and Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Sec.4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: See Abstract and Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: /ttes://nips... cc/
public / guides/
des CoodeSuomi ssi asionPoli iiyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cc/
wttppcccppccccccccccc ppbppc/lc///oo httcc//gg/s/////eeeeceeeecec /code//CCeea "CoeeoSuuaa Submi as ssssssssooolloooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Sec.5 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Sec.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sec.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tt////euut /neuriss....p// httppubblce/ttcccccccceeeeellll
Answer: [Yes]
Justification: See Sec.5.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No potential social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No risk for misusing data and models. All data and models are publicly available for academic usage.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See reference part.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperso a ww withcode com/ / dat asets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
25

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "9T0f5MY8cf", "reviewer": "Reviewer_HNK1", "summary": "The paper introduces QuadMamba, an enhancement of vision State Space Model (SSM) architectures. At its core is a learnable QuadVSS network block that processes the image input patch at two different resolutions. For every 2x2 coarse window with 4 image patches, the method adaptively learns to process one of the 4 patches at the finer 2x resolution using the differentiable Gumbel-softmax formulation. These block are hierarchically stacked similar to SwinTransformer, but using a window shifting scheme in two different directions which is a better fit for SSM. \n\nThe method is demonstrated to obtain good results relative to comparably sized transformer and CNN architectures on Imagenet classification, COCO object detection / instance segmentation, and ADE20k semantic segmentation.", "questions": "- Is your quad idea limited to SSM or would it also work for Vit? Can one learn to downsample specific patches in a Vit? \n- Do you have comparisons between networks made of QuadVSS/VSS or only VSS for several same FLOPS budgets? Do you have comparison of training overhead (compute/memory) for networks made of QuadVSS/VSS or only VSS baseline? \n- L175 Do you always pick the same fine-grained patch for all 2x2 windows, or do they vary by window? Not particularly clear from the exposition/notation. \n- Also, it appears that 7 patches are picked for each region as per L185-187. Don't you want to have 8 patches (powers of 2 are usually more efficient hardware-wise?) \n- The softmax in Eq 7, what labels do you train it on? Is it the final labels? Right now it appears that this part is trained first, and kept fixed.", "limitations": "Limitations are adequately addressed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Intuitive idea to learn how to pick an area that requires higher resolution processing and pack it into SSM via Gumbel Softmax. Solid although not particularly novel stacking of QuadVSS blocks into a SwinTransformer-like network architecture. \n- Competitive results relative to ViT and CNN, and seems to improve a little bit over other Mamba methods, although the gains there seem quite incremental. \n- Ablation over multiple network parameter decisions.", "weaknesses": "# Significance\nThe contribution seems a bit incremental. The overall idea of image traversals into several windows to enforce better locality was already explored before in LocalMamba. The gains in the experimental section over some of the other Mamba network variants (EfficientVMamba, VMamba, Swin-S etc) are not too large. The idea that we do not have to process all areas in high resolution but only 1/4 of them does seem useful, although does not seem strictly limited to SSMs -- e.g. would ViT methods also benefit? \n\n# Experimental results\n- It seems that the natural baseline to this method is to compare a network made of VSS modules, as opposed to QuadVSS (the main novelty). Such a comparison was done in Fig 5, however it is not too clear how rigorous it is. It would be helpful to compare more directly VSS / QuadVSS equivalents with same FLOPS, for several different FLOPS thresholds. This does not seem to have been done - it would validate more strongly the fact that allocating parameters selectively to higher resolutions actually helps (as opposed to using the baseline VSS module pyramid). \n- Unclear what the training overhead of a QuadVSS block is compared to a VSS block (in terms of memory, compute)? \n- As opposed to just a table, it would be helpful to have plots with flops vs quality as axes, with a separate curve in that graph for each model family. Such a plot can make the comparison more obvious, given that different methods have somewhat different amounts of flops. \n- In Sec 4.3, it's unclear why for the Tiny model, we stack more blocks in the second stage, while for Small and Base we stack more in the third stage. Any particular reason? Third stage is more standard VSS modules, as opposed to your QuadVSS innovation. Also later in Table 6 yet different stackings are best (2,4,6,2). It's hard to discern the logic in all these choices. \n\n# Clarity and questions\nSome details of the approach were not particularly clear to me. \n- L175 Do you always pick the same fine-grained patch for all 2x2 windows, or do they vary by window? Not particularly clear from the exposition/notation. \n- Also, it appears that 7 patches are picked for each region as per L185-187. Don't you want to have 8 patches (powers of 2 are usually more efficient hardware-wise?) \n- The softmax in Eq 7, what labels do you train it on? Is it the final labels? Right now it appears that this part is trained first, and kept fixed. \n- (minor) In Eq 6, it seems 'v_local' is aggregated across the whole image, shouldn't it be called 'global' instead? \n- (minor) L186: where is 'local adjacency score' defined? This is the first mention of the term. \n- (minor) Fig 5: S-QuadVSS is not defined, I assume it's the opposite direction shift, but helps to be explicit. \n\n# Minor Nits and Typos \n159: infOrmative\nTable 1: VMamaba\n606: Uperhead [62] (I believe it should be UperNet)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1.Significance\n\n**Q1A1 Comparing to LocalMamba:** How to effectively preserve 2D spatial dependencies is an important challenge in adapting sequence models into the vision domain. Though it has been partially explored by previous works such as PlainMamba [1] and LocalMamba [2], it is non-trivial that our proposed QuadVSS is a **data-adaptive** and **light-weight** design for the locality module. \n\n- Data-adaptive:  LocalMamba opts for a differential neural network search during training for the optimal locality strategy for each layer. Once trained,  the architecture of LocalMamba is fixed. \nFor varied input images, LocalMamba used the same fixed locality strategy. In contrast, our QuadMamba has a learnable data-dependent locality strategy, which means it can dynamically and adaptively generate the optimal scanning sequences for different input data and various downstream tasks.  \n\n- Light-weight: LocalMamba, which requires handcrafting the network search space and extra optimization losses, brings much more complexity during training.  Differently, our method brings minimal additional complexity in terms of training and optimization, as the quadtree-based scanning module is lightweight. \n\n**Q1A2 Gains and improvement to existing Mamba:** We believe this is a misunderstanding, and the gain of our method over existing ones is actually significant. We apologize for the possible slight confusion in Table 1's main results, which is due to misalignments between how our model variants and others (like VMamba) are named. For instance, VMamba-T (9.1 GFLOPs) does not correspond to QuadMamba-S (5.5 GFLOPs) but QuadMamba-B (9.3 GFLOPs) instead. To make the comparison clearer, we highlight several comparisons under similar model sizes to show the advantages of QuadMamba. Our model variants (Lite/Tiny) show clear superiority on extreme light model levels (under 10M). Moreover, our small and base models achieve comparable and better results compared to methods of similar model sizes. It is also worth noting that our method of bringing locality into Mamba is generalized, practical, and easy to implement. compared to other complex methods. More details are found in the attached PDF. \n\n**Q1A3 Applicable to ViT architecture:** Our quadtree-based window partition module and sequence construction strategy are specially designed for the recent vision Mamba models. The idea of our QuadMamaba originated from the coarse-to-fine feature representation philosophy, found in many prior arts such as InceptionNet [3], Multiscale Transformer [4], Focal Transformer [5], and Quad-attention Transformer [6]. However,  the casual sequence modeling scheme in the Mamba model is completely different from the non-casual attention scheme in vision transformers. The constructed sequence for Mamba has to casually scan each token from the input. The length of the token sequence in Mamba has to be kept the same for multi-layer feature processing.  Thus, it brings much difficulty in preserving spatial adjacency in Mamba. Differently, due to the flexibility of the attention scheme, the quantity of the output tokens can easily maintained if the query tokens remain unchanged. Thus, our method is highly customized for Mamba.\n\n### Q2.Experimental results\n\n**Q2A1 More comparisons with the baseline:** To demonstrate the effectiveness of the proposed QuadVSS block, we will add additional ablation studies on different model sizes. The table below shows the improvement in tiny model levels/FLOPS thresholds. \n\n|Variant|Block|Params.(M)|Top-1 Acc.|\n|:------|:-------:|:------:|:------:|\n|Mamba-T|(2,6,2,2) |8.5 |76.9|\n|QuadMamba-T|(2,6,2,2)|10.2 |78.2|\n\n\n**Q2A2 Training overheads:** In the attached PDF files, we plot the GPU memory vs. Batch size and training curve. Our QuadMamba has an affordable GPU memory overhead compared to the baseline. We find that our method does not result in any difficulty in training convergence.  \n\n**Q2A3 Plots with Flops vs. Performance:** We plot the detailed comparison of Flops and Performance in the attached PDF file. To clearly show the advantages of our method, we rename the QuadMamba model variants (Lite -> B1, Tiny -> B2, Small -> B3, and Base -> B4). \n\n**Q2A4 Block numbers in different stages:** According to our observation,  QuadVSS blocks work better in early model stages, where the feature resolution is higher. However, stacking more blocks in the second stage (high resolution) brings significant computation overload, especially for high-resolution images. To strike a balance of FLOPs between different stages, we allocate more QuadVSS blocks in the third stage (low resolution) instead of the second stage for larger models such as QuadMamba-Small and -Base (channels=144). For smaller models like QuadMamba-Lite and -Tiny, allocating more QuadVSS blocks to the second stage is more affordable because of fewer number of channels (channels=48), so it is unnecessary to defer the QuadVSS blocks to the third stage.\n\n### Q3.Clarity and questions\n\n**Q3A1 Window hyper-parameter:** We always pick the 2x2 windows for two-level window partitions. Considering the 224x224 image resolution and four-time downsampling ratio, the two-level window partition strategy is enough for image classification. A more level window partition will be explored.\n\n\n**Q3A2 Patch hyper-parameter:** The image features, which are partitioned by learnable modules, are then reshaped to a 1D token sequence for the Mamba block. For the Mamba, the 1D token sequence is completely the same in terms of computation flows. Thus, the computing efficiency remains unchanged with different patch numbers. \n\n**Q3A3 Label of Eq.7:**  As the QuadMamba brings no extra training complexity, there is no training label for the softmax in Eq. 7. We design a differential sequence construction strategy and apply the Gumbel-Softmax trick, which can help optimize Eq.7.\n\n\n### Q4.Minor nits and typos\n\n We will revise the main text carefully.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N0tBEICNE5", "reviewer": "Reviewer_Kbfx", "summary": "Authors propose a technique to adapt Mamba to vision tasks. They propose a novel Quad-tree based approach instead of flattening tokens for images in a raster scan mechanism to avoid losing local dependencies. They evaluate on three vision tasks of object recognition, objet detection and instance & semantic segmentation.", "questions": "- Table 2 not clear what is the result for LocalVMamba and VMamba with S for fair comparison to QuadMammba-S", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting novelty and contribution designed specifically for Mamba type of models adapted to vision tasks.\n- Practically beneficial algorithm on the efficiency side where the show lower FLOPs and parameters with better or on par performance to SOA", "weaknesses": "- It provides a tradeoff though and it's not clear which is more favourable when compared to VMamba with corresponding T/S/B in Table 1. Meaning you can see when comparing VMamba-T (88.2) with QuadMamba-T (78.2) it is much lower, it is only at the base variant they become on-par but with reduced FLOPS/parameters on QuadMamba side. The fact it doesn't seem that on the lighter weight variants the performance is on par in fact it is much lower with around 10%.\n\n- What’s the reported inference time for Table 1 in addition to the FLOPs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1.Clear and fair model comparisons\n\nThanks for your valuable advice. We regret that our Li/T/S/B naming system may confuse readers compared with other methods. To clearly show our method's advantages, we rename the QuadMamba model variants (Lite -> B1, Tiny -> B2, Small -> B3, and Base -> B4). Our B1 and B2 variants are compared with efficient models, such as lite CNN models and efficient ViT models. Our B3 and B4 maintain similar model sizes as the tiny/small variants in transformer and vision Mamba backbone methods. Thus, our B1/B2 should be compared with those under 10M model parameters. Our B3 should be compared to tiny variants whose model parameters are around 30M, and our B4 to small variants around 50M. For instance, VMamba-T (9.1 GFLOPs) does not correspond to QuadMamba-S (5.5 GFLOPs) but QuadMamba-B (9.3 GFLOPs) instead. To make the comparison clearer, we highlight several comparisons under similar model sizes to show the advantages of the QuadMamba. Our model variants (Lite/Tiny) demonstrate clear superiority on extreme light model levels (under 10M). Moreover, our small and base models achieve comparable and better results compared to methods of similar model sizes. \n\nThe detailed plots and comparisons are found in the attached PDF file. We highlight several comparisons under fair conditions to show the advantages of QuadMamba:\n\n| Model | #Params.(M) | Top1 Acc|\n|:------:|:------:|:------:|\n|Vim-Ti| 7| 76.1 | \n|LocalVim-Ti| 8| 76.2 | \n|PVT| 13.2| 75.1 | \n|QuadMamba-B1 (Lite)| 5.4| 74.2 | \n|QuadMamba-B2 (Tiny)| 10 | 78.2 | \n|Vim-S| 22| 80.5 | \n|LocalVim-S| 28| 81.2 | \n|QuadMamba-B3 (Small)| 31 | 82.4 | \n\n\n### Q2.Reported inference time\n\nIn Table 9 of the supplementary materials, we benchmark the throughput of the QuadMamba model variants on an A800 GPU platform. The results are also shown below. Compared to the vanilla VMamba model, our QuadMamba model has negligible inference latency. Moreover, the throughput of our lite and tiny models is much higher compared to the efficient CNN and Transformer models, which implies better inference efficiency.\n\n| Model|Renamed| #Params.(M) | Flops(G) | Throughput(img/s)|\n|:------:|:-------:|:------:|:------:|:------:|\n|QuadMamba-Li|B1| 5.4 | 0.8 | 1754 |\n|QuadMamba-T|B2| 10.3 | 2.1 | 1586 |\n|QuadMamba-S|B3| 31.2 | 5.5 | 1252 |\n|QuadMamba-B|B4| 50.6 | 9.3 | 582 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o0lCYJDJsa", "reviewer": "Reviewer_ALic", "summary": "This paper proposes a vision mamba backbone for various vision tasks. It aims to adapt the recent popular mamba model originated from the language domain to vision tasks. The authors propose a learnable quad-tree partition strategy which can adaptively generate multi-scale visual sequences with spatial prior for mamba model. The authors also make contributions on differentiable training and shifted token interactions. Experimental results show the effectiveness of the proposed vision mamba, including image classification, object detection and segmentation. It shows the potential to be applied in various vision mamba models. Ablation studies are also extensively conducted.", "questions": "I have listed some questions in the previous section.", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Strong Motivation. This paper is well-motivated and important. Effectively adapting the Mamba model to vision tasks has gained wide attention recently. Multi-grained image information and 2D priors are essential to building a vision backbone, which has led to a strong motivation for this paper. Treating the 2D image as token sequences in the language domain needs exploration and adaptation. \n\n- Novel \\& applicable method.  The quadtree-based window partition is novel in vision Mamba. The learnable module part is lightweight and conceptually simple. The quadtree search technique is seen in other tasks, such as the point cloud domain. This paper only learns to partition the semantic-rich content with a handcrafted partition strategy, which avoids developing a search algorithm that is too complex. The other two points, the differentiable trick and shifted scheme, are also interesting. It has the potential to be applied to constructing token sequences for other vision mamba models.  \n\n- Good experimental results. The method proposed is tested in image classification, detection, and segmentation tasks, which proves its effectiveness over other vision mamba models. The ablation studies and visualization results also give more explanations about the quadtree-based sequence construction. \n\n- Well-written presentation.  The writing is clear and precise. In general, the presentation of this work is easy to follow.\n\nThe experiments are well performed and the presented method compares favorably to the considered baselines.", "weaknesses": "I have a number of suggestions and questions that may help to further improve the paper:\n\n- It would be helpful to demonstrate the impact of the hyper-parameters in the model design. For example, is it helpful to partition more window levels?\n\n- More illustrations and analysis are needed for the learnable parameters. It can show the relationship between model cost and performance gain to other researchers/readers. \n\n- It would be useful to give the details about the training settings and compare them to other vision mamba works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1.More illustrations of hyper-parameters\nThanks for your valuable advice. We will add more analysis on the hyper-parameters. The impacts of the increased number of window partition levels are explored in Table 5 of the main text. We examine the choice of partition resolutions in the bi-level quadtree-based partition strategy. Experimentally we deduce the optimal resolutions to be {1/2, 1/4} for the coarse- and fine-level window partition, respectively. It is worth noting that the design space is handcrafted and may be extended to more levels. We will explore a more flexible scheme in the future. \n\n### Q2.More analysis of learnable parameters\n\nThanks for your insightful opinion. We will include more analysis on the learnable parameters in the revised manuscript. To show the relationship between model complexity and performance, we plot the performance against flops for different methods and models in Figure 2 of the attached PDF file. We also conduct an ablation experiment on the local and global context embedding for the prediction module. The method of combining both the local and global contexts outperforms the one using only the global context vector. \n\n| Variant| Embedding | Top-1 Acc.(%) |\n|:------|:-------:|------:|\n| A | Global | 74.0 |\n| B | Global+Local| 74.2 |\n\n### Q3.More details about training settings\n\nWe will add more illustrations of training details in the supplementary materials. To compare fairly with previous vision Mamba methods, our training settings strictly align with VMamba, whose training configurations were inherited from Swin Transformer. The training details for image classification are as follows: QuadMamba models are trained from scratch for 300 epochs, with a 20-epoch warm-up period, using a batch size of 1024. The training process utilizes the AdamW optimizer with betas set to (0.9, 0.999), a momentum of 0.9, an initial learning rate of $1\\times 10^{-3}$, a weight decay of 0.05, and a cosine decay learning rate scheduler. Additional techniques such as label smoothing (0.1) and EMA (decay ratio of 0.9999) are also applied. No other training techniques are employed.\n\nFor object detection and semantic segmentation, our method also strictly follows VMamba and Swin Transformer. The training details are as follows: Object detection and instance segmentation experiments are conducted on COCO 2017, which contains 118K training, 5K validation and 20K test-dev images. An ablation study is performed using the validation set, and a system-level comparison is reported on test-dev. For the ablation study, we consider four typical object detection frameworks: Cascade Mask R-CNN, ATSS, RepPoints v2, and Sparse RCNN in mmdetection. We utilize the same settings as in previous works: multi-scale training (resizing the input such that the shorter side is between 480 and 800 while the longer side is at most 1333), AdamW optimizer (initial learning rate of 0.0001, weight decay of 0.05, and batch size of 16), and $3 \\times$ schedule (36 epochs).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YyJhE5Lhkl", "reviewer": "Reviewer_VDbd", "summary": "This paper presents QuadMamba, a novel Mamba architecture for visual tasks such as image classification and dense predictions. Unlike the classic Vision Mamba, which splits 2D visual data using fixed windows, the authors introduce learnable windows by using a lightweight module that predicts 2D locality more informatively, allowing for the ignoring of irrelevant windows and further splitting of the most informative windows into sub-windows, capturing more fine-grained information. This coarse-to-fine fashion is made possible by the new implementation of a splitting operator with Hadamard product and element-wise summation, making the pipeline fully differentiable. The authors experiment on the classic benchmarks for image classification (ImageNet), object detection (COCO), and semantic segmentation (ADE20K), showing that the method achieves state-of-the-art results.", "questions": "As mentioned in weakness section, I have only two questions related to lightweight module, extreme case of splitting all quadrants to sub-quadrants in each layer. Besides, it would be great if the authors can provide some experiments with more than two levels partition as mentioned in conclusion, for example, by using the lite model on semantic segmentation tasks should be enough. I would be very helpful to get some insights for possible future works.", "limitations": "Yes, the authors explicit mentioned the limitation which is that the partition with more than two levels is yet to be explored.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- First, the paper is very well written and most of the figures are done very well so that it is easy to fully understand the story of paper.\n\n- I also like the idea coarse-to-fine scanning, to capture more relevant information in each layer in QuadMamba. It is also worth noting that as the authors mentioned, direct sampling from 2D visual data based on index is not differentiable, but the authors has proposed a solution to overcome it which I consider is a strong contribution.\n\n- The experiment shows that QuadMamba always outperforms the classic CNN, Vision Transformer on popular tasks (image classification, object detection, image segmentation).", "weaknesses": "I do not see any major weakness for this paper but only have a few questions for better understanding the paper and improve its clarify:\n\n- The lightweight module to predict 2D locality, informative windows, is it shared across layers or specific to each layer? It seems this module is shared across layers but I feel it needs to be different as each layer capture different information.\n\n- As the authors mentioned in limitation, in remote sensing images which might be helpful to have more than two levels partition in QuadMamba. However, I feel this experiment can still be experimented with the current architecture. In this case, what can happen is that the lightweight module will split into sub-quadrants for all windows ( since all of them are mostly capturing relevant for predicting the output). Did the authors observe this  extreme case to subplot all quadrants into sub-quadrants can happen with the current architecture?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1.Details about the lightweight prediction module\n\nThanks for your valuable advice. We will depict the prediction module more clearly in the revised manuscript. In our design, each QuadVSS block has its specific prediction module that determines the informative regions of the current layer. The potential reason is that each layer needs to attend to different regions as the feature resolution and network depth change, given that features of different depths and resolutions are known to have different responce patterns in terms of regions and context. The shift scheme also helps the model select the informative regions more flexibly, which can enhance the feature representation quality. \n\n### Q2.The effects of more window-level partitions\n\nThanks for your thoughtful question. The downstream tasks (detection and segmentation) need the pre-training weight of the ImageNet classification. Our two-level window partition strategy mainly considers the input image resolution (224x224) of the image classification task. The input image resolution in dense prediction tasks is generally higher than that in the image classification task. Thus, our module design may not be optimal for downstream dense prediction tasks involving very high resolutions. We will further improve this window partition scheme in future work. A similar phenomenon is observed in Table 4 of the main text: A moderate local window size (14x14) perform better than overly-small (2x2) and overly-large (28x28) window sizes. \n\nMoreover, we conduct a simple experiment in semantic segmentation to explore the proposed question. We split the image into sub-quadrants for all windows using fixed local window sizes.  The size is increased four times in the table below.  It is worth noting that semantic segmentation results are highly related to the pre-training weights from the ImageNet Classification. Since we do not train each model variant on ImageNet classification, the results may not truly reflect the effects of window size in the segmentation experiment.  In the table below, the 8x8 window setting outperforms the 1x1 setting. This indicates that more window partitions can improve the segmentation performance in high-resolution inputs. It is hard to find the optimal local window size for a specific downstream task. The quadtree variant without pertaining achieves the results in between the 8x8 setting and 32x32 setting. It indicates that the two-level quadtree window partition setting may not be flexible enough to handle the high-resolution downstream task.  It is because we designed our hyper-parameter by mainly considering the image classification task (lower image resolution).  Thus, we will be devoted to designing more flexible window partition schemes to handle various downstream tasks in the future. \n\n\n| Model | Pre-training| Local Window | mIoU |\n|:------|:-------:|:-------:|------:|\n| Tiny (B2) | Yes|Quadtree | 44.3 |\n| Tiny (B2) | None|Quadtree | 38.0 |\n| Tiny (B2) | None|w/o (1x1) | 36.8 |\n|  Tiny (B2)| None|8x8 | 38. 2|\n|  Tiny (B2)| None|32x32 | 37.9 |\n|  Tiny (B2)| None|128x128 | 36. 9|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, 

We thank the reviewers for the positive reviews of our work and constructive comments. Here is a list of  new figures and tables in the attached PDF file, and references referred to in other responses.

**Figures and tables** :

Figure 1. Plots of performance, mode size, and FLOPs in ImageNet classification.

Figure 2. Plots of performance, mode size, and FLOPs in COCO detection and ADE20K segmentation.

Figure 3. Training curve and GPU memory consumption of our method. 

Table 1. Fair Comparisons in ImageNet classification. 

**References** :

[1]  PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition. Chenhongyi Yang, etc. BMVC2024

[2] LocalMamba: Visual State Space Model with Windowed Selective Scan. Huang, etc. ArXiv2024. 

[3] Going deeper with convolutions. Christian Szegedy, etc. Neurips2014

[4] Multiscale Vision Transformers. Haoqi Fan, etc. CVPR2021

[5] Focal Self-attention for Local-Global Interactions in Vision Transformers. Jianwei Yang, etc. Neurips2021

[6] QuadTree Attention for Vision Transformers. Shitao Tang, etc. ICLR2022

Meta-review: The manuscript introduces a novel state space architecture for computer vision. The main novelty with respect to numerous previous attempts [26, 41, 49, 66, 80] (all only in Arxiv) corresponds to dynamic multi-resolution reasoning through learnable quadtree-based scanning. The reported performance is competitive with respect to prior state-space, convolutional and transformer models. Still, the overall per-MAC improvement is marginal and incremental at best.
Decision: Accept (poster)