Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection
Authors: Guowen Zhang
Lue Fan
Chenhang HE
Zhen Lei
Zhaoxiang Zhang
Lei Zhang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Point Cloud based 3D Object Detection
State Space Models
Group-free
Primary area: machine_vision
TL;DR: A simple and effective 3D backbone for point cloud based 3D object detection.

Abstract: Serialization-based methods, which serialize the 3D voxels and group them into multiple sequences before inputting to Transformers, have demonstrated their effectiveness in 3D object detection. However, serializing 3D voxels into 1D sequences will inevitably sacrifice the voxel spatial proximity. Such an issue is hard to be addressed by enlarging the group size with existing serialization-based methods due to the quadratic complexity of Transformers with feature sizes. Inspired by the recent advances of state space models (SSMs), we present a Voxel SSM, termed as Voxel Mamba, which employs a group-free strategy to serialize the whole space of voxels into a single sequence. The linear complexity of SSMs encourages our group-free design, alleviating the loss of spatial proximity of voxels. To further enhance the spatial proximity, we propose a Dual-scale SSM Block to establish a hierarchical structure, enabling a larger receptive field in the 1D serialization curve, as well as more complete local regions in 3D space. Moreover, we implicitly apply window partition under the group-free framework by positional encoding, which further enhances spatial proximity by encoding voxel positional information. Our experiments on Waymo Open Dataset and nuScenes dataset show that Voxel Mamba not only achieves higher accuracy than state-of-the-art methods, but also demonstrates significant advantages in computational efficiency. The source code is available at https://github.com/gwenzhang/Voxel-Mamba.

Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection
Guowen Zhang Lue Fan Chenhang He', Zhen L
Zhaoxiang Zhang".3... Lei Zhang
'he Hong Kong Polytechnic University
CCntre for Artificial Intelligence and Robotics, HKISI, CAS
Innttitute of Automation, Chinese Academy of Sciences "Schoo of Artificial Intelligence, University of Chinese Academy of Sciences guowen wwnnenonnennnnn......om.m polyu.hk {cache, cslzhang/@com polyu edu.hk cn
{lue. fan, zlei, zhaoxiang zhang (hhang@ia... @iia.aiiaaa..a aaia.aac.omm ac
Abstract
iitalizzationbbaee methods, which serialize the 3D voxels and group them into multiple sequences before inputting to Transformers, have demonstrated their effectiveness in 3D object detection. However, serializing 3D voxels into ID sequences will inevitably sacrifice the voxel spatial proximity. Such an issue is hard to be addressed by enlarging the group size with existing serialization- based methods due to the quadratic complexity of Transformers with feature sizes. Inspired by the recent advances of state space models (SSMs), present Voxel SSM, termed as Voxel Mamba, which employs a group-free strategy to serialize the whole space of voxels into single sequence. The linear complexity of SSMs encourages our group-free design, alleviating the loss spatial proximity of voxels. To further enhance the spatial proximity, we propose Dual-scale SSM Block to establish a hierarchical structure, enabling larger receptive field in the ID serialization curve, well as more complete local regions in 3D space. Moreover, we implicitly apply window partition under the group-free framework by positional encoding, which further enhances spatial proximity by encoding voxel positional information. Our experiments on Waymo Open Dataset and nuScenes dataset show that Voxel Mamba not only achieves higher accuracy than -att--of-the-art methods, but also demonstrates significant advantages in computational efficiency. Voxel-Mambal
The source code is available at httpss//gthhub.oo///we//iit
1 Introduction
LiDAR-based 3D object detection from point clouds plays an important role in applications of
autonomous driving 199 virtual reality [47], and robots [5 The sparsely, unevenly and irregularly distributed point cloud data make the efficient and effective 3D object detection a very challenging
task. To address these standiin challenges, researchers have recently proposed several strategies to improve the model architecture. One strategy is to switch from PointNet-basee models 49 79 [6]
to sparse convolutional neural network (SpCNN)-based models [70
27 in order for more effective feature extraction. However, the sparse convolution is unfriendly for deployment and optimization, requiring tremendous engineering efforts. Therefore, another strategy is to switch from
10
SpCNN to iialiization-baseee Transformers to address this issue [13 65| 38 69 42). These methods
usually group non-empn non-empty 3D voxels into multiple short sequences by serialization techniques such as
Coorrssponiing Authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Sbún
Voxel Sets
Voxel Sets
One Single Sequence
(a) blaee Grouping
(b) Curve-based Grouping
(c) Voxel Mamba
Figure 1: Comparison between (a) window- baaded grouping, (b) curve-based grouping, and (c)
proposed single group modeling by Voxel Mamba.
window partition 65 and (b), where sequence is group of voxels to be processed by Transformer layers.
Z-shape sorting [66], and Hilbert sorting [69]. as shown in Figs.
However, the serialization of voxels will inevitably sacrifice their spatial proximity. Some neighboring voxels can be far apart from each other after serialization, as illustrated by the two red points in Fig. (b). Such loss of proximity is difficult to be addressed in the existing serialization 13 because the group size is limited by the quadratic complexity of Transformers. This issue becomes even worse when neighboring voxels are grouped into different
methods 69 66
38 65
groups. Inspired by the recent success of State Space Models (SSMs) [2 20 language and vision, in this work we propose simple yet effective group-fres free design to address the loss of proximity. Specifically, we introduce a Voxel SSM, termed as Voxel Mamba, for 3D object detection from point cloud. The linear computational complexity SSMs makes feasible to treat all voxels as single group and sort them into single sequence. This results in a group-free modeling voxels, which is more efficient and deployment- seeen-fri than previous methods since no padding tokens are needed. Nonetheless, even we can sort all voxels into a group-free sequence, it cannot be ensured that all them are within an effective receptive field.
59
18
82
36) in
To enhance the spatial proximity of Voxel Mamba, we further propose two modules with it. The first is the Dual-scale SSM Block (DSB) by introducing the downsampling operations SSMs. In specific, the forward SSM branches process the high-resolution voxel features, while the backward branches extract features from the low-resolution representation. In this way, we integrate the hierarchical design with the bidirectional design in a more economical way. More importantly, the hierarchy brings larger effective receptive field the serialized sequence so that the spatial proximity in local 3D regions can be enhanced. The second module we introduced is the Implicit Window Partition (IWP). The window partition widely used strategy in previous methods [K] [5] to enhance the proximity of voxels inside window. However, it impedes the proximity of voxels across windows and contradicts with our group-free principle. We therefore propose an implicit window partition scheme embrace its strengths while discarding its weaknesses. In specific, encode the voxel positions inside and across windows into embeddings for feature lcarning without explicitly conducting spatial window partition. In this way, better voxel proximity be achieved under group-free design with minimal computational cost.
Our contributions are summarized follows:
We propose Voxel Mamba, a group-free backbone for voxel-based detection. Voxel Mamba abandons the grouping operation and serializes voxels into one single sequence, enabling better efficiency.
To mitigate the loss of spatial proximity due to serialization, we propose the Dual-scale SSM Block (DSB) and the Implicit Window Partition (IWP) to enhance the spatial proximity preservation of Voxel Mamba.
Our method achieves superior performance to previous state-of-the-ar methods on the large-scale Waymo Open dataset [60] and nuScenes 2 datasets.

---

2 Related Work
3D Object Detection from Point Clouds. There are two major point cloud representations for 3D object detection, i.e. point-based and voxel-based ones. As in PointNet [50 511 point-based methods [49 48 56
52 37 directly extract geometric features from small regions of raw points. However, those methods suffer from low inference efficiency and limited context features. Voxel-
based methods [34 76 70 27 79, [5/17] convert raw points into regular grids through voxelization and then process them with sparse convolution [70] or Transformers [13] 25 65]. Voxel-based
methods are currently the main stream for 3D object detection. In terms of model architecture, voxel-based methods can be categorized into two groups, i.e., SpCNN-based 770 79 55 g I0]
38 (6 ones. Limited by the high computation complexity, SpCNN-based methods can only use small convolution kernels with restricted receptive fields, and Transformer based methods can only employ a small number of voxels in each group. In contrast, our proposed Voxel Mamba can capture long-range dependencies within the entire sequence while achieving faster inference speed than existing state-of-the-art methods.
and Transformers- [65 13 25
State Space Models. Inspired by the continuous state space models (SSMs) in control systems, researchers [18] [1]22]]88 59] [0] have introduced the SSMs into deep neural networks as a novel alternative transformation, demonstrating the potential of SSMs. S4 [21] introduces a new parameterization method to SSMs to reduce the computation and memory cost. S5 (59] employs MIMO SSMs and perform efficient parallel scans based on S4. More recently, Mamba (0) introduces input-dependent SSMs and builds a generic backbone, which is fairly competitive with the well-tuned Transformers. Vision Mamba [82) employs bidirectional SSMs and position embedding to learn global visual context for vision tasks. Vmamba (36] employs 2Dsselective-scan bridge the gap between 1D scanning and 2D plain traversing. PointMamba [4 is a pioneering work to leverage SSM for point cloud analysis, achieving impressive performance point cloud object understanding. Subsequently, many SSM-based methods [24 78 77 68) are introduced for point cloud processing. In this paper, we investigate the utilization of SSMs to establish straightforward yet robust baseline for LiDAR-based 3D object detection in driving scenes.
to CNNs and Transformers. LSSLs [22] adopts a simple eequence-to-sequee
Space-filling Curve The space-filling curve 4 is series of fractal curves that can go through each point in a multi- dinli--dmeenionall space without repetition. The classical space-filling curve includes Hilbert curve 22), Z-order curve [46), and sweep curve, etc. Those methods can perform dimension reduction while maintaining spatial topology and locality. Many researchers [5] 69
66 38
65]
3 40)
have introduced space-filling curves for point cloud processing- HilbertNet (5) uses the Hilbert curve to collapse 3D structures into 2D space to reduce computation and GPU occupation. PointGPT B
utilizes the Morton- curve 44] to introduce sequential properties. OctFormer (66) preserves Z-order during octreelization and adopts octrec-attention for efficient context learning. PTV3 [69] streamlines the complex interaction with the space-filligg fillinni curve serialization. For 3D object detection, some methods 665 38 employ window sweep curves to group voxel features for parallel computation. We employ the Hilbert curve due to its advantageous characteristic of locality preservation.
Point Cloud Grouping. LiDAR point clouds are sparsely and non-uniformly distributed with varying densities. Therefore, existing methods group points or voxels to facilitate parallel computation and reduce complexity. In point cloud analysis, some works [51] 67
use the K nearest neighbor (KNN) method to create groups of query points. However, the heavy computation burden makes KNN hard to scale for outdoor scenes. For 3D object detection, VoTr 42) uses a GPU-based hash table to search neighborhoods and generate fixed-length voxel groups. Window-based Voxel Transformers 13 group voxels by employing a window-bar based sorting strategy, such as the rotating partition. To reduce the reliance on relative position in grouping operations, some recent works [66] 69
65
38
have been proposed to group voxels based on space-filling curves. However, grouping is merely a compromise for computational complexity, which restricts the flow of information and effective receptive field. To tackle this problem, we model the entire voxels into one single sequence and allow each voxel aware of global context information.
3 Methods
In this section, we present Voxel Mamba, group-free Voxel State Space Model-hased 3D backbone that can be applied to most voxel-based 3D detectors. We first introduce the preliminary concepts
3

---

DSB
DSB
Vousitaation
Mamba
Ditecton Beed
Input point clouds
Porward SSM
Forward SSM
Output boxes
Radkward SSM
Up
Down
Badkward SS3
BEN
di
Hilbert Input Layer
Linear
Non-umply Voxels
Empty Vinals
Upsanapling
LayerNorm
LayerNorm
Down Sampling
SSID
Backward SSM
Sit
Situ
Rilbert Iupui
Conv
Hilbert Ieputt I ayer
Dowzsampling
L Inver
Dual-rrale SSMs Block (DSB)
Forward SSM Inpat Sequence
Backward SSM Input Sequence
State Space Model (SSM)
Figure 2: Top: The overall architecture of our proposed Voxel Mamba with N Dual-scale SSM Blocks (DSBs). Bottom: Illustration of the DSB, including a residual connection, a forward SSM branch, and a backward SSM branch.
associated with our method, followed by overall architecture of Voxel Mamba. Then, we describe in detail the fundamental components of Voxel Mamba, including the Hilbert Input Layer (HIL., Dual-scale SSM Block (DSB), and Implicit Window Partition (IWP).
3.1 Preliminaries
The state space sequence (SSM) model is continuous-time latent state model, which maps ID input signal srt) E R² to an output signal ytt) E R² through hidden state h(t) E R The system can be represented the following linear ordinary differential equation:
(K((0 = Ah(t) + Br(t), v(() Ch(() Dr((t,,
where A NNN, E Nx1 and C g learnable parameters, and IR_ denotes residual connection.
To apply SSM discrete sequence,
discrete the continuous-time SSM with timescale parameter A 22 20 36). The zero-order hold (ZOH) transformation can be used to discrete the con- tinuous parameters A, B A exp( A), ) B (AA) -1 (expp(AA-ai I)- AA. The discretized
version of Eq. 11 can be written in the following recurrent form:
h& A Br
(2)
Vk
Chyy + Dr
Finally, the convolutional mode can be used for efficient parallel training:
K = (CB, CAAB, CA CA"B),
(3)
where L is the length of the input sequence and K E R is the structured convolution kernel.
SSM combines the advantages of convolution and self-attention with near-linear computation and dynamic weights. It demonstrates stronger ability than Transformers in modeling long-range de- pendencies 21 20) which inspires us to develop group-free framework for point cloud based object detection.

---

3.2 Overall Architecture
An overview of our proposed Voxel Mamba is shown in Figure 2 As in previous works 65 Voxel Mamba transforms point clouds into sparse voxels by voxel feature encoding strategy. Unlike prior Transformer- based methods that perform extensive window partitioning and voxel grouping, in Voxel Mamba we serialize the voxel of the entire scene into single sequence by using the Hilbert Input Layer (Sec. 3.3). Then, a Dual-scale SSM Block (Sec. 3.4) working on the voxel sequence is proposed, which allows voxels to be processed with global context. To enlarge the effective receptive fields, DSB adopts finer-grained perception of the voxel sequence the forward path, and down-samples the voxels sequence in the backward path. The backward path extracts features from the low-resolution BEV representation, with an increased downsampling factor deeper blocks. To enhance the spatial proximity in sequences, Voxel Mamba adopts Implicit Window Partition (Sec. 3.5 to preserve 3D positional information in the extracted voxel features, and projects them to a BEV feature map. Our proposed architecture is flexible and can be applied to most existing 3D object detection frameworks.
74
31
3.3 Hilbert Input Layer
The space-filling curve (e.g,, Hilbert 129 and Z-order [46), known for preserving spatial locality, is widely used for dimensionality reduction. Space-fillingg curves, such as the Hilbert shown in Fig. 2 can traverse all elements in space without repetition and preserve spatial topology. To improve the voxel proximity in serialization, we propose the Hilbert Input Layer to reorder the voxel sequence.
Denote the coordinates of voxel features as C = {(x, v)=) E R I I, D, S n). We map a voxel onto its traversal position à within the Hilbert curve. Specifically, we transform (I, y. z) into its binary format with log2n bits. For example, x converted to (Fm.mm-1....)
where m = [log2n] Then, following [58], we iterate from I' Im 2m to Iτ, 1/1, 21 bits and perform exchanges and inversions to adjust the order of bits. An exchange is conducted when the current bit is O; otherwise, an invert conducted. We concatenate all bits as (Im (m 2m=mm 111--111111111------- 7m-1 ................................ To3 and apply a global 3yn-fold Gray decoding (58) on it to obtain the traversal position h. Subsequently, all voxels are sorted into single sequence based on their traversal position h.
In our implementation, we record the traversal position corresponding to the coordinates of all potential voxels. The voxels serialized by querying and sorting their traversal positions. We employ a distinct traversal order for each BEV resolution in Dual-scale SSM blocks. Notably, the serialization process only takes approximately 0.7ms for sequence of length 10
3.4 Dual-scale SSM Block
Though space-fillingg curves can preserve the 3D structure certain degree, proximity loss is inevitable due to the dimension collapse from 3D to ID. As result, local snippet of the curve can only cover a partial region 3D space. discussed Sec. I placing all voxels in a single group cannot ensure that the effective receptive field (ERF) 12 could cover all voxels. Therefore, in this subsection we introduce the Dual-scale SSM block (DSB) to build a hierarchy of state space structures and consequently improve the ERF of the model.
forward SSM branch As shown in Fig. 2 the DSB block designed with a residual connection 23 and backward SSM branch. It operates on two serialized voxel sequences generated by the Hilbert Input Layer, enabling a seamless flow of information throughout the voxel sequence. The forward branch processes the original voxel sequence, maintaining high-resolution details. The backward branch, however, operates on a sample voxel sequence derived from low-resolution BEV representation. This dual-scale path allows DSB to incorporate larger-scale voxel features, enhancing the model's ability to model long dependencies among voxels. Specifically, given a voxel sequence F and its corresponding coordinates C, DSB is computed as:
F, = LN(FSSM(HILLFF + IWE(C))))
Fg UDLLN
+ IWE(C'))))
F = F
where HIL(.) represents the Hilbert Input Layer, FSSM(.) and BSSM(-) denote the forward and backward SSM. LN(-) stands for Layer Normalization, and C' is the coordinates of downsampled
5

---

sparse voxels. Besides, (-wn(-) and Up(-) refer to the downsampling and upsampling operations, respectively, and (-) means Implicit Window Embedding. Overall, DSB integrates the widely adopted bidirectional design 82 36) with the hierarchical design, building sufficient receptive field to mitigate the loss of proximity without introducing additional parameters.
3.5 Implicit Window Partition
The window partition strategy is widely used in previous 3D detectors 13 65 to enhance the voxel proximity. In these methods, the whole field is partitioned into multiple local windows and the voxels within a window form a group. Therefore, the voxels inside window will have sufficient proximity; however, the voxels in different windows will have minimal proximity. this section, we aim to introduce the advantages of window partition into framework while avoiding its weaknesses.
To fulfill our goal, we propose an Implicit Window Partition (IWP) strategy. Unlike previous methods, we do not explicitly partition voxels into windows and apply Transformer or SSM within each window. In contrast, we calculate the voxel coordinates inside and across windows, and then encode coordinates to embeddings, termed as Implicit Window Embedding (IWE), which is formulated as:
yy h
IWE MLP (2,
1, mod w, y² mod h)), i 0,
(5)
where [:] is the floor function, w, h define the window shape, and 2, x. ,y are the coordinates of tokens. (x", y and (x*,y") represent the coordinates before and after an implicit window shift. The IWE is shared across all layers with the same stride. Thus, its computation cost only comes from shallow MLPs. With IWE, voxels in the serialized ID curve are aware of their positions and consequently their proximity in 3D space.
3.6 The Voxel Mamba Backbone
With the proposed Hilbert Input Layer, DSB and IWP strategies, we build Voxel Mamba, group-free sparse voxel backbone. The architecture of Voxel Mamba is illustrated in Figure 2 It comprises N DSB blocks, which are organized into different stages based on their downsampling rates. SpConv [9] is employed to progressively decrease the feature map resolution along the Z-axis in each stage. Before sparse tokens are fed into the BEV backbone, we scatter them into dense BEV features. On the Waymo dataset, we adopt the BEV backbone from Centerpoint-Pille 774, and employ the same setting as DSVT [65] for the detection head and loss function. On the nuScenes dataset, we only replace the 3D backbone of DSVT 655 with our Voxel Mamba backbone.
4 Experiments
4.1 Datasets and Evaluation Metrics
Waymo Open Dataset contains 230k annotated samples, partitioned into 160k for training, 40k for validation and 30k for testing. Each frame covers large perception range (150m x 150m). The mean average precision (mAP) and its weighted variant by heading accuracy (mAPH) are used as evaluation metrics. They are further categorized into Level I for objects detected by over five points, and Level 2 for those detected with at least one point.
nuScenes consists of 40k labeled samples, with 28k for training, 6k for validation and 6k for testing. For 3D object detection, nuScenes employs the mean average precision (mAP) and the nuScenes detection score (NDS) to measure model performance.
4.2 Implementation Details
Our method is implemented based on the open-source framework OpenPCDet 63 sizes are defined as (0.32m, 0.32m, 0.1875m) for Waymo and (0.3m, 0.3m, 0.2m) for nuScenes. We stack six DSB blocks, divided into three stages, for the Voxel Mamba backbone network. The downsampling rates for DSBs' backward branches in each stage are 21,2,4). Specifically, we employ SpConv [9] and its counterpart SpInverseConv as downsampling and upsampling operators in the
The voxel
DSB backward branch. On the Waymo dataset, we follow the training schemes in [65] [4] to optimize

---

Table 1: Performance comparison the validation set of Waymo Open Dataset (single-frame setting). Symbol means that the result is not available.
ALL ALD mAPH)
Vehiole (AP/APH)
Podestrian (AP/APH)
Cyclist (AP/APH)
Method
Categury
L1
1.2
L1
L2
L1
L2
L1
L.2
PointPillar 31
63.3
57.5
71.671.0
/62.5
70.6 56.7
62.9/30.2
64.4/62.3
61.9/599
Centerpoiat Pillar
2D CNN
76.1
68.0/67.5
-/62.6
-/67.6
7841 77.9
70.3 69.8
82.5/77.1
74.9/69.8
73.2/ 12.2
70.6/
SECOND 80
63.1
57.2
72.3/717
68.7/58.2 63,9/63.3
60.7/51.3
60,6/ 59.3
58.3/57.1
Par-A2
75.2/66.9
66.2/58.6
66.1
PV-RCNN
S
69.6
63.3
77.5
69.01 68.4
75.01 65.7
66.01
67.8/66.4
65.4764.0
Centerpoint- Voxel
67.6
76.61 76.0
68.9
73.4
75.0/6588
72.1/71.0
09.5/68.5
PV-RCNN++
75.2
68.6
79.1 78.6
69.3/69.9
74.6
76.9/6.3
73.5
70.7
SpCNN
74.8
68.8
77.6
69.7
80,2/74.6
72.2/6700
73.7/72.7
71.0/ 70.1
VoxeINeXt
77.2
69.9/69.4
72.3/722
HEDNet [6]
79.4
73.4
81.1/806
73.2/
84,4180.0
76.6/726
78.7
75.8/749
PillarNet
74.6
68.4
79.17 78.6
70.5
80.6/7400
72.3/662
72.3/712
69.7168.7
FSD [4
77.3
70.8
79.2 78.8
70.5 70.1
82.6/77.3
73.9/69.1
77.1
74./733.
ConQueR 81
77.9
71.6
78.47 77.9
71.0/ 70.5
82.4/ 76.6
75.8/ 70.1
71.5/664
75.2/74.1
VeTR 42
75.0 74.3
65.9/65.3
VoxSeT [5]
72.2
66.2
74.5 74.0
66.0/ 65.6
80.0 72.4
72.5
71.6/
69.0/67.7
13
SWFormer 61
77.8 77.3
69.21 68.8
80.9/72.7
72.5/64.9
CenterFormee 80
Group based
73.2
69.1
75.0 74.4
69.9/69.4
78.0/72.4
73.1/67.7
73.8/727
71.3/70.2
FlatFormer
67.2
69.0 68.6
71.5/65.
68.6/67.5
PTv3 60
70.5
71.2./ 20.8
76.3/ 70.4
71.5/70.4
DSVT-Voxel
79.7 79.3
83.7/78.9
77.576.5
74.6/
Voxel Mamba (ours)
free
79.6
73.6
80.8 80.3
72.6 72.2
85.0 80.8
77.7/736
78.61 77.6 75.7/748
Table 2: Performance comparison on the test set of Waymo Open Dataset, Symbol result is not available. "3f" stands for 3-frame model.
means that the
ALL (3D mAPH)
Vehicle (AP APH)
Pedestrian (AP APH)
Cyclist APH) LI L2
Method
L1
L2
L1
L2
LI
L2
PointPillar 31
68.6/ 68.1
60.5 /0.11
68.0 55,5
61.4/ 50.1
M3DETR 23
67.1
61.9
77.7/77.1
70.5
58.5
60.6 52.0
67.3/65.7
65.3 63.8
3D-MAN 73
78.7
70.4/70.0
64.0/60.3 70.0/66.0
PV-RCNN+ 15
75.7
70.2
81.6/8122
73.9/73.5
80.4/75.0
79.1/
71.9
69.3 68.2
CenterPoint 74
77.2
71.9
81.1 80.6
73.4/73.0
80.5
74.6/71.5
73.7
7
RSN 62
80.7 80.3
71.9/71.6
78.9
7
SST-3r 133
78.3
72.8
81.0/ 80.6
73.1/72.7
83.3
76.9/73.5
73.2 74.6
Graph-RCNN RCNN 71
77.0
71.6
83.6/83.1
76.0
81.9/
75.6/70.55
72.5/71.3
69.8 68.7
FSDv1 14
78.2
72.4
82.7/82.3
74.4/44.1
87.9
75.9
75.6/
7
FSDv2 15
79.0
73.3
82.4 82.0
74.4/74.0
83.8/78.9
77.4
7 77.1/76.0
PillarNeXt-3f
+2
79.0
83.3 82.8
76.2/75.8
84.4/81.4
78.8
76.0
73.8/72.7
71.6/
Voxel Mamba (burs)
79.6
74.3
84.4 84.0
77.6
84.8 80.6
79.0 74.9
75.4/74.3
72.6/71.5
the model using Adam optimizer with weight decay 0.05, one-cycle learning rate policy, learning rate 0.0025, and batch size 24 for 24 epochs. On the nuScenes dataset, we follow the training scheme adopted DSVT (65]. We train Voxel Mamba with a weight decay of 0.05, one-cycle learning rate policy, max learning rate of 0.004, and batch size of 32 for 20 epochs. The voxel features on both datasets consist of 128 channels. All the models are trained in the training and inference of Voxel Mamba follow DSVT 65
RTX A6000 GPUs. Other settings
4.3 Comparison with State-of-the-art Methods
Waymo. We first compare Voxel Mamba with state-of-the-ar methods on the Waymo Open dataset. Table D shows the results on the validation set. Our proposed Voxel Mamba achieves 79.6/73.4 on L1/L2 mAPH, which are +1.4 and +1.5 better than DSVT- Voxel. Since our framework differs from DSVT only the 3D backbone, it can be concluded that Voxel Mamba has superior ability in capturing voxel features. In comparison with window-based (e.g., DSVT) or curve-based (e.g., PTv3) group-free method Voxel Mamba consistently delivers better results. Table
grouping methods, shows the results on the test split. Voxel Mamba reaches 79.66 79.6/74.3 in terms of L1/L2 mAPH. which even better than the 3-frame setting of PillarNeXt and SST.
nuScenes. We then compare Voxel Mamba with previous state-of-the-a methods on nuScenes.
Table 3 shows the results on the validation set. Voxel Mamba achieves impressive results with 71.9
7

---

Table 3: Comparison with the state-of-the-ar detectors the nuScenes dataset validation split.
Method
NDS
mAP
Car
Truck
Bus
T.L.
C.V.
Ped.
M.T.
Bike
T.C.
B.R.
CenterPoint
66.5
59.2
84.9
57.4
70.7
38.1
16.9
85.1
59.0
42.0
69.8
68.3
VoxeINeXt 8
66.7
60.5
83.9
53.5
70.5
38.1
21.1
84.6
62.8
50.0
69.4
69.4
TransFusion-L.
1
70.1
65.5
86.9
60.8
73.1
43.4
25.2
87.5
72.9
57.3
77.2
70.3
PillarNeXt [32
68.4
62.2
85.0
57.4
67.6
35.6
20.6
86.8
68.6
53.1
77.3
69.7
HEDNNt [75]
71.4
66.7
87.7
60.6
77.8
50.7
28.9
87.1
74.3
56.8
76.3
66.9
DSVT [65
71.1
66.4
87.4
62.6
75.9
42.1
25.3
88.2
74.8
58.7
77.8
70.9
Voxel Mamba (ours)
87.9
62.8
76.8
45.9
24.9
89.3
77.1
58.6
80.1
71.5
Table 4: Comparison with the state-of-the detectors on the nuScenes dataset test split.
Method
NDS
mAP
Car
Truck
Bus
T.L.
C.V.
Ped.
M.T.
Bike
T.C.
B.R.
PointPillars 31
45.3
30.5
68.4
23.0
28,2
23.4
4.1
59.7
27.4
1.1
30,8
38.9
3DSSD 72
56.4
42.6
81.2
47.2
61.4
30.5
12.6
70.2
36.0
8.6
31.1
47.9
CenterPoint
14
65.5
58.0
84.6
51.0
60.2
53.2
17.5
83.4
53.7
28.7
76.7
70.9
FCOS-LiDAR 64
65.7
82.2
52.9
48.8
84.5
68.0
39.0
79.2
70.7
AFDetV2 30
68.5
62.4
86.3
54.2
62.5
58.9
26.7
85.8
63.8
34.3
80.1
71.0
UVTR-L 33
69.7
63.9
86.3
52.2
62.8
59.7
33.7
84.5
68.8
41.1
74.7
74,9
VISTA [I]
69.8
63.0
84.4
55.1
63.7
54.2
25.1
82.8
70.0
45.4
78.5
71.4
Focals Conv
6
70.0
63.8
86.7
56.3
67.7
59.5
23.8
87.5
64.5
36.3
81.4
74.1
VoxeINeXt
70.0
64.5
84.6
53.0
64.7
55.8
28.7
85.8
73.2
45.7
79.0
74.6
TransFusion-I.
70.2
65.5
86.2
56.7
66.3
58.8
28.2
86.1
68.3
44.2
82.0
78.2
LinK 39
71.0
66.3
86.1
55.7
65.7
62.1
30.9
85.8
73.5
47.5
80.4
75.5
HEDNet 75
72.0
67.7
87.1
56.5
70.4
63.5
33.6
87.9
70.4
44.8
85.1
78.1
LargeKemel33
70.6
65.4
85.5
53.8
64.4
59.5
29.7
85.9
72.7
46.8
79.9
75.5
PillarNet 533
71.4
66.0
87.6
$7.5
63.6
63.1
27.9
87.3
70.1
42.3
83.3
77.2
FSDv2 15
71.7
83.7
66.4
59.1
32.5
87.1
71.4
51.7
80.3
78.7
DSVT 65
72.7
68.4
86.8
67.3
63.1
37.1
88.0
73.0
47.2
84.9
Voxel Mamba (ours)
73.0
69.0
86.8
57.1
68.0
63.2
35.4
89.5
74.7
50.8
86.9
77.3
NDS and 67.5 mAP, which is +0.5 and +0.8 higher than the previous best method. Compared with DSVT, Voxel Mamba achieves +1.1 higher performance on mAP. The results on the test split are shown in Table 4 Our method also exhibits the best mAP and NDS.
Inference Efficiency. We compare Voxel Mamba with other state-of-the-an thettrr methods in inference speed and performance accuracy in Fig. 3 Notably, Voxel Mamba outperforms DSVT (65 and PV-RCNN+ [54] by at least++1. +1.5 in detection accuracy, while achieving faster speed. Some methods, such CenterPoint [74] and PointPillar [31. are faster than Voxel Mamba; however, their accuracy substantially lower.
We further compare Voxel Mamba with previous well-designed architectures (SpCNN, Transformers, and 2D CNN) in GPU memory in Table Compared with CenterPoint-Pille Voxel Mamba requires only an additional 0.5 GB GPU memory but achieves +9.0 higher accuracy in L2 mAPH. While Transformer-basee methods like SST [3] and DSVT 65] use group partitioning, they still consume more memory than our group-free Voxel Mamba. All the experiments are evaluated on an NVIDIA A100 GPU with the same environment.
4.4
Ablation Studies
To better investigate the effectiveness of Voxel Mamba, conduct set of ablation studies by using the nuScenes validation set. We follow OpenPCDet 63 to train all models for 20 epochs.
Effectiveness of Space-filling Curves. There are some potential alternatives to Hilbert curve for preserving locality. Here, we compare Hilbert curve with some commonly used space-filling curves (Z-order (66) and window partition space-filling curves (i.e,, the row of 'Random Curve'), there will be notable decline in performance, which indicates that spatial proximity is crucial in the group-free setting. By using the Z-order curve
[13) in 3D detection. As shown in Table 6(a). without using

---

Performance vs. Speed
Voxel Mamba (Ours)
Table 5: Comparison with other well-designed architectures on GPU memory.
DSVT-Voxel
70.3
LargeKemel3D
Parformanca (MAP@L2
+
Abalation
Backbone
Memory (GB)
CenterPoint Voxel
PointPillar 31
3.6
65.0
Part_A A2
VoxSet
Centerpoint- Pillar
2D CNN
3.2
PV-RCNN
CenterPoint-Rilar
Part-A2 57
SpCNN
2.9
PV-RONN+ (ResNet)
17.2
00.0
SST [3
6.8
Transformers
Second
DSVT-Voxel
4.2
PointRillar
Voxel Mamba (Ours)
SSMs
3.7
20 per (FPS)
Figure 3: Detection performance (mAPH/L2) vs.
speed (FPS) on Waymo.
Table Ablations on the nuScenes validation split. In (d), Centerpoint- Pillar used as the baseline.
Space-filing Curve
mAP
NDS
Ablation
mAP NDS
Baseline
63.3
69.1
Random Curve
66.0
71.0
+ bidirectional SSMs
65.8
70.9
Window Partition
67.3
71.7
(Hilbert curve)
Z-Order Curve
67.0
71.7
+ Voxel
66.3 71.0
+ DSB
66.7
71.3
Hilbert Curve
67.5
71.9
+ IWP
67.5
71.9
(a) Ablation on space- filling curves.
(b) Effect each component in Voxel Mamba.
Downstrides mAP NDS
Pos Embeding
mAP NDS
[1,1,1]
66.6 71.4
Baseline
66.7
71.3
[1,2,]
66.9 71.8
Absolute position
66.9
71.2
[2,2,2]
65.6 70.8
Cos, Sin
66.6
71.4
[4,4,4]
66.2 71.2
Ours (w/o shift)
67.3
71.9
[1,2,4)
67.5 71.9
Ours
67.5
71.9
(c) Ablation on the downsampling rates of DSB.
(d) Ablation IWE.
and window partition to introduce spatial proximity, the mAP and NDS much improved. The scrialization based on the Hilbert curve can further enhance the model performance.
Effectiveness of Each Component. To more clearly illustrate the effectiveness of the different components in Voxel Mamba, we conduct experiments by adding each them baseline, which is set to Centerpoint Pillar [74). As shown in Ta)le/O(b, bidirectional SSMs with a Hilbert- based group- free sequence can significantly improve the accuracy over the baseline, which validates the feasibility of our group-free strategy. Besides, converting pillar to voxel can enhance much the detector's performance without group size constraints. Voxel Mamba with DSB obtain better performance than the plain bidirectional SSMs. This is because DSB can build larger ERFs and mitigate the loss of proximity. Furthermore, IWE further boosts Voxel Mamba's performance for its capability in capturing 3D position information and increasing voxel proximity.
Downsampling Rates of DSB. We evaluate the impact of different downsampling rates in DSB by adjusting the stride {d1, d2, d in the backward SSM branch at each stage. d; = means the original resolution used. The results are shown Table T.). We see that transitioning from (1,1,1) to [1,2,2] and to [1,2,4] enhances performance due to an enlarged effective receptive field and improved proximity by using larger downsampling rates at late stages. However, DSBs with (2,2,2) or (4,4,4) compromise performance compared to [1,1,1), indicating that using larger downsampling rates at early stages will lose some fine details. Thus, we set the stride as (1,2,4) to strike a balance between effective receptive fields and detail preservation.

---

0.4
02
0.0
Figure 4: The effective receptive fields (ERFs) of Voxel Mamba (left), group-based bidirectional Mamba (middle) and DSVT (right).
Effectiveness of IWE. d))) validates the capability of IWE enhance spatial proximity. We compare IWE with some commonly used positional embedding methods [I3] 65) in 3D detection. Absolute position denotes the direct encoding of voxel coordinates using an MLP. The results demonstrate that IWE can significantly improve the detection performance by offering features with rich 3D positional and proximate information.
4.5 Effective Receptive Field of Voxel Mamba
Fig. dillustrates the Effective Receptive Fields [2] (ERFs) of window partition-based method DSVT (65, group-based bidirectional Mamba and our proposed group-free method Voxel Mamba. For clear visualization, all models take pillars as inputs. The group partition in the group-based bidirectional Mamba is configured identically to DSVT. Then, we randomly select voxels of interest from the ground truth bounding box and calculate the ERF at each non-empty voxel position. Subsequently, we merge the ERFs into single image by taking the maximum value at each voxel location. A wider activation area indicates larger ERF. From Fig. we see that Voxel Mamba exhibits a notably larger ERF than DSVT and group-based bidirectional Mamba, which can be attributed to the benefits of group-free operation. The larger ERF can cover more complete local region and enhance the spatial proximity in ID sequences.
5 Conclusion
In this paper, proposed Voxel Mamba, group-free SSM-based 3D backbone for point cloud based 3D detection. We first analyzed the proximity loss of group partition current serialization-bar 3D detection methods. By taking the advantage of linear complexity of SSMs, proposed group-free free strategy to alleviate the loss of spatial proximity in 3D to 1D serialization. We further proposed the DSB block and IWP strategy to build larger effective receptive fields and improve the spatial proximity of our Voxel Mamba framework. Experiments demonstrated that Voxel Mamba achieved state-of-the results Waymo and nuScene datasets. Without elaborated optimization, consumed less memory than group-based Voxel Transformer methods, and our group-free strategy was more efficient and eeployment-friendde than group partition. Voxel Mamba provided an efficient group-free solution for sparse point clouds for 3D tasks.
model
Limitations. While the proposed Voxel Mamba achieves state-of-the-art performance in point cloud based 3D object detection, it still has some limitations to be further addressed. First, in the Hilbert Input Layer, the curve templates occupy approximately 0.1 GB of GPU memory, which may become substantial as the voxel resolution increases. Besides, a more elaborately designed downsampling and upsampling operation could improve more the model efficiency. We will investigate these problems in future work.
A Acknowledgments. This work was supported in part by the InnoHK Program
10

---

References
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1090~1099, 2022.
[2] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11633 2020.
[3] Guangyan Chen, Meiling Wang. Yi Yang, Kai Yu, Li Yuan, and Yufeng Yue. Pointgpt: Auto- regressively generative pre-training from point clouds. Advances in Neural Information Process- ing Systems, 36, 2024.
[4] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. arXiv preprint arXiv:2306.16927, 2023.
[5] Wanli Chen, Xinge Zhu, Guojin Chen, and Bei Yu. Efficient point cloud analysis using hilbert curve. In European Conference on Computer Vision, pages 730-747. Springer, 2022.
[6] Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[7] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Largekernel3d: Scaling up kernels in 3d sparse cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13498, 2023.
[8] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21674-21683, 2023.
[9] Spconv Contributors. Spconv: Spatially sparse convolution library. htttp://github..om
traveller5//spcony 2022.
[10] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1201-1209, 2021.
[11] Shengheng Deng, Zhihao Liang, Lin Sun, and Kui Jia. Vista: Boosting 3d object detection via dual cross-view spatial attention. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8448-8457, 2022.
[12] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding, Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pages 11963-11775, 2022.
[13] Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 2022.
[14] Lue Fan, Feng Wang, Naiyan Wang, and Zhao-Xiang Zhang. Fully sparse 3d object detection. Advances in Neural Information Processing Systems, 35:35 1-363, 2022.
[15] Lue Fan, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Fsd v2: Improving fully sparse 3d object detection with virtual voxels. arXiv preprint wrXiv22308003555 2023.
[16] Lue Fan, Yuxue Yang, Yiming Mao, Feng Wang, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Once detected, never lost: Surpassing human performance in offline lidar based 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19829,19829, 2023.
11

---

[17] Lue Fan, Yuxue Yang, Feng Wang. Naiyan Wang, and Zhaoxiang Zhang. Super sparse 3d object detection. IEEE transactions on pattern analysis and machine intelligence, 2023.
[18] Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprini arXiv:22/2. 14052, 2022.
[19] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. 2012 IEEE conference on computer vision and puttern recognition, pages 3354-333 3361. IEEE, 2012.
[20] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00755 2023.
[21] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arviv preprint arXiv:2111.00396 2021.
[22] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572-585, 2021.
[23] Tianrui Guan, Jun Wang, Shiyi Lan, Rohan Chandra, Zuxuan Wu, Larry Davis, and Dinesh Manocha. M3detr: Multi-representation, multi-scale, mutual-relation 3d object detection with transformers. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 772-782, 2022.
[24] Xu Han, Yuan Tang, Zhaoxuan Wang, and Xianzhi Li. Mamba3d: Enhancing local features for 3d point cloud analysis via state space model. arXiv preprint arXiv:2404. 14966, 2024.
[25] Chenhang He, Ruihuang Li, Shuai Li, and Lei Zhang. Voxel set transformer: A set-to-set approach 3d object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[26] Chenhang He, Ruihuang Li, Guowen Zhang, and Lei Zhang. Scatterformer: Efficient voxel transformer with scattered linear attention. arXiv preprint arXiv:2401.00911 10091, 2024.
[27] Chenhang He, Hui Zeng, Jianqiang Huang. Xian- Sheng Hua, and Lei Zhang. Structure aware single-stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11873-1882, 2020.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[29] David Hilbert and David Hilbert. Über die stetige abbildung einer linie auf ein fläähenstück. Dritter Band: Analysis: Grundlagen der Mathematik Physik Verschiedenes: Nebst Einer Lebens- geschichte, pages 1-2, 1935.
[30] Yihan Hu, Zhuangzhuang Ding, Runzhou Ge, Wenxin Shao, Li Huang, Kun Li, and Qiang Liu. Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 969-979, 2022.
[31] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697-12705, 2019.
[32] Jinyu Li, Chenxu Luo, and Xiaodong Yang. Pillarnext: Rethinking network designs for 3d object detection in lidar point clouds. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17567-17576, 2023.
[33] Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel- based representation with transformer for 3d object detection. Advances in Neural Information Processing Systems, 35:18442- 18855, 2022.
12

---

[34] Dingkang Liang, Xin Zhou, Xinyu Wang, Xingkui Zhu, Wei Xu, Zhikang Zou, Xiaoqing Ye. and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv preprint arXiv:2402. 10739, 2024.
[35] Jiuming Liu, Jinru Han, Lihao Liu, Angelica Aviles-Rivero, Chaokang Jiang, Zhe Liu, and Hesheng Wang. Mamba4d: Efficient long-sequence point cloud video understanding with disentangled spatial-temporal state space models. arXiv preprint (tti:24055/43336 2024.
[36] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang. Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401. 2024.
[37] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. 2021 ieee. In CVF International Conference Computer Vision (ICCV), pages 2929- 2938,993,2222 2021.
[38] Zhijian Liu, Xinyu Yang, Haotian Tang, Shang Yang, and Song Han. Flatformer: Flattened win- dow attention for efficient point cloud transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1200-1211, 2023.
[39] Tao Lu, Xiang Ding, Haisong Liu, Gangshan Wu, and Limin Wang. Link: Linear kernel for lidar-based 3d perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 100--115,, 2023.
[40] Chuanyu Luo, Nuo Cheng, Sikun Ma, Han Li, Xiaohan Li, Shengguang Lei, and Pu Li. Lest: Large-scale lidar semantic segmentation with transformer. arXiv preprint arXiv:2307.09366 2023.
[41] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
[42] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3164-3173, 2021.
[43] Mohamed F Mokbel, Walid Aref, and Ibrahim Kamel. Analysis of multi-dimennional space-filling curves. GeoInformatica, 7:179-209, 2003.
[44] Guy M Morton. A computer oriented geodetic data base and new technique in file sequencing. 1966.
[45] Yong-Joo Oh and Yoshio Watanabe. Development of small robot for home floor cleaning. In Proceedings of the 41st SICE Annual Conference. SICE 2002., volume 5, pages 3222-3223. IEEE, 2002.
[46] Jack A Orenstein. Spatial query processing in an object-oriented database system. In Pro- ceedings of the 1986 ACM SIGMOD international conference Management of data, pages 326-336, 1986.
[47] Youngmin Park, Vincent Lepetit, and Woontack Woo. Multiple 3d object tracking for augmented reality. In 2008 7th IEEE/ACM International Symposium Mixed and Augmented Reality, pages 117-120. IEEE, 2008
[48] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277-9286, 2019.
[49] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 918-927, 2018.
[50] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
13

---

[51] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[52] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Feaf3d: Fully convolutional anchor- free 3d object detection. In European Conference on Computer Vision, pages 477-493. Springer, 2022.
[53] Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: Real-time and bigh-performance pillar- based 3d object detection. In European Conference on Computer Vision, pages 35-52. Springer, 2022.
[54] Shaoshuai Shi, Chaoxu Guo, Jiang, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng
Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10529-10538, 2020.
[55] Shaoshuai Shi, Li Jiang, Jiajun Deng. Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang. and Hongsheng Li. Pv-renn++: Poxnt-voxel feature set abstraction with local vector representation for 3d object detection. International Journal of Computer Vision, 131(2)5311551, 2023.
[56] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770-779,200 2019.
[57] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE transactions on pattern analysis and machine intelligence, 23(8):2647-2664, 2020.
[58] John Skilling. Programming the hilbert curve. In AIP Conference Proceedings, volume 707,
pages 381-387. American Institute of Physics, 2004.
[59] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.0490)0 2022.
[60] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard. Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446-2454, 2020.
[61] Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In European Conference on Computer Vision, pages 426-442. Springer, 2022.
[62] Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Sminchisescu, and Dragomir Anguelov. Rsn: Range sparse net for efficient, accurate lidar 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5725-5725 5734, 2021.
[63] OpenPCDet Development Team. Openpedet: An open-source toolbox for 3d object detection from point clouds. https:://// httpgggtttuu..eeee http://ppnnmmmlaapppeeeeeee
2020.
[64] Zhi Tian, Xiangxiang Chu, Xiaoming Wang, Xiaolin Wei, and Chunhua Shen. Fully convo- lutional one-stage 3d object detection on lidar range images. Advances in Neural Information Processing Systems, 35:34499-34911, 2022.
[65] Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, and Liwei Wang. Dsvt: Dynamic sparse voxel transformer with rotated sets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 13520-13529, 2023.
[66] Peng-Shune Leng-Shuai Wang. Octformer: Octree-based transformers for 3d point clouds. arXiv preprint ar50v:2305.03045, 2023.
[67] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (tog), -8-2 2019.
14

---

[68] Zicheng Wang. Zhenghao Chen, Yiming Wu, Zhen Zhao, Luping Zhou, and Dong Xu. Pointramba: A hybrid transformer-mabteee framework for point cloud analysis. arXiv preprint arXiv:2405. /50000.,554,,,, 2024.
[69] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. arXiv preprint mrXiv2231..1001 2023.
[70] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
[71] Honghui Yang, Zili Liu, Xiaopei Wu, Wenxiao Wang, Wei Qian, Xiaofei He, and Deng Cai. Graph r-cnn: Towards accurate 3d object detection with semantic-decorated local graph. In European Conference on Computer Vision, pages 662 679. Springer, 2022.
[72] Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11040-11048, 2020.
[73] Zetong Yang. Yin Zhou, Zhifeng Chen, and Jiquan Ngiam. 3d-man: 3d multi-framee attention network for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1863-1872, 2021.
[74] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVE conference on computer vision and paltern recognition, pages 11784-11793, 2021.
[75] Gang Zhang, Chen Jumnan, Guohuan Gao, Jianmin Li, and Xiaolin Hu. Hednet: hierarchical encoder-decoder network for 3d object detection in point clouds. Advances in Neural Information Processing Systems, 36, 2024.
[76] Guowen Zhang, Junsong Fan, Liyi Chen, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. General geometry-aware weakly supervised 3d object detection. arXiv preprint arX1v:240.11348, 2024.
[77] Tao Zhang, Xiangtai Li, Haobo Yuan, Shunping Ji, and Shuicheng Yan. Point could mamba: Point cloud learning via state space model. arXiv preprint arXiv:2403.00762, 2024.
[78] Qingyuan Zhou, Weidong Yang, Ben Fei, Jingyi Xu, Rui Zhang, Keyi Liu, Yeqi Luo, and Ying He. 3dmambaipf: A state space model for iterative point cloud filtering via differentiable rendering. arXiv preprint LrXiv:2404.0552., 2024.
[79] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490 2018
[80] Zixiang Zhou, Xiangchen Zhao, Yu Wang. Panqu Wang, and Hassan Foroosh. Centerformer: Canter-based transformer for 3d object detection. In European Conference on Computer Vision, pages 496-513, Springer, 2022.
[81] Benjin Zhu, Zhe Wang, Shaoshuai Shi, Hang Xu, Lanqing Hong, and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 9296-9305. 2023.
[82] Lianghui Zhu, Bencheng Liao, Qian Zhang. Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.0941 249477 2024.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the limitation in Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our paper is easy to follow and provide implementation details in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
[Yes]
Justification:
The data and code are available currently.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwiiicgguie//eeeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the implementation details in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We introduce the required resource in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We focus on point cloud-based 3D object detection, which can be applied to autonomous driving. For positive socictal impact, our approach enhances the precision of 3D detection, thereby augmenting the safety aspect of autonomous driving. For negative societal impact, false detection results can potentially lead to traffic accidents.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the related original paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "UVzZveuyMK", "reviewer": "Reviewer_dTn9", "summary": "This paper proposes a new 3D detection architecture, Voxel Mamba. Voxel Mamba serializes voxels using a Hilbert Input Layer and then applies a forward SSM and a backward SSM at lower resolutions. It achieves state-of-the-art performance on both the Waymo Open Dataset and nuScenes dataset.", "questions": "In table 5 (d), the Hilbert curve is marginally better than the Z-order curve. In 5(c), adding Hilbert improves mAP and NDS by 0.1. \nCurious, what is the latency of generating the Hilbert curve and Z-order Curve. What about the latency of generating Random Curves?\n\nFor random curves, do you use the same random curves for all frames or random sample a curve for each frame? Do you use the same random sampled curves for different resolutions, or use the same random curve for all resolutions in the backward SSM?\n\nWhat would be the expected performance and latency if you flattened the voxels to 1D but used window attention (either shifted or scanned) instead of Mamba? This could help assess the trade-offs between different approaches to handling long sequence lengths.", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: \n\nAddresses the challenge of long sequence lengths in group-free 3D detection models by using the Mamba SSM module to extract features from all voxels, avoiding inefficient sorting and grouping operations found in window-based sparse 3D detectors.\n\nQuality:\n\n Demonstrates strong 3D detection performance on both Waymo Open Dataset and nuScenes, supported by extensive ablation studies in Table 5, highlighting the effectiveness of Voxel Mamba's design choices.\n\nClarity: \n\nThe paper is well-written with clear visualizations, notably Figure 4, which illustrates the larger effective receptive field of Voxel Mamba compared to group-based methods.", "weaknesses": "Please see the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1] Latency of generating curves.**\n\nIn our implementation, we record the traversal position in the space-filling curves of all potential voxels offline. The voxels are serialized by simply looking up these traversal positions. Thus, the latency for serializing voxels based on Hilbert and Z-order curves is identical. The detailed generation times of the Hilbert Input Layer are shown in the **shared responses Q1**.\n\n**[Q2] Random Curves setting.**\n\nIn the Hilbert Input Layer (HIL) implementation,  we use fixed templates for each resolution in the ASSMs. With the downsampling rate {1,2,4} in ASSMs, Voxel Mamba requires three different scale curve templates.\n\nHowever, the \"random\" in Table 5(a) means that we do not perform any sorting on the input sequence for SSMs. We observed that the Voxel Feature Extractor (VFE) and the downsampling and upsampling operations tend to preserve the spatial proximity of neighboring voxels in the 3D space in the 1D sequence. This inherent characteristic leads to a favorable outcome for SSMs. To ensure a real random order, we generate three random templates and use them for all frames following the HIL setting.\nThe following shows the results on the Waymo open dataset with 20% training data. These results indicate that disrupting voxel proximity significantly degrades performance, highlighting the effectiveness of our Hilbert Input Layer.\n\n| Method       | L2 mAPH | Vehicle (L1) | Vehicle (L2) | Pedestrian (L1) | Pedestrian (L2) | Cyclist (L1) | Cyclist (L2) |\n|--------------|---------|--------------|--------------|-----------------|-----------------|--------------|--------------|\n| Random order | 51.7    | 56.4 / 55.7  | 49.1 / 48.4  | 67.0 / 57.9     | 58.6 / 50.6     | 60.4 / 58.2  | 58.1 / 56.0  |\n| Hilbert      | 71.6    | 79.0 / 78.5  | 70.7 / 70.2  | 84.0 / 79.1     | 76.7 / 72.0     | 76.5 / 75.4  | 73.7 / 72.7  |\n\n**[Q3] Hilbert Input Layer + attention.**\n\nWe truly appreciate this valuable comment. We conducted an experiment by replacing the window grouping in DSVT with the serialization approach in Voxel Mamba. We use the same group size of 48 as DSVT, and only pad the last voxel groups for each sample to enable the parallel computation of Transformers. We employ curve shift as an alternative to the X/Y-axis partition and perform experiments on 20% Waymo training data. The following table compares this variation with Voxel Mamba and DSVT.\n\n| Method                   | L2 mAPH | Vehicle (L1) | Vehicle (L2) | Pedestrian (L1) | Pedestrian (L2) | Cyclist (L1) | Cyclist (L2) | Latency (ms) |\n|--------------------------|---------|--------------|--------------|-----------------|-----------------|--------------|--------------|--------------|\n| DSVT                     | 69.7    | 77.9 / 77.4  | 69.5 / 69.1  | 82.2 / 76.3     | 74.8 / 69.1     | 74.7 / 73.6  | 71.9 / 70.9  | 94           |\n| Hilbert + self-attention | 68.9    | 77.3 / 76.8  | 69.0 / 68.5  | 82.0 / 75.7     | 74.5 / 68.7     | 73.4 / 72.3  | 70.7 / 69.6  | 85            |\n| Voxel Mamba              | 71.6    | 79.0 / 78.5  | 70.7 / 70.2  | 84.0 / 79.1     | 76.7 / 72.0     | 76.5 / 75.4  | 73.7 / 72.7  | 90           |\n\nOne can see that this variant with the Hilbert Input Layer shows slightly lower performance than DSVT. We hypothesize that this drop is due to the limited group size in the window attention mechanism. This constraint might limit the full potential of the HIL voxel flattening. While it does not outperform Voxel Mamba in accuracy, it does offer a favorable trade-off between performance and latency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mOcKqsRJVR", "reviewer": "Reviewer_kB9g", "summary": "The authors introduce Voxel Mamba, a 3D object detection method based on state-space models.  Voxel Mamba introduces a efficient group-free approach to avoid inefficient grouping operations and prevent limitations on the receptive field for 3D object detection in point clouds. The authors also present Asymmetrical State Space Models (ASSMs) to capture multi-scale context information, offering each voxel data-dependent global contexts. Voxel Mamba enables voxel interactions throughout the scene and captures cloud locality using space-filling curves. Extensive experiments validate its performance, positioning Voxel Mamba as a viable alternative for 3D object detection.", "questions": "See Weakness. Besides, there are few other suggestions:\n\n1. Could the authors provide more description of Window Sweep in Table 5(c)? Does it include the Shifted Regional Grouping Shifted Token Groups operation used in SST?\n1. Is a Linear for the up dimension missing from the SSM part of Fig. 2?", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow.\n2. The authors first time delve into the potential of Mamba in 3D object detection, which is an intriguing subject.", "weaknesses": "1. Motivation can be improved. The authors must clarify why SSM is essential for their motivation and quantify the advantages it offers over transformers for this task. The question in lines 55-56 does not demonstrate SSM's superiority. The author's design for SSM is also hardly inspiring for the 3D detection community.\n2. Some of the method's design confuse me. Traversing the downsampled feature maps with a Hilbert curve in the opposite direction can misalign the feature receptive fields. Does this challenge model training with varying inputs, even with IWPE included? Additionally, given its use of multi-scale features, how much does ASSM outperform FPN?\n3. The experimental comparisons may be inadequate. The authors present a pillar version for Voxel Mamba but fail to compare its performance and efficiency against current state-of-the-art pillar-based methods in the main table.\n4. The experiment's details should be more clearly documented. For example, does Voxel Mamba adopts IoU-rectification scheme? The comparative setup in the ablation study could have been accounted for in more detail and analyzed why the method designed by the authors was effective.\n5. How does Voxel Mamba integrate with multi-frame temporal 3D detection?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1] Motivation.**  \n\nThanks for the question and we are sorry for not making the motivation clear enough.  The motivation of our work is to leverage the advantages of SSM in linear attention for more effective feature learning in 3D point cloud based object detection.  The advantage of SSM over Transformer lies in its ability to more efficiently explore large context (as shown in Fig. 4 of the main paper), i.e., an extremely long voxel sequence, for effective feature learning. From the table below, one can see that for more than 4K voxel contexts, a single transformer layer consumes more than 5ms, which is impractical for perception tasks. However, even for a sequence of 16K voxels, SSM only consumes 0.94ms. \n\n| Method       | 1K     | 2K     | 4K      | 8K      | 16K     |\n|--------------|--------|--------|---------|---------|---------|\n| Transformer  | 0.47 ms| 1.61 ms| 5.74 ms | 26.02 ms| 114.20 ms|\n| SSM          | 0.41 ms| 0.43 ms| 0.50 ms | 0.61 ms | 0.94 ms  |\n\nUnlike previous window-based voxel transformers, we are the first to model the voxel sequence from the entire scene. By utilizing Mamba's selective scan mechanism, we can effectively retain useful information from the context without the need for complex voxel grouping. This group-free approach is insightful for handling large-scale sparse data.\n\n**[Q2] Effectiveness of multi-scale design.**  \n\nWe appreciate the reviewer's insightful observation. In fact, as a sequence-based model, the receptive fields of voxel features only rely on the sequence length, and they do not necessarily need to be aligned. Mamba can selectively retain useful information after scanning the sequence back and forth. Here we further compare ASSM with FPN on the Waymo dataset with 20% training data. Following Voxel Mamba, we utilize Spconv and its counterpart SpInverseConv as downsampling and upsampling operations. We keep the same downstrides in both structures. The table below shows that our ASSM achieves superior performance in the 3D object detection tasks.\n\n| Model Size            | L2 mAPH | Vehicle (L1) | Vehicle (L2) | Pedestrian (L1) | Pedestrian (L2) | Cyclist (L1) | Cyclist (L2) |\n|-----------------------|---------|--------------|--------------|-----------------|-----------------|--------------|--------------|\n| Mamba with FPN        | 68.2    | 77.4 / 76.9  | 69.1 / 68.6  | 81.3 / 74.0     | 73.8 / 66.9     | 72.1 / 71.0  | 69.4 / 68.3  |\n| Vxoel-Mamba-Pillar    | 69.5    | 78.1 / 77.6  | 69.8 / 69.3  | 82.4 / 75.5     | 75.0 / 68.5     | 74.6 / 73.4  | 71.8 / 70.7  |\n\n**[Q3] Comparison with state-of-the-art pillar-based methods.** \n\nThanks for the suggestion. The following table compares our pillar-based Voxel Mamba with previous state-of-the-art pillar-based methods. We can see that Voxel Mamba outperforms previous methods on both L2 mAP and mAPH while achieving comparable inference speed.\n\n| Method                 | L2 mAP/mAPH | Vehicle (L1) | Vehicle (L2) | Pedestrian (L1) | Pedestrian (L2) | Cyclist (L1) | Cyclist (L2) | Latency (ms) |\n|------------------------|-------------|--------------|--------------|-----------------|-----------------|--------------|--------------|--------------|\n| SST                    | -           | 76.2 / 75.8  | 68.0 / 67.6  | 81.4 / 74.1     | 72.8 / 65.9     | -            | -            | 71           |\n| DSVT-Pillar            | 73.2 / 71.0 | 79.3 / 78.8  | 70.9 / 70.5  | 82.8 / 77.0     | 75.2 / 69.8     | 76.4 / 75.4  | 73.6 / 72.7  | 64           |\n| Voxel Mamba-Pillar     | 73.5 / 71.2 | 80.2 / 79.7  | 72.1 / 71.7  | 83.5 / 77.6     | 76.1 / 70.5     | 75.0 / 74.0  | 72.2 / 71.3  | 66           |\n\n**[Q4] Detailed experiments and ablation studies.** \n\nWe apologize for any confusion regarding the details of our experiments. For fair comparison, our experiment settings follow the schemes in DSVT, using the Adam optimizer with weight decay of 0.05, a one-cycle learning rate policy, and a maximum learning rate of 2.5e-3. Models were trained with a batch size of 24 for 24 epochs on 8 NVIDIA A6000 GPUs. During inference, we use class-specific NMS with IoU thresholds of 0.7, 0.6, and 0.55 for vehicles, pedestrians, and cyclists, respectively. Ground-truth copy-paste data augmentation was used during training and disabled in the last epoch. Our setting aligns closely with the previous state-of-the-art DSVT, and the primary difference is on the backbone. In addition, **Voxel Mamba does not adopt the IoU-rectification scheme**.\n\nIn our ablation study, we assessed the effectiveness of various components and drew the following conclusions:   \n(a) Space-filling curves enhance spatial proximity and locality, significantly improving performance (Table 5d).   \n(b) Position embedding with IWPE boosts detection performance by providing rich 3D positional and proximate information (Table 5a).   \n(c) Downsampling rates of ASSM show that larger effective receptive fields improve performance, but very large downsampling rates at early stages can degrade performance. We found a balance with a stride of {1,2,4} (Table 5).   \n(d) Bidirectional SSMs with a Hilbert-based group-free sequence significantly improves accuracy, validating our group-free strategy. ASSM enhances effective receptive fields and proximity, while IWPE further boosts Voxel Mamba’s performance by capturing 3D positional information (Table 6c).   \n\n**[Q5] Multi-frame results.**  \n\nPlease refer to the **shared responses Q2** for details.\n\n**[Q5] Description of window sweep.**\n  \nThe window sweep adopts the same regional grouping method as SST, extended to 3D voxels.\nWe use a fixed BEV window size of (12, 12) and implement Region Shift between each two ASSM blocks.\n\n**[Q6] Detail of Figure 2.**  \n\nWe apologize for this problem. We will revise Figure 2 to accurately reflect the complete architecture.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nnrtxASFNJ", "reviewer": "Reviewer_2cxV", "summary": "This paper proposes a novel backbone named Voxel Mamba for point cloud 3D detection. Different from previous methods that group the voxels into fixed-length sequences through padding, this paper adopt an interesting group-free strategy to sort all voxels into a single sequence through the space-filling curve. The paper also proposes the Asymmetrical State Space Models (ASSMs) and Implicit Window-based Position Embedding (IWPE) to enlarge the receptive fields of Voxel SSMs. \nExperiments are mostly based on challenging outdoor datasets, the Waymo Open Dataset and nuScene. Experiments demonstrates Voxel Mamba surpasses previous methods.", "questions": "Please see Weaknesses above. Overall, the proposed group-free strategy has better efficiency than previous group-based operations without token padding. Voxel Mamba also demonstrates better performance than existing sparse convolution or transformer structures. As such, I recommend acceptance at this stage.", "limitations": "Limitations have been included", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper presents a novel approach to deal with irregularity problem of point cloud detection. By taking the advantage of linear complexity of SSM, this paper employ a group-free strategy which is more efficient and deployment-friendly than previous group-based methods.\n2.\tASSMs and IWPE are well motivated. The multi-scale design and implicit window partition are integrated under group-free design in an economical way. Besides, the figures in this paper are clear and greatly aid in the comprehension of the proposed methods.\n3.\tThe proposed Voxel Mamba exhibits a notably larger ERF than group-based methods DSVT.  The visualization effectively supports their claim.\n4.\tThe experiments are thorough and convincingly demonstrate the effectiveness of the proposed method.", "weaknesses": "1.\tThe experiments of model architectures are a bit insufficient. In this paper, the authors extend Mamba to bidirectional Mamba (line 68) and further to Voxel Mamba (line 74). However, there are no experiments demonstrating the performance gains from these upgrades. As Voxel Mamba is based on Mamba, it would be necessary to report the accuracy of apply group-free bidirectional Mamba and Mamba with the same SSM parameters or layers.\n2.\tGiven that the group-free strategy is a significant contribution of this paper, it is necessary to include more analysis and comparisons between group-free and group-based operations. For example, the author cloud provide a detailed latency comparison between window-based grouping (such as dsvt and sst) and HIL. Besides, on Line 188, the authors mention that the mapping processes takes 0.7ms. However, the latency is likely to vary with different sequence lengths. Please provide a clarification regarding the timing.\n3.\tLack of comparison with non-group-free Mamba. Although this work provides quantitative and visualization comparison between group-free and group-based (dsvt) methods. It is necessary to provide the ERFs and detection accuracy for non-group-free Voxel Mamba. As a linear RNN models, what difference does Mamba introduce compared to Transformers.\n4.\tThe analysis of the ablation on ASSM and HIL is a little vague. Specifically, it is unclear why performances drop under the downsampling setting {2,2,2} and {4,4,4} in Table 5(b). Besides, the window sweep, Z-order, and Hilbert methods in Table 5(d) exhibit similar performance. Can I infer that the form of the space-filling curve does not significantly affect the accuracy? Please provide a discussion on them, which would be helpful to improve the quality of this paper.\n5.\tHilbert Input Layer (HIL) sort the voxels by querying the space-filling curve template. As presented in Sec. 3.3, form my understanding, the template needs cover the whole scenes. My concern is the amount of GPU memory required by the template under the current voxel size setting and its contribution to the total memory consumption. And for a more fine-grained size such as (0.1m, 0.1m, 0.15m) in CenterPoint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1] Comparison with other variants.**\n\nThanks for your insightful comments. To demonstrate the improvements of our method, we have conducted additional experiments comparing Voxel Mamba with group-based and group-free bidirectional Mamba. The group-based bidirectional Mamba uses the DSVT Input Layer to partition the voxel groups. The group-free bidirectional Mamba is based on our Hilbert Input Layer. Each variant retains the same number of SSM layers for consistency in comparison.  \n\n| Method | L2 mAPH | Vehicle (L1) | Vehicle (L2) | Pedestrian (L1) | Pedestrian (L2) | Cyclist (L1) | Cyclist (L2) |\n|:--|--:|--:|--:|--:|--:|--:|--:|\n| Group-based bidirectional Mamba | 68.5 | 76.1 / 75.7 | 67.8 / 67.4 | 81.8 / 75.8 | 74.2 / 68.6 | 73.3 / 72.3 | 70.6 / 69.6 |\n| Group-free bidirectional Mamba | 71.0 | 78.3 / 77.9 | 70.0 / 69.6 | 83.0 / 78.1 | 75.6 / 70.9 | 76.3 / 75.2 | 73.4 / 72.4 |\n| Voxel Mamba | 71.6 | 79.0 / 78.5 | 70.7 / 70.2 | 84.0 / 79.1 | 76.7 / 72.0 | 76.5 / 75.4 | 73.7 / 72.7 |\n\nThe above table shows that enhanced spatial proximity can significantly improve the detector's performance. Additionally, our innovative multi-scale design yields additional performance gains, which can be attributed to the expansion of ERFs.\n\n**[Q2] Advantage over group-based operations.**  \n\nThanks for your suggestions. Please kindly refer to **shared response Q1** for the advantage of group-free operations.\n\n**[Q3] Comparison with group-based Mamba.**\n\nTo address the reviewer's concern, we compared the performances between group-free and group-based mamba (refer to the table in our response to Q1). The results demonstrate the effectiveness of our group-free strategy, which can enhance the 3D proximity in the 1D sequences. Besides, we provide the ERFs of group-based (DSVT partition) Mamba in Figure 1 in the rebuttal PDF.\n\n**[Q4] Analysis on ASSMs.**\n\nWe appreciate the reviewer’s insightful comment. The transitioning from \\{1,1,1\\} to \\{1,2,2\\} and to \\{1,2,4\\} enhances performance due to an enlarged effective receptive field and improved proximity by using larger downsampling rates at late stages. ASSMs with \\{2,2,2\\} or \\{4,4,4\\} compromise performance compared to \\{1,1,1\\}, indicating that using larger downsampling rates at early stages will lose some fine details. Thus, we set the stride as \\{1,2,4\\} to strike a balance between effective receptive fields and detail preservation. Additionally, your inference about the similar performance of different space-filling curves is astute. Voxel Mamba is designed with significantly large ERFs, which indeed diminishes the impact of the type of space-filling curves on performance. We will expand our analysis in revision.  \n\n**[Q5] Curve template memory.**\n\nYes, we record the traversal position in the space-filling curves of all potential voxels offline. The curve templates are adaptively generated based on the input BEV resolution. To address the reviewer's concern, the table below lists GPU memory usage for templates. The input BEV resolution is (468, 468), with three scale curve templates for each ASSM scale.\n\n| Template Resolution | Memory (MB) |\n|---------------------|-------------|\n| (512, 512)          | 83.0        |\n| (256, 256)          | 8.6         |\n| (128, 128)          | 1.2         |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "70QTLOXTg1", "reviewer": "Reviewer_vX3v", "summary": "The paper introduces a novel architecture for 3D object detection, addressing the limitations of current methods which rely on grouping operations. The proposed Voxel Mamba leverages State Space Models (SSMs) to capture long-range dependencies and process the entire scene in a single sequence, avoiding inefficient grouping. It employs a group-free strategy, an Asymmetrical State Space Models (ASSMs) block for multi-scale context, and an Implicit Window-based Position Embedding (IWPE) to maintain token locality. Experiments on the Waymo Open and nuScenes datasets demonstrate superior performance in accuracy and efficiency over established methods.", "questions": "1.\tThe role of using downsampling sequences in the backward SSM sequence in Figure 2 is not clearly explained in the paper.What does downsampling do?\n2.\tAre there any improvements made to the object detection baseline, including loss function and pipeline?\n3.\tIs there any redundancy in your title? \"based 3D Object Detection\" itself is a task of 3D point cloud, why is it written \"for Point Cloud\" in front of it?\n4.\tHow does the computational complexity of Voxel Mamba scale with the size and density of the point cloud data?\n5.\tWhat are the specific advantages of using SSMs over traditional CNN or Transformer architectures in the context of 3D object detection from point clouds?\n6.\tCan the authors provide more details on the design choices behind the Implicit Window-based Position Embedding and its impact on the model's receptive field?\n7.\tHow does Voxel Mamba handle dynamic objects or scenes with non-static elements, which are common in autonomous driving scenarios?\n8.\tHow does the performance of Voxel Mamba degrade with respect to various levels of occlusion or truncation of objects in the point cloud data?", "limitations": "1.\tThe paper does not mention the issue of computing resource consumption, or whether higher computing resources are required.\n2.\tThe paper does not explicitly discuss the model's generalization ability, that is, how well the model performs in new scenarios or under different environmental conditions. The paper's focus on autonomous driving scenarios may limit the general applicability of the findings to other fields where 3D object detection is relevant.\n3.\tThe reliance on large-scale datasets for training and evaluation might not fully capture the diversity of real-world scenarios, potentially limiting the model's robustness.\n4.\tThe potential impact of adversarial attacks or model overfitting to the specific characteristics of the training datasets is not discussed, which is important for model security and reliability.\n5.\tThe performance of Voxel Mamba on the Waymo Open and nuScenes datasets may not directly translate to other types of datasets, potentially limiting the model's universal applicability.\n6.\tIt is not clear whether the current Voxel Mamba model can be extended to a multi-task learning framework, such as simultaneously performing object detection, classification, and pose estimation tasks.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe Voxel Mamba's group-free strategy is innovative, potentially offering a more efficient and effective approach to handling point cloud data compared to traditional grouping methods.\n2.\tThe use of State Space Models (SSMs) for point cloud processing is a novel concept that shows promise for capturing long-range dependencies in 3D object detection tasks.\n3.\tThe Asymmetrical State Space Models (ASSMs) block cleverly integrates multi-scale context into the model, which likely enhances the feature representation and detection accuracy.\n4.\tWindow-based Position Embedding (IWPE) is a thoughtful addition to maintain the fine-grained 3D position information crucial for accurate object detection.\n5.\tThe architecture's flexibility to integrate into existing detection frameworks is a significant advantage, as it allows for the incremental adoption of Voxel Mamba in various systems.\n6.\tThe extensive experimental evaluation on large-scale and complex datasets like Waymo Open and nuScenes is commendable and provides strong evidence of the model's capabilities.", "weaknesses": "1.\tThe introduction does not clearly explain what problem the introduction of the State Space Model is aimed at solving, the motivation of the article, and the advantages of Memba.\n2.\tWhile the paper claims high efficiency, specific computational metrics such as runtime and memory usage for different model sizes or input resolutions are not provided for a comprehensive assessment.\n3.\tThe paper does not provide visual experimental results. \n4.\tThe generalization of Voxel Mamba's performance to other datasets or domains beyond autonomous driving scenarios is not discussed, limiting the understanding of its versatility.\n5.\tThe paper lacks a detailed comparison with other state-of-the-art methods in terms of computational efficiency, which is critical for applications with limited computational resources.\n6.\tThe robustness of the model to variations in point cloud density and sparsity is not thoroughly investigated, which could be a concern for outdoor scenes with varying environmental conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1] Clarifying our motivation.**  \n\nThanks for the question. Please kindly refer to our responses to **Q1 of Reviewer kB9g** for details.\n\n**[Q2] Efficiency and performance for different model sizes and resolutions.**\n\nWe appreciate the reviewer's feedback and conducted additional experiments. The results regarding the model size (i.e. feature dimension) and BEV resolution (voxel size) are shown in the following two tables. \n\n|Model Size|L2 mAPH|Vehicle(L1/L2)|Pedestrian(L1/L2)|Cyclist(L1/L2)|Latency(ms)|Memory(GB)|\n|----------|-------|--------------|------------------|--------------|-----------|----------|\n|dim=96|71.3|78.7/70.3|83.6/76.2|76.3/73.6|84|3.6|\n|dim=128|71.6|79.0/70.7|84.0/76.7|76.5/73.7|90|3.7|\n|dim=192|72.4|79.2/70.8|84.7/77.5|77.8/74.9|102|3.9|\n\nThe above table demonstrates a consistent performance improvement as the dimension increases, which correlates with the model scale.\n\n|Voxel Size|L2 mAPH|Vehicle(L1/L2)|Pedestrian(L1/L2)|Cyclist(L1/L2)|Latency(ms)|Memory(GB)|\n|----------|-------|--------------|------------------|--------------|-----------|----------|\n|0.24m|71.5|78.7/70.4|84.0/76.3|76.9/74.0|114|5.4|\n|0.32m|71.6|79.0/70.7|84.0/76.7|76.5/73.7|90|3.7|\n|0.40m|71.4|79.0/70.7|83.6/76.2|76.5/73.8|81|3.2|\n\nThe above table demonstrates that Voxel Mamba remains effective across different voxel sizes. Increasing voxel size reduces input BEV resolution, enhancing efficiency. Voxel Mamba performed best with a voxel size of 0.32m.\n\n**[Q4] Generalization of Voxel Mamba.**\n\nThanks for your question. Please kindly refer to the **shared response Q3**.\n\n**[Q5] More comparison with other state-of-the-art methods.**\n\nWe would like to draw attention to Figure 3 in our manuscript, which provides a comprehensive comparison of our method with state-of-the-art approaches commonly benchmarked in the field of 3D object detection.\n\n**[Q6] Robustness to density and sparsity.**\n\nWe re-benchmarked the performance in different range intervals on Waymo to assess Voxel Mamba's robustness to density and sparsity. The following table indicates that Voxel Mamba is more robust than DSVT.\n\n| Method         | Overall | [0, 30m) | [30m, 50m) | [50m, inf) |\n|----------------|---------|----------|------------|------------|\n| DSVT           | 69.69   | 83.80    | 67.73      | 49.38      |\n| Voxel Mamba    | 71.61   | 85.51    | 70.31      | 50.60      |\n\n**[Q7] Motivation of downsampling in ASSMs.**\n\nThough space-filling curves can preserve the 3D structure to a certain degree, proximity loss is inevitable due to the dimension collapse from 3D to 1D. As a result, a local snippet of the curve can only cover a partial region in 3D space. Placing all voxels in a single group cannot ensure that the effective receptive field (ERF) could cover all voxels. Therefore, we introduce downsampling (hierarchy of state space structures) and consequently improve the ERF of the model.\n\n**[Q8] Clarification on pipeline.**\n\nPlease kindly refer to our responses to Q4 of Reviewer kB9g for details.\n\n**[Q9] Clarification on title.**\n\nWe respectfully disagree with your opinion. To clarify, while 3D object detection is often associated with point clouds, it is not exclusively a point cloud-based task.\n\n**[Q10] Computational complexity with point densities.**\n\nRefer to **shared responses** for details. With increased point densities (4-frame), Voxel Mamba is 20ms faster than DSVT, which is more pronounced in 1-frame and indicates our method's efficiency.\n\n**[Q11] Advantage over Transformers and Spconv.**\n\nFor details on Transformers, please kindly see **Q1 of Reviewer kB9g**. It is important to note that traditional CNNs are typically avoided in 3D detection due to slow inference on sparse data. Compared to SparseConv, Voxel Mamba has significantly larger ERFs, enabling better capture of long-range dependencies in point clouds. Additionally, our method is more deployment-friendly on autonomous driving than previous methods since no padding tokens are needed.\n\n**[Q12] Motivation of IWPE.**\nThe window-based position embedding has been proven effective in previous window-based transformers, as it retains the proximity of voxels within a local window. As a group-free model, we do not explicitly partition the voxels into windows. Instead, we implicitly encode their positions within the window as additional information in the voxel sequence, which significantly enhances the SSM model's ability to extract useful information from the context.\n\n**[Q13] Dynamic object.**\n\nAs indicated in Table 1 of our manuscript, all our experiments were conducted in a single-frame setting. Consequently, the inputs do not include dynamic objects, as each frame is processed independently. To address the concern about non-static elements, we report the multi-frame results in the **shared response**.\n\n**[Q14] Degrade with truncation.**\nTo evaluate Voxel Mamba's robustness to truncation, we compared it with DSVT under various truncation levels using the cutout methods in [1]. The table below shows the mAP of Voxel Mamba and DSVT on the nuScene dataset with truncation severities ranging from 1 to 5. One can see that Voxel Mamba consistently outperforms DSVT across all truncation densities, indicating our method's superior robustness.\n\n| Method        | Clean | Severity 1 | Severity 2 | Severity 3 | Severity 4 | Severity 5 |\n|---------------|-------|------------|------------|------------|------------|------------|\n| DSVT          | 66.4  | 65.1       | 64.7       | 63.1       | 62.2       | 60.5       |\n| Voxel Mamba   | 67.5 (+1.1) | 66.4 (+1.3) | 65.8 (+1.1) | 64.9 (+1.8) | 63.5 (+1.3) | 61.8 (+1.3) |\n\n[1] Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving. CVPR23\n\n**[Q15] Adversarial attacks, dataset overfitting, and multi-task learning**\n\nThanks for your suggestion. In this paper, we aim to establish a strong backbone in 3D object detection, and those issues will be discussed in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Rv0HLdoZK", "reviewer": "Reviewer_rV2B", "summary": "The paper focuses on a new architecture, Voxel Mamba, for 3D object detection using point clouds. It introduces a group-free approach utilizing state space models (SSMs) to address the limitations of previous voxel-based methods like small receptive fields and inefficient grouping. Voxel Mamba leverages asymmetrical SSMs for handling multi-scale contexts and employs space-filling curves for efficient data serialization. They have evaluated the method on the Waymo and nuScenes datasets.", "questions": "1. How does the model perform under different weather conditions or with varying light levels, which are common challenges in outdoor 3D object detection?\n2. What specific modifications would be necessary to adapt this model for indoor use, where point cloud data might come from different types of sensors or have different characteristics like higher density?\n3. Can Voxel Mamba operate in a real-time setting given its computational demands? What are the latency metrics when deployed in such environments?", "limitations": "See the above \"Questions\" and \"Weaknesses\" parts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Integrating group-free state space models and space-filling curves significantly advances traditional voxel-based methods. This allows for better handling of long-range dependencies and multi-scale contexts.\n2. The manuscript provides comprehensive empirical evidence, showing improvements in accuracy and efficiency on prominent benchmarks (nuScenes and Waymo). \n3. The use of the Hilbert curve for input serialization and the introduction of asymmetrical state space models are well-explained and demonstrate a deep understanding of the challenges in 3D object detection.", "weaknesses": "1. There are typos, e.g., Line 293: \"finner\" should be \"finer\"; \n2. The manuscript depth, while comprehensive, might be overwhelming for readers unfamiliar with the domain, particularly the advanced concepts like Hilbert curves and asymmetrical state space models. It also might pose challenges in implementation, especially for teams with limited expertise in these areas.\n3. The Voxel Mamba might require substantial computational resources, e.g., GPU memory and processing power, which could limit its deployment in constrained environments.\n4. The complexity of the Voxel Mamba model doesn't yield proportionately significant performance improvements. Voxel Mamba achieves superior performance on nuScenes and Waymo. However, while statistically significant, the improvement margins might be weighed against the increase in model complexity, implementation difficulty, and computational resource requirements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1, Q2] Manuscript refinement.**\n\nWe apologize for the typos and presentation issues. We promise to thoroughly proofread the entire manuscript and correct the grammatical and typographical errors as much as possible. Considering that some concepts may be difficult for newcomers to this field, we will include more intuitive explanations and visualizations to aid their understanding. For sure we will release our code and pre-trained models to provide detailed implementation.\n\n**[Q3, Q4] Computational complexity.**\n\nFirstly, it's important to clarify that Voxel Mamba's computational complexity significantly differs from Mamba in language models. In our implementation, its layers and parameters are carefully aligned with standard 3D object detectors.\n\nBesides, in Table 5(e) and Figure 3 of the main paper, we compared our method with previous state-of-the-art (SOTA) methods in terms of latency, performance, and GPU memory. Here, we provide a more detailed comparison between Voxel Mamba and previous SOTA serialization-based methods. Actually, Voxel Mamba has comparable to or even lower computational demands than other SOTA serialization-based methods. As shown in the following table, Voxel Mamba differs from DSVT only in the 3D backbone but outperforms DSVT by +1.5 L2 mAPH with less GPU memory and inference latency. All the results are tested on the same device.\n\n| Ablation       | Type     | Backbone      | Memory (GB) | Latency (ms) | L2 mAPH    |\n|----------------|----------|---------------|-------------|--------------|------------|\n| SST            | Pillar   | Transformers  | 6.8         | 71           | 64.6       |\n| VoxSet         | Pillar   | Transformers  | 5.3         | 73           | 67.7       |\n| Voxel Mamba    | Pillar   | SSMs          | 3.5         | 66           | **71.2**   |\n| DSVT           | Voxel    | Transformers  | 4.2         | 94           | 72.1       |\n| Voxel Mamba    | Voxel    | SSMs          | 3.7         | 90           | **73.6**   |\n\n**[Q5] Robustness to weather and light.**\n\nThanks for the constructive suggestion. Adverse weather can cause scattered points due to attenuation and backscattering. To evaluate Voxel Mamba's robustness to adverse weather, we simulated various fog densities using the fog augmentation method in [1] on the point clouds during inference. The table below shows the mAP results on the nuScene dataset with fog severity from 1 to 5 (heavy). One can see that Voxel Mamba consistently outperforms DSVT across all fog densities. Moreover, with the increase of severity, the performance advantage of Voxel Mamba increases, indicating our method's superior robustness to adverse weather.\n\n| Method        | Clean | Severity 1 | Severity 2 | Severity 3 | Severity 4 | Severity 5 |\n|---------------|-------|------------|------------|------------|------------|------------|\n| DSVT          | 66.4  | 65.6       | 63.4       | 55.3       | 47.2       | 25.1       |\n| Voxel Mamba   | 67.5 (+1.1) | 66.9 (+1.3) | 64.9 (+1.5) | 58.0 (+2.7) | 50.7 (+3.5) | 26.8 (+1.7) |\n\n[1] Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather. ICCV21\n\n**[Q6] Indoor semantic segmentation.**\n\nThanks for the question. Please kindly refer to **shared response Q3** for details. \n\n**[Q7] Real-time Voxel Mamba.**\n\nIt is a common practice to optimize 3D detection algorithms for deployment in autonomous driving systems. Frameworks such as TensorRT are widely used to enhance the memory efficiency and inference speed. Considering that the frequency of outdoor LiDAR sensors ranges from 5 to 20 Hz, the optimized Voxel Mamba can indeed meet the real-time requirements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for the valuable comments and suggestions. We first address the common concerns, followed by the detailed responses to each reviewer separately. We hope our responses can clarify the reviewers' concerns and make our contributions clearer.

**Q1. The latency comparison between group-free and group-based operations**   

We perform a detailed comparison of the Hilbert Input Layer (HIL) and DSVT Input Layer across different point cloud sizes and densities. To ensure that the distributions of LiDAR sensor data can be well described, we stack frames to gradually increase the point cloud density. The following table shows the runtime of serialization with the same voxel size (0.32m, 0.32m, 0.1875m) on the Waymo open dataset. One can see that compared to window-based grouping operations, our group-free approach is 13.74ms faster in single-frame scenarios. Moreover, the advantage of HIL becomes more pronounced with the increase of point cloud density.  
 
| Method | 1-frames | 2-frames | 3-frames | 4-frames |
|--------|----------|----------|----------|----------|
| X/Y-Axis Grouping in DSVT | 19.41 ms | 32.30 ms | 34.10 ms | 36.78 ms |
| HIL in Voxel Mamba | 5.67 ms | 7.74 ms | 8.20 ms | 8.52 ms |

**Q2. Multi-frame setting performance**  

| Method | Data | Latency (ms)        | L2 mAPH | Vehicle (L1) | Vehicle (L2) | Pedestrian (L1) | Pedestrian (L2) | Cyclist (L1) | Cyclist (L2)  |  
|--------|------|--------------------|-------------|------------------|---|---------------------|---|-------------------|---|  
| DSVT-4frames | 20% | 158.6 | 76.1 / 74.8 | 80.8 / 80.3 | 72.8 / 72.4 | 85.1 / 82.3 | 78.3 / 75.6 | 79.8 / 78.9 | 77.3 / 76.4 |
| DSVT-4frames | 100% | 158.6 | 76.9 / 75.6 | 81.8 / 81.4 | 74.1 / 73.6 | 85.6 / 82.8 | 78.6 / 75.9 | 80.4 / 79.6 | 78.1 / 77.3 |
| **Voxel Mamba-4frames** | 20% | 138.7 | **77.2 / 75.9** | 81.9 / 81.5 | 74.0 / 73.6 | 86.7 / 84.0 | 80.0 / 77.4 | 80.0 / 79.0 | 77.5 / 76.6 |

The above table compares the performance of DSVT and our Voxel Mamba under the multi-frame setting. One can see that Voxel Mamba significantly outperforms DSVT, indicating its superior ability to leverage temporal and spatial information across frames. Notably, Voxel Mamba trained on 20\% training data surpasses DSVT trained on the 100\% data.

**Q3. Indoor Semantic Segmentation**  

To further demonstrate the general applicability of Voxel Mamba, we extended our experiments to the indoor 3D semantic segmentation task. We adopt the widely used encoder-decoder structure, with both components consisting of ASSMs. Following previous works, we partition the encoder into five stages, with the number of blocks in each stage being {2, 2, 2, 6, 2}. Downsampling is performed between each stage to reduce spatial dimension progressively. The feature dimensions of the blocks across these five stages are {32, 64, 128, 256, 512}. The downsampling rates for ASSMs' backward branches in each encoder stage are {1, 1, 1, 2, 2}. Our method is implemented based on the open-source framework Pointcept. The table below shows the semantic segmentation performance on Scannet. Our method outperforms Swin3D by 90.8% mIoU. Please note that due to limited time, this is just preliminary results. We believe better results can be obtained with refined design and implementation.

| Method                   | Val mIoU (%) |
|--------------------------|--------------|
| Stratified Transformer   | 74.3         |
| OctFormer                | 75.7         | 
| Swin3D                   | 76.3         |
| Voxel Mamba              | 77.1         |

Meta-review: This paper introduces an interesting adaptation of the Mamba architecture for the 3D detection task. The accuracy of the target task is impressive, and all readers appreciate the authors' contribution. Minor concerns regarding computation overhead, GPU memory consumption associated with Hilbert input layer (HIL), technical details, contribution analysis on the group-free strategy, additional comparison, typos, and explanation of the key ideas were raised during the rebuttal phase. The authors provided solid feedback to relieve the concerns. As a result, all reviewers recommend paper acceptance.
Decision: Accept (spotlight)