Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection

Authors: guowen  zhang
lue  fan
chenhang  he
zhen  lei
zhaoxiang  zhang
lei  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 13:36:09

Abstract: Serialization-based methods, which serialize the 3D voxels and group them into multiple sequences before inputting to Transformers, have demonstrated their effectiveness in 3D object detection. However, serializing 3D voxels into 1D sequences will inevitably sacrifice the voxel spatial proximity. Such an issue is hard to be addressed by enlarging the group size with existing serialization-based methods due to the quadratic complexity of Transformers with feature sizes. Inspired by the recent advances of state space models (SSMs), we present a Voxel SSM, termed as Voxel Mamba, which employs a group-free strategy to serialize the whole space of voxels into a single sequence. The linear complexity of SSMs encourages our group-free design, alleviating the loss of spatial proximity of voxels. To further enhance the spatial proximity, we propose a Dual-scale SSM Block to establish a hierarchical structure, enabling a larger receptive field in the 1D serialization curve, as well as more complete local regions in 3D space. Moreover, we implicitly apply window partition under the group-free framework by positional encoding, which further enhances spatial proximity by encoding voxel positional information. Our experiments on Waymo Open Dataset and nuScenes dataset show that Voxel Mamba not only achieves higher accuracy than state-of-the-art methods, but also demonstrates significant advantages in computational efficiency. The source code is available at https://github.com/gwenzhang/Voxel-Mamba.

Main paper: 
 introduction: 
 LiDAR-based 3D object detection from point clouds plays an important role in applications of autonomous driving [19, 4] , virtual reality [47] , and robots [45] . The sparsely, unevenly and irregularly distributed point cloud data make the efficient and effective 3D object detection a very challenging task. To address these long-standing challenges, researchers have recently proposed several strategies to improve the model architecture. One strategy is to switch from PointNet-based models [49, 79, 56] to sparse convolutional neural network (SpCNN)-based models [70, 54, 10, 14, 55, 27] in order for more effective feature extraction. However, the sparse convolution is unfriendly for deployment and optimization, requiring tremendous engineering efforts. Therefore, another strategy is to switch from SpCNN to serialization-based Transformers to address this issue [13, 65, 38, 69, 42] . These methods usually group non-empty 3D voxels into multiple short sequences by serialization techniques such as window partition [65, 13, 38] , Z-shape sorting [66] , and Hilbert sorting [69] , as shown in Figs. 1 (a ) and (b), where a sequence is a group of voxels to be processed by Transformer layers.However, the serialization of voxels will inevitably sacrifice their spatial proximity. Some neighboring voxels can be far apart from each other after serialization, as illustrated by the two red points in Fig. 1 (b) . Such a loss of proximity is difficult to be addressed in the existing serialization methods [69, 66, 38, 65, 13] because the group size is limited by the quadratic complexity of Transformers. This issue becomes even worse when neighboring voxels are grouped into different groups. Inspired by the recent success of State Space Models (SSMs) [21, 20, 59, 18, 82, 36] in language and vision, in this work we propose a simple yet effective group-free design to address the loss of proximity. Specifically, we introduce a Voxel SSM, termed as Voxel Mamba, for 3D object detection from point cloud. The linear computational complexity of SSMs makes it feasible to treat all voxels as a single group and sort them into a single sequence. This results in a group-free modeling of voxels, which is more efficient and deployment-friendly than previous methods since no padding tokens are needed. Nonetheless, even we can sort all voxels into a group-free sequence, it cannot be ensured that all of them are within an effective receptive field.To enhance the spatial proximity of Voxel Mamba, we further propose two modules with it. The first is the Dual-scale SSM Block (DSB) by introducing the downsampling operations in SSMs. In specific, the forward SSM branches process the high-resolution voxel features, while the backward branches extract features from the low-resolution representation. In this way, we integrate the hierarchical design with the bidirectional design in a more economical way. More importantly, the hierarchy brings a larger effective receptive field for the serialized sequence so that the spatial proximity in local 3D regions can be enhanced. The second module we introduced is the Implicit Window Partition (IWP). The window partition is a widely used strategy in previous methods [13, 65] to enhance the proximity of voxels inside a window. However, it impedes the proximity of voxels across windows and contradicts with our group-free principle. We therefore propose an implicit window partition scheme to embrace its strengths while discarding its weaknesses. In specific, we encode the voxel positions inside and across windows into embeddings for feature learning without explicitly conducting spatial window partition. In this way, better voxel proximity can be achieved under our group-free design with minimal computational cost.Our contributions are summarized as follows:• We propose Voxel Mamba, a group-free backbone for voxel-based 3D detection. Voxel Mamba abandons the grouping operation and serializes voxels into one single sequence, enabling better efficiency.• To mitigate the loss of spatial proximity due to serialization, we propose the Dual-scale SSM Block (DSB) and the Implicit Window Partition (IWP) to enhance the spatial proximity preservation of Voxel Mamba.• Our method achieves superior performance to previous state-of-the-art methods on the large-scale Waymo Open dataset [60] and nuScenes [2] datasets.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: [NA] Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 3D Object Detection from Point Clouds. There are two major point cloud representations for 3D object detection, i.e., point-based and voxel-based ones. As in PointNet [50, 51] , point-based methods [49, 48, 56, 52, 37] directly extract geometric features from small regions of raw points. However, those methods suffer from low inference efficiency and limited context features. Voxelbased methods [54, 76, 70, 27, 79, [15] [16] [17] convert raw points into regular grids through voxelization and then process them with sparse convolution [70] or Transformers [13, 25, 65] . Voxel-based methods are currently the main stream for 3D object detection. In terms of model architecture, voxel-based methods can be categorized into two groups, i.e., SpCNN-based [70, 79, 54, 55, 8, 10] and Transformers-based [65, 13, 25, 42, 38, 26] ones. Limited by the high computation complexity, SpCNN-based methods can only use small convolution kernels with restricted receptive fields, and Transformer based methods can only employ a small number of voxels in each group. In contrast, our proposed Voxel Mamba can capture long-range dependencies within the entire sequence while achieving faster inference speed than existing state-of-the-art methods.State Space Models. Inspired by the continuous state space models (SSMs) in control systems, researchers [18, 21, 59, 20] have introduced the SSMs into deep neural networks as a novel alternative to CNNs and Transformers. LSSLs [22] adopts a simple sequence-to-sequence transformation, demonstrating the potential of SSMs. S4 [21] introduces a new parameterization method to SSMs to reduce the computation and memory cost. S5 [59] employs MIMO SSMs and perform efficient parallel scans based on S4. More recently, Mamba [20] introduces input-dependent SSMs and builds a generic backbone, which is fairly competitive with the well-tuned Transformers. Vision Mamba [82] employs bidirectional SSMs and position embedding to learn global visual context for vision tasks. Vmamba [36] employs a 2D-selective-scan to bridge the gap between 1D scanning and 2D plain traversing. PointMamba [34] is a pioneering work to leverage SSM for point cloud analysis, achieving impressive performance in point cloud object understanding. Subsequently, many SSM-based methods [24, 35, 78, 77, 68] are introduced for point cloud processing. In this paper, we investigate the utilization of SSMs to establish a straightforward yet robust baseline for LiDAR-based 3D object detection in driving scenes.Space-filling Curve. The space-filling curve [43] is a series of fractal curves that can go through each point in a multi-dimensional space without repetition. The classical space-filling curve includes Hilbert curve [29] , Z-order curve [46] , and sweep curve, etc. Those methods can perform dimension reduction while maintaining spatial topology and locality. Many researchers [5, 69, 66, 38, 65, 3, 40] have introduced space-filling curves for point cloud processing. HilbertNet [5] uses the Hilbert curve to collapse 3D structures into 2D space to reduce computation and GPU occupation. PointGPT [3] utilizes the Morton-order curve [44] to introduce sequential properties. OctFormer [66] preserves Z-order during octreelization and adopts octree-attention for efficient context learning. PTV3 [69] streamlines the complex interaction with the space-filling curve serialization. For 3D object detection, some methods [65, 38] employ window sweep curves to group voxel features for parallel computation. We employ the Hilbert curve due to its advantageous characteristic of locality preservation.Point Cloud Grouping. LiDAR point clouds are sparsely and non-uniformly distributed with varying densities. Therefore, existing methods group points or voxels to facilitate parallel computation and reduce complexity. In point cloud analysis, some works [51, 67] use the K nearest neighbor (KNN) method to create groups of query points. However, the heavy computation burden makes KNN hard to scale for outdoor scenes. For 3D object detection, VoTr [42] uses a GPU-based hash table to search neighborhoods and generate fixed-length voxel groups. Window-based Voxel Transformers [13, 61, 65, 38] group voxels by employing a window-based sorting strategy, such as the rotating partition.To reduce the reliance on relative position in grouping operations, some recent works [66, 69, 40] have been proposed to group voxels based on space-filling curves. However, grouping is merely a compromise for computational complexity, which restricts the flow of information and effective receptive field. To tackle this problem, we model the entire voxels into one single sequence and allow each voxel be aware of global context information.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See the limitation in Sec. 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methods: 
 In this section, we present Voxel Mamba, a group-free Voxel State Space Model-based 3D backbone that can be applied to most voxel-based 3D detectors. We first introduce the preliminary concepts associated with our method, followed by the overall architecture of Voxel Mamba. Then, we describe in detail the fundamental components of Voxel Mamba, including the Hilbert Input Layer (HIL), Dual-scale SSM Block (DSB), and Implicit Window Partition (IWP).The state space sequence (SSM) model is a continuous-time latent state model, which maps a 1D input signal x(t) ∈ R L to an output signal y(t) ∈ R L through hidden state h(t) ∈ R N . The system can be represented as the following linear ordinary differential equation:EQUATIONwhere A ∈ R N ×N , B ∈ R N ×1 and C ∈ R 1×N are learnable parameters, and D ∈ R 1 denotes a residual connection.To apply SSM to a discrete sequence, we can discrete the continuous-time SSM with a timescale parameter ∆ [21, 20, 36] . The zero-order hold (ZOH) transformation can be used to discrete the continuous parameters A, B as A = exp(∆A), B = (∆A) -1 (exp(∆A) -I) • ∆A. The discretized version of Eq.( 1) can be written in the following recurrent form:EQUATIONFinally, the convolutional mode can be used for efficient parallel training:EQUATIONwhere L is the length of the input sequence and K ∈ R L is the structured convolution kernel.SSM combines the advantages of convolution and self-attention with near-linear computation and dynamic weights. It demonstrates stronger ability than Transformers in modeling long-range dependencies [21, 20] , which inspires us to develop a group-free framework for point cloud based 3D object detection.An overview of our proposed Voxel Mamba is shown in Figure 2 . As in previous works [65, 74, 31] , Voxel Mamba transforms point clouds into sparse voxels by a voxel feature encoding strategy. Unlike prior Transformer-based methods that perform extensive window partitioning and voxel grouping, in Voxel Mamba we serialize the voxel of the entire scene into a single sequence by using the Hilbert Input Layer (Sec. 3.3). Then, a Dual-scale SSM Block (Sec. 3.4) working on the voxel sequence is proposed, which allows voxels to be processed with a global context. To enlarge the effective receptive fields, DSB adopts a finer-grained perception of the voxel sequence in the forward path, and down-samples the voxels sequence in the backward path. The backward path extracts features from the low-resolution BEV representation, with an increased downsampling factor in deeper blocks. To enhance the spatial proximity in sequences, Voxel Mamba adopts Implicit Window Partition (Sec. 3.5) to preserve 3D positional information in the extracted voxel features, and projects them to a BEV feature map. Our proposed architecture is flexible and can be applied to most existing 3D object detection frameworks.The space-filling curve (e.g., Hilbert [29] and Z-order [46] ), known for preserving spatial locality, is widely used for dimensionality reduction. Space-filling curves, such as the Hilbert shown in Fig. 2 , can traverse all elements in a space without repetition and preserve spatial topology. To improve the voxel proximity in serialization, we propose the Hilbert Input Layer to reorder the voxel sequence.C = {(x, y, z) ∈ R 3 |0 ≤ x, y, z ≤ n}.We map a voxel onto its traversal position h within the Hilbert curve. Specifically, we transform (x, y, z) into its binary format with log 2 n bits. For example, x is converted to (x m x m-1 ...x 0 ), where m = ⌊log 2 n⌋. Then, following [58] , we iterate from x m , y m , z m to x 1 , y 1 , z 1 bits and perform exchanges and inversions to adjust the order of bits. An exchange is conducted when the current bit is 0; otherwise, an invert is conducted. We concatenate all bits as (x m y m z m x m-1 y m-1 z m-1 . . . x 0 y 0 z 0 ) and apply a global 3m-fold Gray decoding [58] on it to obtain the traversal position h. Subsequently, all voxels are sorted into a single sequence based on their traversal position h.In our implementation, we record the traversal position h corresponding to the coordinates of all potential voxels. The voxels are serialized by querying and sorting their traversal positions. We employ a distinct traversal order for each BEV resolution in Dual-scale SSM blocks. Notably, the serialization process only takes approximately 0.7ms for a sequence of length 10 6 .Though space-filling curves can preserve the 3D structure to a certain degree, proximity loss is inevitable due to the dimension collapse from 3D to 1D. As a result, a local snippet of the curve can only cover a partial region in 3D space. As discussed in Sec. 1, placing all voxels in a single group cannot ensure that the effective receptive field (ERF) [41, 12] could cover all voxels. Therefore, in this subsection we introduce the Dual-scale SSM block (DSB) to build a hierarchy of state space structures and consequently improve the ERF of the model.As shown in Fig. 2 , the DSB block is designed with a residual connection [28] , a forward SSM branch and a backward SSM branch. It operates on two serialized voxel sequences generated by the Hilbert Input Layer, enabling a seamless flow of information throughout the voxel sequence. The forward branch processes the original voxel sequence, maintaining high-resolution details. The backward branch, however, operates on a down-sampled voxel sequence derived from a low-resolution BEV representation. This dual-scale path allows DSB to incorporate larger-scale voxel features, enhancing the model's ability to model long dependencies among voxels. Specifically, given a voxel sequence F and its corresponding coordinates C, DSB is computed as:EQUATIONwhere HIL(•) represents the Hilbert Input Layer, FSSM(•) and BSSM(•) denote the forward and backward SSM, LN(•) stands for Layer Normalization, and C ′ is the coordinates of downsampled sparse voxels. Besides, Down(•) and Up(•) refer to the downsampling and upsampling operations, respectively, and IWE(•) means Implicit Window Embedding. Overall, DSB integrates the widely adopted bidirectional design [82, 36] with the hierarchical design, building sufficient receptive field to mitigate the loss of proximity without introducing additional parameters.The window partition strategy is widely used in previous 3D detectors [13, 65] to enhance the voxel proximity. In these methods, the whole field is partitioned into multiple local windows and the voxels within a window form a group. Therefore, the voxels inside a window will have sufficient proximity; however, the voxels in different windows will have minimal proximity. In this section, we aim to introduce the advantages of window partition into our framework while avoiding its weaknesses.To fulfill our goal, we propose an Implicit Window Partition (IWP) strategy. Unlike previous methods, we do not explicitly partition voxels into windows and apply Transformer or SSM within each window. In contrast, we calculate the voxel coordinates inside and across windows, and then encode coordinates to embeddings, termed as Implicit Window Embedding (IWE), which is formulated as:EQUATIONwhere ⌊•⌋ is the floor function, w, h define the window shape, and z, x i , y i are the coordinates of tokens. (x 0 , y 0 ) and (x 1 , y 1 ) represent the coordinates before and after an implicit window shift.The IWE is shared across all layers with the same stride. Thus, its computation cost only comes from shallow MLPs. With IWE, voxels in the serialized 1D curve are aware of their positions and consequently their proximity in 3D space.With the proposed Hilbert Input Layer, DSB and IWP strategies, we build Voxel Mamba, a group-free sparse voxel backbone. The architecture of Voxel Mamba is illustrated in Figure 2 . It comprises N DSB blocks, which are organized into different stages based on their downsampling rates. SpConv [9] is employed to progressively decrease the feature map resolution along the Z-axis in each stage. Before sparse tokens are fed into the BEV backbone, we scatter them into dense BEV features. On the Waymo dataset, we adopt the BEV backbone from Centerpoint-Pillar [74] , and employ the same setting as DSVT [65] for the detection head and loss function. On the nuScenes dataset, we only replace the 3D backbone of DSVT [65] with our Voxel Mamba backbone.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
datasets and evaluation metrics: 
 Waymo Open Dataset contains 230k annotated samples, partitioned into 160k for training, 40k for validation and 30k for testing. Each frame covers a large perception range (150m × 150m). The mean average precision (mAP) and its weighted variant by heading accuracy (mAPH) are used as evaluation metrics. They are further categorized into Level 1 for objects detected by over five points, and Level 2 for those detected with at least one point.nuScenes consists of 40k labeled samples, with 28k for training, 6k for validation and 6k for testing. For 3D object detection, nuScenes employs the mean average precision (mAP) and the nuScenes detection score (NDS) to measure model performance.Our method is implemented based on the open-source framework OpenPCDet [63] . The voxel sizes are defined as (0.32m, 0.32m, 0.1875m) for Waymo and (0.3m, 0.3m, 0.2m) for nuScenes. We stack six DSB blocks, divided into three stages, for the Voxel Mamba backbone network. The downsampling rates for DSBs' backward branches in each stage are {1, 2, 4}. Specifically, we employ SpConv [9] and its counterpart SpInverseConv as downsampling and upsampling operators in the DSB backward branch. On the Waymo dataset, we follow the training schemes in [65, 74] to optimize the model using Adam optimizer with weight decay 0.05, one-cycle learning rate policy, max learning rate 0.0025, and batch size 24 for 24 epochs. On the nuScenes dataset, we follow the training scheme adopted in DSVT [65] . We train Voxel Mamba with a weight decay of 0.05, one-cycle learning rate policy, max learning rate of 0.004, and batch size of 32 for 20 epochs. The voxel features on both datasets consist of 128 channels. All the models are trained on 8 RTX A6000 GPUs. Other settings in the training and inference of Voxel Mamba follow DSVT [65] .Waymo. We first compare Voxel Mamba with state-of-the-art methods on the Waymo Open dataset.Table 1 shows the results on the validation set. Our proposed Voxel Mamba achieves 79.6/73.4 on L1/L2 mAPH, which are +1.4 and +1.5 better than DSVT-Voxel. Since our framework differs from DSVT only on the 3D backbone, it can be concluded that Voxel Mamba has superior ability in capturing voxel features. In comparison with window-based (e.g., DSVT) or curve-based (e.g., PTv3) grouping methods, our group-free method Voxel Mamba consistently delivers better results. Table 2 shows the results on the test split. Voxel Mamba reaches 79.6/74.3 in terms of L1/L2 mAPH, which is even better than the 3-frame setting of PillarNeXt and SST.nuScenes. We then compare Voxel Mamba with previous state-of-the-art methods on nuScenes.Table 3 shows the results on the validation set. Voxel Mamba achieves impressive results with 71.9 NDS and 67.5 mAP, which is +0.5 and +0.8 higher than the previous best method. Compared with DSVT, Voxel Mamba achieves +1.1 higher performance on mAP. The results on the test split are shown in Table 4 . Our method also exhibits the best mAP and NDS.Inference Efficiency. We compare Voxel Mamba with other state-of-the-art methods in inference speed and performance accuracy in Fig. 3 . Notably, Voxel Mamba outperforms DSVT [65] and PV-RCNN++ [54] by at least +1.5 in detection accuracy, while achieving faster speed. Some methods, such as CenterPoint [74] and PointPillar [31] , are faster than Voxel Mamba; however, their accuracy is substantially lower.We further compare Voxel Mamba with previous well-designed architectures (SpCNN, Transformers, and 2D CNN) in GPU memory in Table 5 . Compared with CenterPoint-Pillar, Voxel Mamba requires only an additional 0.5 GB GPU memory but achieves +9.0 higher accuracy in L2 mAPH. While Transformer-based methods like SST [13] and DSVT [65] use group partitioning, they still consume more memory than our group-free Voxel Mamba. All the experiments are evaluated on an NVIDIA A100 GPU with the same environment.To better investigate the effectiveness of Voxel Mamba, we conduct a set of ablation studies by using the nuScenes validation set. We follow OpenPCDet [63] to train all models for 20 epochs.Curves. There are some potential alternatives to Hilbert curve for preserving locality. Here, we compare Hilbert curve with some commonly used space-filling curves (Z-order [66] and window partition [13] ) in 3D detection. As shown in Table 6 (a), without using space-filling curves (i.e., the row of 'Random Curve'), there will be a notable decline in performance, which indicates that spatial proximity is crucial in the group-free setting. By using the Z-order curve and window partition to introduce spatial proximity, the mAP and NDS are much improved. The serialization based on the Hilbert curve can further enhance the model performance.Effectiveness of Each Component. To more clearly illustrate the effectiveness of the different components in Voxel Mamba, we conduct experiments by adding each of them to a baseline, which is set to Centerpoint-Pillar [74] . As shown in Table 6 (b), bidirectional SSMs with a Hilbert-based groupfree sequence can significantly improve the accuracy over the baseline, which validates the feasibility of our group-free strategy. Besides, converting pillar to voxel can enhance much the detector's performance without group size constraints. Voxel Mamba with DSB obtain better performance than the plain bidirectional SSMs. This is because DSB can build larger ERFs and mitigate the loss of proximity. Furthermore, IWE further boosts Voxel Mamba's performance for its capability in capturing 3D position information and increasing voxel proximity.Downsampling Rates of DSB. We evaluate the impact of different downsampling rates in DSB by adjusting the stride {d 1 , d 2 , d 3 } in the backward SSM branch at each stage. d i = 1 means the original resolution is used. The results are shown in Table 6 (c). We see that transitioning from {1,1,1} to {1,2,2} and to {1,2,4} enhances performance due to an enlarged effective receptive field and improved proximity by using larger downsampling rates at late stages. However, DSBs with {2,2,2} or {4,4,4} compromise performance compared to {1,1,1}, indicating that using larger downsampling rates at early stages will lose some fine details. Thus, we set the stride as {1,2,4} to strike a balance between effective receptive fields and detail preservation. Effectiveness of IWE. Table 6 (d) validates the capability of IWE to enhance spatial proximity. We compare IWE with some commonly used positional embedding methods [13, 65] in 3D detection. Absolute position denotes the direct encoding of voxel coordinates using an MLP. The results demonstrate that IWE can significantly improve the detection performance by offering features with rich 3D positional and proximate information.Fig. 4 illustrates the Effective Receptive Fields [41, 12] (ERFs) of window partition-based method DSVT [65] , group-based bidirectional Mamba and our proposed group-free method Voxel Mamba. For clear visualization, all models take pillars as inputs. The group partition in the group-based bidirectional Mamba is configured identically to DSVT. Then, we randomly select voxels of interest from the ground truth bounding box and calculate the ERF at each non-empty voxel position. Subsequently, we merge the ERFs into a single image by taking the maximum value at each voxel location. A wider activation area indicates a larger ERF. From Fig. 4 , we see that Voxel Mamba exhibits a notably larger ERF than DSVT and group-based bidirectional Mamba, which can be attributed to the benefits of group-free operation. The larger ERF can cover a more complete local region and enhance the spatial proximity in 1D sequences. 
conclusion: 
 In this paper, we proposed Voxel Mamba, a group-free SSM-based 3D backbone for point cloud based 3D detection. We first analyzed the proximity loss of group partition in current serialization-based 3D detection methods. By taking the advantage of linear complexity of SSMs, we proposed a group-free strategy to alleviate the loss of spatial proximity in 3D to 1D serialization. We further proposed the DSB block and IWP strategy to build larger effective receptive fields and improve the spatial proximity of our Voxel Mamba framework. Experiments demonstrated that Voxel Mamba achieved state-of-the-art results on Waymo and nuScene datasets. Without elaborated optimization, our model consumed less memory than group-based Voxel Transformer methods, and our group-free strategy was more efficient and deployment-friendly than group partition. Voxel Mamba provided an efficient group-free solution for sparse point clouds for 3D tasks.Limitations. While the proposed Voxel Mamba achieves state-of-the-art performance in point cloud based 3D object detection, it still has some limitations to be further addressed. First, in the Hilbert Input Layer, the curve templates occupy approximately 0.1 GB of GPU memory, which may become substantial as the voxel resolution increases. Besides, a more elaborately designed downsampling and upsampling operation could improve more the model efficiency. We will investigate these problems in future work.Acknowledgments. This work was supported in part by the InnoHK Program.NeurIPS Paper Checklist 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We focus on point cloud-based 3D object detection, which can be applied to autonomous driving. For positive societal impact, our approach enhances the precision of 3D detection, thereby augmenting the safety aspect of autonomous driving. For negative societal impact, false detection results can potentially lead to traffic accidents.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the related original paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix:  


Decision: This paper introduces an interesting adaptation of the Mamba architecture for the 3D detection task. The accuracy of the target task is impressive, and all readers appreciate the authors' contribution. Minor concerns regarding computation overhead, GPU memory consumption associated with Hilbert input layer (HIL), technical details, contribution analysis on the group-free strategy, additional comparison, typos, and explanation of the key ideas were raised during the rebuttal phase. The authors provided solid feedback to relieve the concerns. As a result, all reviewers recommend paper acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a new 3D detection architecture, Voxel Mamba. Voxel Mamba serializes voxels using a Hilbert Input Layer and then applies a forward SSM and a backward SSM at lower resolutions. It achieves state-of-the-art performance on both the Waymo Open Dataset and nuScenes dataset.
strength_weakness: strength_weakness: Please see the questions section.
questions: questions: In table 5 (d), the Hilbert curve is marginally better than the Z-order curve. In 5(c), adding Hilbert improves mAP and NDS by 0.1. 
Curious, what is the latency of generating the Hilbert curve and Z-order Curve. What about the latency of generating Random Curves?

For random curves, do you use the same random curves for all frames or random sample a curve for each frame? Do you use the same random sampled curves for different resolutions, or use the same random curve for all resolutions in the backward SSM?

What would be the expected performance and latency if you flattened the voxels to 1D but used window attention (either shifted or scanned) instead of Mamba? This could help assess the trade-offs between different approaches to handling long sequence lengths.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: The authors introduce Voxel Mamba, a 3D object detection method based on state-space models.  Voxel Mamba introduces a efficient group-free approach to avoid inefficient grouping operations and prevent limitations on the receptive field for 3D object detection in point clouds. The authors also present Asymmetrical State Space Models (ASSMs) to capture multi-scale context information, offering each voxel data-dependent global contexts. Voxel Mamba enables voxel interactions throughout the scene and captures cloud locality using space-filling curves. Extensive experiments validate its performance, positioning Voxel Mamba as a viable alternative for 3D object detection.
strength_weakness: strength_weakness: 1. Motivation can be improved. The authors must clarify why SSM is essential for their motivation and quantify the advantages it offers over transformers for this task. The question in lines 55-56 does not demonstrate SSM's superiority. The author's design for SSM is also hardly inspiring for the 3D detection community.
2. Some of the method's design confuse me. Traversing the downsampled feature maps with a Hilbert curve in the opposite direction can misalign the feature receptive fields. Does this challenge model training with varying inputs, even with IWPE included? Additionally, given its use of multi-scale features, how much does ASSM outperform FPN?
3. The experimental comparisons may be inadequate. The authors present a pillar version for Voxel Mamba but fail to compare its performance and efficiency against current state-of-the-art pillar-based methods in the main table.
4. The experiment's details should be more clearly documented. For example, does Voxel Mamba adopts IoU-rectification scheme? The comparative setup in the ablation study could have been accounted for in more detail and analyzed why the method designed by the authors was effective.
5. How does Voxel Mamba integrate with multi-frame temporal 3D detection?
questions: questions: See Weakness. Besides, there are few other suggestions:

1. Could the authors provide more description of Window Sweep in Table 5(c)? Does it include the Shifted Regional Grouping Shifted Token Groups operation used in SST?
1. Is a Linear for the up dimension missing from the SSM part of Fig. 2?
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper proposes a novel backbone named Voxel Mamba for point cloud 3D detection. Different from previous methods that group the voxels into fixed-length sequences through padding, this paper adopt an interesting group-free strategy to sort all voxels into a single sequence through the space-filling curve. The paper also proposes the Asymmetrical State Space Models (ASSMs) and Implicit Window-based Position Embedding (IWPE) to enlarge the receptive fields of Voxel SSMs. 
Experiments are mostly based on challenging outdoor datasets, the Waymo Open Dataset and nuScene. Experiments demonstrates Voxel Mamba surpasses previous methods.
strength_weakness: strength_weakness: 1.	The experiments of model architectures are a bit insufficient. In this paper, the authors extend Mamba to bidirectional Mamba (line 68) and further to Voxel Mamba (line 74). However, there are no experiments demonstrating the performance gains from these upgrades. As Voxel Mamba is based on Mamba, it would be necessary to report the accuracy of apply group-free bidirectional Mamba and Mamba with the same SSM parameters or layers.
2.	Given that the group-free strategy is a significant contribution of this paper, it is necessary to include more analysis and comparisons between group-free and group-based operations. For example, the author cloud provide a detailed latency comparison between window-based grouping (such as dsvt and sst) and HIL. Besides, on Line 188, the authors mention that the mapping processes takes 0.7ms. However, the latency is likely to vary with different sequence lengths. Please provide a clarification regarding the timing.
3.	Lack of comparison with non-group-free Mamba. Although this work provides quantitative and visualization comparison between group-free and group-based (dsvt) methods. It is necessary to provide the ERFs and detection accuracy for non-group-free Voxel Mamba. As a linear RNN models, what difference does Mamba introduce compared to Transformers.
4.	The analysis of the ablation on ASSM and HIL is a little vague. Specifically, it is unclear why performances drop under the downsampling setting {2,2,2} and {4,4,4} in Table 5(b). Besides, the window sweep, Z-order, and Hilbert methods in Table 5(d) exhibit similar performance. Can I infer that the form of the space-filling curve does not significantly affect the accuracy? Please provide a discussion on them, which would be helpful to improve the quality of this paper.
5.	Hilbert Input Layer (HIL) sort the voxels by querying the space-filling curve template. As presented in Sec. 3.3, form my understanding, the template needs cover the whole scenes. My concern is the amount of GPU memory required by the template under the current voxel size setting and its contribution to the total memory consumption. And for a more fine-grained size such as (0.1m, 0.1m, 0.15m) in CenterPoint.
questions: questions: Please see Weaknesses above. Overall, the proposed group-free strategy has better efficiency than previous group-based operations without token padding. Voxel Mamba also demonstrates better performance than existing sparse convolution or transformer structures. As such, I recommend acceptance at this stage.
limitations: limitations: Limitations have been included
paper_summary: paper_summary: The paper introduces a novel architecture for 3D object detection, addressing the limitations of current methods which rely on grouping operations. The proposed Voxel Mamba leverages State Space Models (SSMs) to capture long-range dependencies and process the entire scene in a single sequence, avoiding inefficient grouping. It employs a group-free strategy, an Asymmetrical State Space Models (ASSMs) block for multi-scale context, and an Implicit Window-based Position Embedding (IWPE) to maintain token locality. Experiments on the Waymo Open and nuScenes datasets demonstrate superior performance in accuracy and efficiency over established methods.
strength_weakness: strength_weakness: 1.	The introduction does not clearly explain what problem the introduction of the State Space Model is aimed at solving, the motivation of the article, and the advantages of Memba.
2.	While the paper claims high efficiency, specific computational metrics such as runtime and memory usage for different model sizes or input resolutions are not provided for a comprehensive assessment.
3.	The paper does not provide visual experimental results. 
4.	The generalization of Voxel Mamba's performance to other datasets or domains beyond autonomous driving scenarios is not discussed, limiting the understanding of its versatility.
5.	The paper lacks a detailed comparison with other state-of-the-art methods in terms of computational efficiency, which is critical for applications with limited computational resources.
6.	The robustness of the model to variations in point cloud density and sparsity is not thoroughly investigated, which could be a concern for outdoor scenes with varying environmental conditions.
questions: questions: 1.	The role of using downsampling sequences in the backward SSM sequence in Figure 2 is not clearly explained in the paper.What does downsampling do?
2.	Are there any improvements made to the object detection baseline, including loss function and pipeline?
3.	Is there any redundancy in your title? "based 3D Object Detection" itself is a task of 3D point cloud, why is it written "for Point Cloud" in front of it?
4.	How does the computational complexity of Voxel Mamba scale with the size and density of the point cloud data?
5.	What are the specific advantages of using SSMs over traditional CNN or Transformer architectures in the context of 3D object detection from point clouds?
6.	Can the authors provide more details on the design choices behind the Implicit Window-based Position Embedding and its impact on the model's receptive field?
7.	How does Voxel Mamba handle dynamic objects or scenes with non-static elements, which are common in autonomous driving scenarios?
8.	How does the performance of Voxel Mamba degrade with respect to various levels of occlusion or truncation of objects in the point cloud data?
limitations: limitations: 1.	The paper does not mention the issue of computing resource consumption, or whether higher computing resources are required.
2.	The paper does not explicitly discuss the model's generalization ability, that is, how well the model performs in new scenarios or under different environmental conditions. The paper's focus on autonomous driving scenarios may limit the general applicability of the findings to other fields where 3D object detection is relevant.
3.	The reliance on large-scale datasets for training and evaluation might not fully capture the diversity of real-world scenarios, potentially limiting the model's robustness.
4.	The potential impact of adversarial attacks or model overfitting to the specific characteristics of the training datasets is not discussed, which is important for model security and reliability.
5.	The performance of Voxel Mamba on the Waymo Open and nuScenes datasets may not directly translate to other types of datasets, potentially limiting the model's universal applicability.
6.	It is not clear whether the current Voxel Mamba model can be extended to a multi-task learning framework, such as simultaneously performing object detection, classification, and pose estimation tasks.
paper_summary: paper_summary: The paper focuses on a new architecture, Voxel Mamba, for 3D object detection using point clouds. It introduces a group-free approach utilizing state space models (SSMs) to address the limitations of previous voxel-based methods like small receptive fields and inefficient grouping. Voxel Mamba leverages asymmetrical SSMs for handling multi-scale contexts and employs space-filling curves for efficient data serialization. They have evaluated the method on the Waymo and nuScenes datasets.
strength_weakness: strength_weakness: 1. There are typos, e.g., Line 293: "finner" should be "finer"; 
2. The manuscript depth, while comprehensive, might be overwhelming for readers unfamiliar with the domain, particularly the advanced concepts like Hilbert curves and asymmetrical state space models. It also might pose challenges in implementation, especially for teams with limited expertise in these areas.
3. The Voxel Mamba might require substantial computational resources, e.g., GPU memory and processing power, which could limit its deployment in constrained environments.
4. The complexity of the Voxel Mamba model doesn't yield proportionately significant performance improvements. Voxel Mamba achieves superior performance on nuScenes and Waymo. However, while statistically significant, the improvement margins might be weighed against the increase in model complexity, implementation difficulty, and computational resource requirements.
questions: questions: 1. How does the model perform under different weather conditions or with varying light levels, which are common challenges in outdoor 3D object detection?
2. What specific modifications would be necessary to adapt this model for indoor use, where point cloud data might come from different types of sensors or have different characteristics like higher density?
3. Can Voxel Mamba operate in a real-time setting given its computational demands? What are the latency metrics when deployed in such environments?
limitations: limitations: See the above "Questions" and "Weaknesses" parts.

Comments: title: We are open to any further discussion
We are open to any further discussion.
comment: We really appreciate for this reviewer's recognition on the innovation of our work. 
Yes, the contextual information is important in point cloud object detection. Since 3D point clouds lack distinctive textures and appearances, the detectors will more heavily rely on contextual information for learning semantic information. We have developed several designs to enhance Voxel Mamba's ability to capture contextual information in the paper. To increase contextual information for voxel tokens, we proposed a group-free strategy to serialize the whole space voxels into a single sequence. Besides, we also integrated a multi-scale design to broaden the effective receptive field and capture context information at various scales.

We used RNN to address the inefficiency of Transformers when expanding input context. In the large context setting, we adopted Mamba (RNN) design to enhance the efficiency, as latency is crucial in outdoor 3D object detection for autonomous driving. As shown in the table following, RNNs have a significant speed advantage over Transformers with the longer input voxel sequences, which often reach the order of $10^5$ in 3D detection scenarios.

| Method     | 1K       | 2K       | 4K       | 8K        | 16K        |
|------------|----------|----------|----------|-----------|------------|
| Transformer| 0.47 ms  | 1.61 ms  | 5.74 ms  | 26.02 ms  | 114.20 ms  |
| SSM        | 0.41 ms  | 0.43 ms  | 0.50 ms  | 0.61 ms   | 0.94 ms    |

If there is any misinterpretation of your question, we are open to further discussion
The author's rebuttal addressed many of the issues I raised. However, I still have some doubts about this article. Although the author has further elaborated on the motivation, there are still some areas of doubt. For visual perception tasks such as object detection, is contextual information really important, and what is the interpretability of using RNN type network frameworks. Overall, the paper is quite innovative, so I'm inclined to rate it as boardline acceptable.
Thank you very much for your dedicated efforts to review our paper and suggestions for our work.
We sincerely thank this reviewer for the feedback and support. Your constructive comments are valuable for us to improve the quality of this work. Following your suggestions, we will polish the paper and add necessary clarifications in the revision.
We sincerely thank this reviewer for finding our responses useful and raising the score. We will incorporate additional experiments and detail explanations in the revised manuscript, as suggested by this reviewer. Thanks again for your constructive comments!
Thanks for your careful responses and clarification, which have effectively addressed my concerns and misunderstandings. I have read the authors' responses and other reviewers' comments.

It is good to see that the unclear experiments' detail and some ablation study analysis have been supplemented. The additional experiment and comparison, which demonstrate the efficiency of Voxel Mamba's structure and SSM, have been provided during the rebuttal. 

I strongly encourage the author to incorporate the additional experiments, especially the comparison between Voxel Mamba and DSVT input layer, from the rebuttal into the revised manuscript.

Given these improvements and clarification, I have an overall favorable opinion and upgrade my rating.
Thank you for providing additional results. I will keep my rating.
Thanks for the author’s comprehensive response. These address my concerns. The additional experiments show the advantages of group-free strategy and the effectiveness of ASSMs. The curve templates do not significantly impact GPU memory consumption. This work presents a meaningful advancement in 3D detection.
Dear reviewer:

Thank you so much for reviewing our paper. We hope our explanation could resolve your concern. During this discussion phase, we welcome any further comments or questions regarding our response and main paper. If there requires further clarification, please do not hesitate to bring them up. We will promptly address and resolve your inquiries. We are looking forward to your feedback.
Dear reviewer:

Thank you so much for reviewing our paper. We hope our explanation could resolve your concern. During this discussion phase, we welcome any further comments or questions regarding our response and main paper. If there requires further clarification, please do not hesitate to bring them up. We will promptly address and resolve your inquiries. We are looking forward to your feedback.
Dear Reviewer,

This is a gentle reminder to please review the rebuttal provided by the authors. Your feedback is crucial to the decision-making process. Please consider updating your score after reading the rebuttal.

Thank you for your help with the NeurIPS!

Best, Your AC
Dear Reviewer,

This is a gentle reminder to please review the rebuttal provided by the authors. Your feedback is crucial to the decision-making process. Please consider updating your score after reading the rebuttal.

Thank you for your help with the NeurIPS!

Best, Your AC
Dear Reviewer,

This is a gentle reminder to please review the rebuttal provided by the authors. Your feedback is crucial to the decision-making process. Please consider updating your score after reading the rebuttal.

Thank you for your help with the NeurIPS!

Best, Your AC
Dear Reviewer,

This is a gentle reminder to please review the rebuttal provided by the authors. Your feedback is crucial to the decision-making process. Please consider updating your score after reading the rebuttal.

Thank you for your help with the NeurIPS!

Best, Your AC
Dear Reviewer,

This is a gentle reminder to please review the rebuttal provided by the authors. Your feedback is crucial to the decision-making process. Please consider updating your score after reading the rebuttal.

Thank you for your help with the NeurIPS!

Best, Your AC