LION: Linear Group RNN for 3D Object Detection in Point Clouds
Authors: Zhe Liu
Jinghua Hou
Xinyu Wang
Xiaoqing Ye
Jingdong Wang
Hengshuang Zhao
Xiang Bai
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Object Detection
Linear RNN
Primary area: machine_vision

Abstract: The benefit of transformers in large-scale 3D point cloud perception tasks, such as 3D object detection, is limited by their quadratic computation cost when modeling long-range relationships. In contrast, linear RNNs have low computational complexity and are suitable for long-range modeling. Toward this goal, we propose a simple and effective window-based framework built on Linear group RNN (i.e., perform linear RNN for grouped features) for accurate 3D object detection, called LION. The key property is to allow sufficient feature interaction in a much larger group than transformer-based methods. However, effectively applying linear group RNN to 3D object detection in highly sparse point clouds is not trivial due to its limitation in handling spatial modeling. To tackle this problem, we simply introduce a 3D spatial feature descriptor and integrate it into the linear group RNN operators to enhance their spatial features rather than blindly increasing the number of scanning orders for voxel features. To further address the challenge in highly sparse point clouds, we propose a 3D voxel generation strategy to densify foreground features thanks to linear group RNN as a natural property of auto-regressive models. 
Extensive experiments verify the effectiveness of the proposed components and the generalization of our LION on different linear group RNN operators including Mamba, RWKV, and RetNet. Furthermore, it is worth mentioning that our LION-Mamba achieves state-of-the-art on Waymo, nuScenes, Argoverse V2, and ONCE datasets. Last but not least, our method supports kinds of advanced linear RNN operators (e.g., RetNet, RWKV, Mamba, xLSTM and TTT) on small but popular KITTI dataset for a quick experience with our linear RNN-based framework.

LION: Linear Group RNN for 3D Object Detection in Point Clouds
Zhe Liu Jinghua Hou Xinyu Wang Xiaoqing Ye Jingdong Wang",
Hengshuang Zhao? Xiang Bai
Huazhong University of Science and Technology
The University of Hong Kong BBiidu Inc., China
https:/ petps://happinesssl/ waapeesss...gghubb wttthbb..o/pojjectee
Abstract
The benefit of transformers in large-scale 3D point cloud perception tasks, such as 3D object detection, is limited by their quadratic computation cost when modeling long-range relationships. In contrast, linear RNNs have low computational com- plexity and are suitable for long-range modeling. Toward this goal, we propose a simple and effective window-based framework built on LInear grOup RNN (i.e., perform linear RNN for grouped features) for accurate 3D object detection, called LION. The key property is to allow sufficient feature interaction in a much larger group than transformer-burt based methods. However, effectively applying linear group RNN to 3D object detection in highly sparse point clouds is not trivial due to its lim- simply introduce
itation in handling spatial modeling- To tackle this problem, 3D spatial feature descriptor and integrate into the linear group RNN operators to enhance their spatial features rather than blindly increasing the number of scanning orders for voxel features. To further address the challenge in highly sparse point clouds, we propose 3D voxel generation strategy to densify foreground features thanks to linear group RNN as natural property of auto-regressive models. Ex- tensive experiments verify the effectiveness of the proposed components and the generalization of our LION on different linear group RNN operators including Mamba, RWKV, and RetNet. Furthermore, is worth mentioning that our LION- Mamba achieves state-of-the the-rrt on Waymo, nuScenes, Argoverse V2, and ONCE datasets. Last but not least, our method supports kinds of advanced linear RNN operators (e.g., RetNet, RWKV, Mamba, XLSTM and TTT) on small but popular KITTI dataset for a quick experience with our linear RNN-based framework.
Introduction
3D object detection serves fundamental technique in 3D perception and is widely used in navigation robots and self-driving cars. Recently, transformer based [56] feature extractors have made significant progress in general tasks of Natural Language Processing (NLP) and 2D vision by flexibly modeling long-range relationships. To this end, some researchers have made great efforts to transfer the success of transformers 3D object detection. Specifically, to reduce the computation costs, SST [15] and SWFo Former [53] divide point clouds into pillars and implement window attention for pillar feature interaction
local 2D window. Considering some potential information loss of the pillar-based manners along the height dimension, DSVT-Voxel [60] further adopts voxel-based formats and implements set attention for voxel feature interaction in limited group size.
"Equal contribution. *Correspondigg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Wayma in ARIL1
Small Groups
Argoverse
Transformer
maScenes mAP (va)
Waymn MAPRILI
223
Window
DSVT
DISCumE NDS (val)
20.1
124
Wayme MAPHLLI
Large Groups
Linear Group RNN
muScenes mAP (test)
ONCE MAP
Bebennes NDS (tear
Window
EIION
CCCnteePeint
DDVVT
n nHANNe
Porformance comparkon
(c) LION (Ours)
Figure 1: (a) Comparison of different 3D backbones in terms of detection performance on Waymo [52], nuScenes [4], Argoverse V2 [62] and ONCE [37] datasets. Here, we adopt Mamba [22] as the default operator of our LION. Besides, we present the simplified schematic of DSVT (b) [60] and our LION (c) for implementing feature interaction in 3D backbones.
Although the above methods have achieved some success in 3D detection, they perform self-attention for pillar or voxel feature interaction with only small group size due computational limitations, locking the potential of transformers for modeling long-range relationships. Moreover, it is worth noting that modeling long-range relationships can benefit from large datasets, which will be important for achieving foundational models in 3D perception tasks in the future. Fortunately, in the field of large language models (LLM) and 2D perception tasks, some representative linear RNN operators such as Mamba [22] and RWKV [40] with linear computational complexity have achieved competitive performance with transformers, especially for long sequences, Therefore, question naturally arises: can we perform long-range feature interaction in larger groups at a lower computation cost based on linear RNNs in 3D object detection?
To this end, we propose a window-based framework based on LInear grOup RNN (i.e., perform linear RNN for grouped features in a window-based framework) termed LION for accurate 3D object detection in point clouds. Different from the existing method DSVT (b) in Figure 1, our LION (c) could support thousands of voxel features to interact with each other in a large group for establishing the long- -rage relationship. Nevertheless, effectively adopting linear group RNN to construct proper 3D detector highly sparse point cloud scenes remains challenging for capturing the spatial information of objects. Concretely, linear group RNN requires sequential features as inputs. However, converting voxel features into sequential features may result the loss of spatial information (e.g., two features that are close in 3D spatial position might be very far in this ID sequence). Therefore, we propose a simple 3D spatial feature descriptor and decorate the linear group RNN operators with it, thus compensating for the limitations of linear group RNN in 3D local spatial modeling.
Furthermore, to enhance feature representation in highly sparse point clouds, we present a new 3D voxel generation strategy based on linear group RNN to densify foreground features. A common manner of addressing this is to add an extra branch distinguish the foregrounds, as seen previous methods [53, 17, 70]. However, this solution is relatively complex and rarely used in 3D backbone due to its lack of structural elegance. Instead, we simply choose the high response of the feature map in the 3D backbone as the areas for voxel generation. Subsequently, the auto-regressive property of linear group RNN can be effectively employed to generate voxel features.
Finally, as shown in Figure (a), we compare LION with the existing representative methods. We can clearly observe that our LION achieves tfit--of-thearr on a board autonomous datasets in terms of detection performance. To summarize, our contributions are as follows: 1) We propose simple and effective window-basee 3D backbone based on the linear group RNN named LION to allow long-range feature interaction. 2) We introduce a simple 3D spatial feature descriptor and integrate it with the linear group RNN, compensating for the lack of capturing 3D local spatial information. 3) We provide a new 3D voxel generation strategy to densify foreground features, producing a more discriminative feature representation in highly sparse point clouds. 4) We verify the generalization of our LION with different linear group RNN mechanisms (e.g., Mamba, RWKV, RetNet). In particular,

---

(5.W.
(R,
Genention
Voxel Merging
Goneration
Voxel Merging
Backbome
Detection
Noselization
LJON Block #1
Voxel
LION
Voxel
BFV
LION Backbone
Point Clouds
Figure 2: The illustration of LION, which mainly consists of N LION blocks, each paired with a voxel generation for feature enhancement and voxel merging for down-sampling features along the height dimension. (H, W, D) indicates the shape of the 3D feature map, where H, W, and D are the length, width, and height of the 3D feature map along the X-axis, Y-axis, and Z-axis. N is the number of LION blocks. In LION, we first convert point clouds to voxels and partition these voxels into a series of equal-size groups. Then, we feed these grouped features into LION 3D backbone to enhance their feature representation. Finally, these enhanced features are fed into a BEV backbone and detection head for final 3D detection.
LION-Mamba achieves state-of-the-art challenging Waymo [52], nuScenes [4], Argoverse V2 [62], and ONCE [37] dataset, which further illustrates the superiority of LION.
2 Related Work
3D Object Detection in Point Clouds. 3D object detectors in point clouds can be roughly divided into point-based and voxel-based. For point-based methods [6, 68, 42, 10, 32, 39, 24, 50, 73, 67, 43, 65, 5], they usually sample point clouds and adopt point encoder [44, 45] to directly extract point features. However, the point sampling and grouping utilized by point-based methods is time-consuming. To avoid these problems, voxel-based methods [13, 12, 34, 48, 49, 51, 23, 59, 63, 69, 64, 70] convert the input irregular point clouds into regular 3D voxels and then extract 3D features by 3D sparse convolution. Although these methods achieve promising performance, they still limited by the local receptive field of 3D convolution. Therefore, some methods [8, 36) adopt the large kernel to enlarge the receptive field and achieve better performance.
Linear RNN. Recurrent Neural Networks (RNNs) are initially developed to address problems in Natural Language Processing (NLP), such as time series prediction and speech recognition, by effectively capturing temporal dependencies in sequential data. Recently, to overcome the quadratic computational complexity of transformers, significant advancements have been made in iime-parallelizable data-dependent RNNs (called linear RNNs in this paper) [46, 38, 40, 41, 55, 11, 66, 22, 54, 3]. These models retain linear complexity while offering efficient parallel training capabilities, allowing their performance to match or even surpass that transformers. Due to their scalability and efficiency, linear RNNs poised to play an increasingly important role in various fields and some works [1, 14, 29, 72] have applied linear RNNs to 2D/3D vision filed. In this paper, we aim further extend linear RNNs to 3D object detection tasks thanks their long-range relationship modeling capabilities.
Transformers in 3D Object Detection. Transformer [56] has achieved great success in many tasks, motivating numerous works to adopt attention mechanisms in 3D object detection to achieve better performance. However, the application of transformers is non-trivial in large-scale point clouds. Many works [16, 53, 35, 60] apply transformers to extract features by partitioning pillars or voxels into several groups based on local windows. Although these approaches achieve promising performance, they usually adopt small groups for feature interaction due to the quadratic computational complexity of transformers, hindering them from capturing long-range dependencies in 3D space. In contrast, propose a simple and effective framework based on linear RNNs named LION to achieve long-range feature interaction for accurate 3D object detection thanks their linear computational complexity.
Method
Due to computational limitations, some transformer-based methods [15, 60, 35] usually convert features into pillars group small size of voxel features to interact with each other within small groups, limiting the advantages of transformers long-range modeling. More recently, some linear RNN operators [22, 40, 55] that maintain linear complexity with the length of the input sequence
3

---

LION Layer
Vonel
Merging
LION
Layer
Voxel
Merging
LION
Layer
Voxal
Expunding
LION
Layer
Voxel Expanding
(a) LION Block
X-axis Partition
3D Sub-Maniflld Convolution
Youel
Merging
Expanding
Lincar Group RNN
LayerNorm
(b) Merging and Expanding
Y-axis Partition
GELU
Add
Spatial Feature Descriptor
Lincar Group RNN
Mappings
Inversed Index Mappings
(d)3D Spatial
(c) LION Layer
Feature Descriptor
Figure (a) shows the structure of LION block, which involves four LION layers, two voxel merging operations, two voxel expanding operations, and two 3D spatial feature descriptors. Here, 1x, 5 x, and 1x indicate the resolution of 3D feature map as (H, W, D). (H/2, W/2, D/2) and (H/4,1) W/4, D/4), D/4), respectively. (b) is the process of voxel merging for voxel down-sampling and
voxel expanding for voxel up-sampling. (c) presents the structure LION layer. (d) shows the details of the 3D spatial feature descriptor.
are proposed model long-rut range feature interaction. More importantly, the linear RNN operators such as Mamba [22] and RWKV [40] have even shown comparable performance with transformers LLM thanks their low computation cost in long-range feature interaction. This further motivates us to adopt linear RNNs to construct a 3D detector for long-range modeling.
3.1 Overview
In this paper, we propose a simple and effective saaddww-bseed framework based on LInear grOup RNN (i.e., perform linear RNN for grouped features in a hindow-based framework) named LION, which can group thousands of voxels (dozens times more than the number of previous methods [15, 60, 35]) for feature interaction. The pipeline of our LION is presented in Figure 2. LION consists of a 3D backbone, a BEV backbone, and a detection head, maintaining a consistent pipeline with most voxel-based 3D detectors [63, 60, 69]. In this paper, contribution lies in the design of the 3D backbone based on linear group RNN, which will be introduced the following.
3D Sparse Window Partition. Our LION is a window-based 3D detector. Thus, before feeding voxel features into our LION block, we need to implement a 3D sparse window partition to group them for feature interaction. Specifically,
first convert point clouds into voxels with the total number of L. Then, we divide these voxels into non-overlap 3D windows with the shape of (T,, Ty. T.). where Tz. Ty and T3 denote the length, width, and height of the window along the X-axis, Y-axis, and Z-axis. Next, we sort voxels along the X-axis for the X-axis window partition and along the Y-axis for the Y-axis window partition, respectively. Finally, to equal-size grouping manner in FlatFormer [35] instead of the classic equal-window grouping manner in SST [15]. That is, partition sorted voxels into groups with equal size K rather than windows of equal shapes for feature interaction. Due to the quadratic computational complexity of transformers, previous transformer-base
computation cost, adopt the
methods [15, 60, 35] only achieve feature interaction using a small group size. In contrast, we adopt a much larger group size K to obtain long-range feature interaction thanks the linear computational complexity the linear group RNN operators.
3.2 LION Block
The LION block is the core component of our approach, which involves LION layer for longerange feature interaction, 3D spatial feature descriptor for capturing local 3D spatial information, voxel merging for feature down-sampling and voxel expanding for feature up-sampling,
shown in Figure (a). Besides, LION block is hierarchical structure to better extract multi-scale features due to the gap of different 3D objects in size. Next, introduce each part of LION block.

---

[LLO
LION Block
Inpur
Diftission
Generation on Auro-regressivee
Ourput
Mosels
Inpur Diffined
Fonaground
The details of voxel generation. For input voxels, we first select the foreground voxels and diffuse them along different directions. Then, we initialize the corresponding features of the diffused voxels zeros and utilize the auto-regressive ability of the following LION block to generate diffused features. Note that we do not present the voxel merging here for simplicity.
Figure 5:
LION Layer. In LION block, we apply LION layer to model a long-range relationship among grouped features with the help of the linear group RNN operator. Specifically, shown in Figure (c), provide the structure of LION layer, which is composed of two linear group RNN operators. The first is used perform aang-range feature interaction based on the X-axis window partition and the second one can extract long-range feature information based on the Y-axis window partition. Taking advantage of two different window partitions, LION layer can obtain more sufficient feature interaction, producing more discriminative feature representation.
3D Spatial Feature Descriptor. Although linear RNNs have the advantages of long-range modeling with low computation cost, it is not ignorable that the spatial information might be lost when input voxel features are flattened into ID sequential features. For example, shown in Figure 4, there
X
02
32
adjacent features (i.e., indexed as 01 and 34) in 3D space. However, after they are flattened into ID sequential features, the distance between them in ID space is very far. We regard this phenomenon as loss of 3D spatial information. To tackle this problem, an available manner of scan orders for voxel features such as VMamba [31]. However, the order of scanning is too dendndesigned. Besides, as the scanning orders increase, the corresponding computation cost also increases significantly. Therefore, it is not appropriate large-scale sparse 3D point clouds to adopt this manner. As shown in Figure 3 (d), we introduce a 3D spatial feature descriptor, which consists of a 3D sub- manifold convolution, a LayerNorm layer, and a GELU activation function. Naturally, we can leverage the 3D spatial feature descriptor to provide rich 3D local position-aware information for the LION layer. Besides, we place the 3D spatial feature descriptor before the voxel merging to reduce spatial information loss in the process of voxel merging. We provide the corresponding experiment in appendix.
33
34
64
65
66
96
increase the number
y
Figure 4: The illustration of spa- tial information loss when flatten- ing into sequences. For exam- ple, there are two adjacent voxels in spatial position (indexed as 01 and 34) but far in the ID quences along the order.
Voxel Merging and Voxel Expanding. enable the network to obtain multi-scale features, our LION adopts hierarchical feature extraction structure. To achieve this, we need to perform feature down-sampling and up-sampling operations in highly sparse point clouds. However, mentioning that we cannot simply apply max or average pooling or up-sampling operations as in 2D images since 3D point clouds possess irregular data formats. Therefore. as shown in Figure (b), adopt voxel merging for feature down-sampling and voxel expanding for feature up-sampling in highly sparse point clouds. Specifically, for voxel merging, we calculate the sampled index mappings merge voxels. In voxel expanding, up-sample the down-sampled voxels by the corresponding inversed index mappings.
worth
3.3 Voxel Generation
Considering the challenge of feature representation in highly sparse point clouds and the potential information loss implementing voxel merging Figure 2, we propose a voxel generation strategy to address these issues with the help of the auto-regressive capacity of the linear group RNN.
5

---

Distinguishing Foreground Voxels without Supervision. In voxel generation, the first challenge is identifying which regions of voxel features need be generated. Different from previous methods [53, 17, 70] that employ some supervised information based on well-fearned BEV features to obtain the foreground region for feature diffusion. However, these approaches may be unsuitable for 3D) backbone and may even compromise its elegance. Interestingly, inspired by [34, 30], we notice that the corresponding high values of feature responses along the channel dimension in the 3D backbone (Refer to Figure 6 in our appendix) are usually the foregrounds. Therefore, we compute the feature response F for the output feature F; of the 7th LION block, where = 1, ..., NV indicates the index of LION block in the 3D backbone. Thus, this above process be formulated as:
C
F = Σ 2 P/
where C' the channel dimension of F.. Next, we sort the feature responses F descending order and select the corresponding Top-m voxels as the foregrounds from the total number I. of non-empty voxels, where m = r + L and r is the ratio of foregrounds. This process can be computed as:
Fm =
Top (PT),
where Topm (F)) means selecting Top-m voxel features from Fj. Fm are the selected foreground features, which will serve for the subsequent voxel generation.
Voxel Generation with Auto-regressive Property. The previous method [53] adopts a K-NN manner to obtain generated voxel features based on their K-NN features, which might be sub-optimal to enhance feature representation due to the redundant features and the limited receptive field. Fortunately, the linear RNN is well-suited for auto-regressive tasks in addition its advantage of handling long sequences. Therefore, we leverage the auto-regressive property of linear RNN to effectively generate the new voxel features by performing sufficient feature interaction with other voxel features in large group. Specifically, for convenience, we define the corresponding coordinates of selected foreground voxel features Fm as Pm- As shown in Figure 5, we first obtain diffused voxels by diffusing Pm with four different offsets (i.e., [-1,-1, 0]. [1,1, 0], [1,-1, O], and [-1,1, 0)) along the X-axis, Y-axis, and Z-axis, respectively. Then, we initialize the corresponding features of diffused voxels by all zeros. Next, we concatenate the output feature F; of the 7th LION block with the initialized voxel features, and feed them into the subsequent (i + 1)th LION block. Finally, thanks to the rugr-segres ability of the LION block, the diffused voxel features can be effectively generated based on other voxel features in large groups. This process can be formulated as:
Fp = F. + 11 1.0] + P11,1.0) + P1p.-1.0 + Fi-1,1.0),
(3)
F Block(Fp),
(4)
where Fl..,e) denotes the initialized voxel features with diffused offsets of y, and along the X-axis, Y-axis, and Z-axis. The @ and Block denote the concatenation and LION block respectively.
4 Experiments
4.1 Datasets and Evaluation Metrics
Waymo Open Dataset. Waymo Open dataset (WOD) [52] is well-known benchmark for large-scale outdoor 3D perception, comprising 1150 scenes which are divided into 798 scenes for training. 202 scenes for validation, and 150 scenes for testing. Each scene includes about 200 frames, covering a perception range of 150m x 150m. For evaluation metrics, WOD employs 3D mean Average Precision (mAP) and mAP weighted by heading accuracy (mAPH), each divided into two difficulty levels: L1 is for objects detected with more than five points and 1.2 is for those at least one point.
nuScenes Dataset. nuScenes [4] is a popular outdoor 3D perception benchmark with perception range of up to 50 meters, Each frame in the scene is annotated with 2Hz. The dataset includes 1000 scenes, which is divided into 750 scenes for training. 150 scenes for validation, and 150 scenes for testing. nuScenes adopts mean Average Precision (mAP) and the NuScenes Detection Score (NDS) as evaluation metries.
Argoverse V2 Dataset. Argoverse V2 [62] a outdoor 3D perception benchmark with a long-range perception of up to 200 meters. It contains 1000 sequences in total, 700 for training, 150 for validation, and 150 for testing. Each frame in the scene is annotated with 10Hz. For the evaluation metric, Argoverse v2 adopts similar mean Average Precision (mAP) metric with nuScenes [4].

---

4.2 Implementation Details
Network Architecture. In our LION, we provide three representative linear RNN operators (i.e.. Mamba [22], RWKV [40], and RetNet [55]). Each of operator adopts bi-directional structure to better capture 3D geometric information inspired by [21]. On WOD, we keep the same channel dimension C' = 64 for all LION blocks in LION-Mamba,
and LION-RetNet. For we set C' = 128. We follow DS
the large version of size (0.32m, 0.32m, 0. 0.757. The number of LION blocks N is set to 4. For these four LION blocks, the window sizes (Tx, Ty. T2) are set to (13. 13, 32), (13, 13, 16). (13, 13,8, 8), and (13, 13, 4), and the corresponding group sizes K are 4096, 2048, 1024, 512, respectively. Besides, the same bester-based detection head and loss function as DSVT [60] for fair comparison. In the voxel generation, we set the ratio r 0.2 to balance the performance and computation cost. For the nuScenes dataset, we replace DSVT [60] 3D backbone with our LION 3D backbone except for changing the grid size to (0.3m. 0.3m, 0.25m). For the Argoverse V2 dataset, we replace the 3D backbones of VoxelNext [9] or SAFDNet [70] with our LION 3D backbone except for setting the grid size to (0.4m, 0.4m, 0.25m). Moreover, it is noted that we only add three extra LION layers to further enhance the 3D backbone features, rather than applying the BEV backbone to obtain the dense BEV features.
[60] to set the grid
adopt
Training Process. On the WOD, we adopt the same point cloud range, data augmentations, learning rate, and optimizer as the previous method [60]. We train our model 24 epochs with batch size of 16 on 8 NVIDIA Tesla V100 GPUs. Besides, we utilize the fade strategy [58] to achieve better performance in the last epoch. For the nuScenes dataset, we adopt the same point cloud range, data augmentations, and optimizer as previous method [2]. Moreover, we find that LION converges faster than previous methods on nuScenes dataset. Therefore, we only train our model for 36 epochs without CBGS [77]. The learning rate and batch size are set to 0.003 and 16, respectively. It is worth noting that the CBGS strategy extends training iterations about 4.5 times, which means that our training iterations are much fewer than previous methods [2] (i.e., 20 epochs with CBGS). For the Argoverse V2 dataset, we adopt the same training process with SAFDNet [70] and SECOND [63], respectively.
4.3 Main Results
In this section, we provide a board comparison of our LION with existing methods WOD, NuScenes and Argoverse V2 datasets for 3D object detection. Furthermore, in the section A.1 of our appendix, we present the experiment in ONCE dataset [37] and provide more types of linear RNN operators (e.g., RetNet, RWKV, Mamba, xLSTM, and TTT) based on our LION framework for 3D detection a small but popular dataset KITTI [20] for a quick experience.
Results on WOD. To illustrate the superiority of our LION, we provide the comparison with existing representative methods on the WOD in Table 1. Here, we also conduct the experiments our LION with different linear group RNN operators, including LION -ammba, LION-RWKV and LJON-RetNC Compared with the transformer- based methods [15, 53, 60], our LION with different linear group RNN operators outperforms the previous state-of-the--a (SOTA) transformer- based 3D backbone DSVT- Voxel [60]. illustrating the generalization of our proposed framework. To further scale up our LION, we present the performance of LION-Mamba-L by doubling the channel dimension of LION-Mamba. It can be observed that LION-Mamba-L significantly outperforms DSVT- Voxel with 1.9 MAPH/L.2, leading to a new SOTA performance. The above promising results effectively demonstrate the superiority of our proposed LION.
In Table 2, we also provide the results with multiple frames as inputs on the WOD test split. For three frames, our LION-L outperforms the representative method PillarNeXt [20] by 3.3 (77.4 vs. 74.1) MAPH/L2, which clearly illustrates the superiority of our methods.
Results on nuScenes. We also evaluate our LION on nuScenes validation and test set [4] further to verify the effectiveness of our LION. As shown in Table 3, on nuScenes validation set, our LION-RetNet, LION-RWKV, and LION-Mamba achieves 71.9, 71.7, and 72.1 NDS, respectively, which outperforms the previous advanced methods DSVT [60] and HEDNet [71]. Besides, our LION-Mamba even brings new SOTA on nuScenes test benchmark, which beats the previous advanced method DSVT with 1.2 NDS and 1.4 mAP, illustrating the superiority of our LION. Note
that all results of our LION are conducted without any test-time augmentation and model ensembling.
7

---

Table I: Performances on the Waymo Open Dataset validation set (train with 100% training data). denotes the two-stage method. Bold denotes the best performance of all methods. "-L." means we double the dimension of channels in LION 3D backbone. RNN denotes the linear RNN operator. All results are presented with single-frame input, no test-time augmentation, and no model ensembling.
Methods
Present at
Operator
Velacle 3D APAPH
L1
L2
Pedestrian 3D AP/APH
L1
L2
Cyeli 3D APAPH
L1
L2
MAPIMAPH
L2
SECOND [63]
Sensoes
72.3/71.7
63.9/63.3
68.7/58.2
60.7151.3
60.6/59.3
58.3/57.0
61.0/57.2
PointPillars [27]
CVPR 19
72.1/11.5
63/666311
70.6/56.7
62.8/50.3
64.4/62.3
61.9/59.9 9
62.8/57.8
CenterPoint (69)
CVPR
14.2/73.6
66. 266.7
76.6/70.5
68.8663.2
72.3/71.1
69.768855
68.2/65.8
PV-RONN; [48]
CVPR 20
78 0/775
69 6964009.0
79.2/73.0
7064664.7
71.5/70.3
09.0667.8
69.6/67.2
PillarNet-44 [47]
BCCV 22
79. 1/78.6
70.9/70.5
$0.6/74.0
72.3666.2
72.3/71.2
69. 79.7768.7
71.0/68.5
FSD]
NeurIPS
Spurs Conv
79 29.2/78.8
70.5/70.1
82.6/77.3
73.9/69.1
77.1/76.0
74.4/73.3
72.9/70.8
AHDetV2 [26]
AAAI 22
77 67767
69. 29.7169.2
80.2/74.6
72.2/67.0
73.7/72.7
71 11,0/70,1
71 11.0668..
PillarNeXt [28)
CVPR 23
78. 4/77.9
70.3/69.8
82.5/77.1
74, 84,9969.8
73.2/72.2
70.6/99.6
71. 9669
Voxe [9]
CVPR
78 2/22/77.
69.
31.3/76,3
73.5%68.6
76. 1/74.9
73.
72.2/70.1
CenterFormer177
BCCV 22
75 0/74/4444
69.9169.4
78.6/73.0
73.6/68.3
72.3/71.3
69.8/68.8
1/68/
PV-RCNN++| [49]
ICCV
79 79.3/78.8
70.6/70.2
81.3/76.3
73,2/68.0
73.7/72.7
71,2/70.2
7,,7699,5
TransFusion [2]
CVPR
-/-
-65.1
-
-13,.7
---
-/65.9
-1649
ConQueR [78]
CVPR 23
76.1/75.6
68.7/68.2
79.0/72.3
70.9664.7
73.9/72.5
71447011
70.3/67.7
FocalFormer3D
ICCV 23
-f-
1/77/
72,7666.8
73.7/72.6
71 71,5669.0
HEDNot [71]
NeurIPS
81.1/80.6
73.2/72.7
$4.4/50.0
78.7/77.7
75.8/74.9
75.3/73.4
SEED (35)
ECCV 24
79.7/79.2
71.8/714
83.1/78.3
75.5/70.8
80.0/78.8
77.3/76.1
74.9/72.8
SST_ [15]
CVPR
RNN Transformer
76.2/75.8
68 06.0/67.6
81,4/74.0
72.8465.9
+
SWFommer [53]
BCCV 22
77.8/77.3
69.2/68.8
80.9/72.7
72.5/64.9
-/-
-
-/-
OcTi [74]
CVPR 23
78. 78.1/77.6
69. 8968/
80.8/74.4
72.5/66.5
72.6/71.5
69.9/68.9 9
70,7/68.2
DSVT-Pilla
CVPR
79.3/78.8
70.9/70.5
82.8/77.0
75.2/69.8
76.4/5.4
73.6/72.7
73.2/7L0
DSve- [60]
CVPR 23
79.7/79.3
71.4/1.0
83.7/78.9
76.1/71.5
77.5/76.5
74.6/73.7
74.0/72.1
LION-RetNet (Ours)
79.0/78.5
70.6/70.2
84.6/80.0
77.2/72.8
79.0/78.0
76.1/75.1
74.6/72.7
LION-RWKV (Ouns)
79.7/79.3
71.3/710
84.6/50.0
77.1/72.7
78.7/77.7
75.8/74.8
74.7/72.8
Mamba (Ours)
79.5/79.1 1
71,1/70.7
84.9/80.4
77,5/73.2
79.7/78.7
76.7/75.8
78.1/73.2
LamNN (Ours)
80.3/79.9
72.0/716
$5.8/1144
78.5/74.3
80.1/79.0
77.2/76.2
75.9/74.0
Table 2: Result of our LION with multiple frames as input on the Waymo Open Dataset test set.
Methods
Preseut at
Francs
Vohicle 3D AP/APH
[.]
1.2
Pedestrian 3D AP/APH
1.1
1.2
Cyclist 3D APAPI
1.1
1.2
mAP/mAPH1
1.2
PV-RCNN++1 [49]
DCV 22
81.6/812
73.9/73.5
80.475.0
74.1/69.0
71.970.8
69.3/68.2
72.4770.2
AFDetV2 [26]
AAAI 22
-/-
72 270.3
PillarNeXt [28]
CVPR
72.2/69.6
FSDE
NeurIPS 22
82,7/82.3 -/-
74.4/74.1 -/-
82,9/71.9 -/-
75.9/71.3
75.6/74 -f-
72.9/71.8
74.4772.4
PillarNeXt (28)
CVPR
-/72.0
ConterPoiate= [69]
CVPR 21
82.8/82.3
75.5/75.1
81.1/78.2
75.1/72.4
74.4/73.3
72.0/71.0
74.2/72.8
PillarNeXt (28)
CVPR 23
83.3/82.
76.2/75.8
84.4/8144
78.8/76.0
73.7/72.7
71.6/70.6
75.5/74,1
LION-Mamba- (ours)
84.7/84.3
77.2/76.9
87.2/84.5
82.0/79.3
79.2/78.3
76.8/75.9
78.7/77.4
Results on Argoverse V2. To further verify the effectiveness of our LION on the long-range perception, we evaluate the experiments on Argoverse V2 validation set. For a fair comparison, we adopt the same detection head [69] with VoxelNext [9] and SAFDNet [70] for long-range perception. As shown in Table 4, our LION RetNet, LION-RWKV, and LION-Mamba achieve the detection performance with 40.7 mAP, 41.1 mAP and 41.5 mAP, all three of which have outperformed the previous SOTA method SAFDNet [70], leading to new SOTA results. These superior results clearly illustrate the effectiveness of our LION.
4.4 Ablation Study
In this section, we conduct ablation studies of LION on the WOD validation set with 20% training data. If not specified, we adopt LION-Mamba as our default model and train our model with 12 epochs in the following ablation studies. For more experiments, please refer to our appendix.
Ablation Study of LION. To illustrate the effectiveness of our proposed LION, we conduct the ablation study for each component, including the design of large group size, 3D spatial feature descriptor, and voxel generation in Table 5. Here, our baseline is proposed LION that removes the design of large group size, 3D spatial feature descriptor, and voxel generation. In Table 5, we observe that the design of large group size even brings 1.1 mAPH/L2 performance improvement, which illustrates the benefits of performing long-range feature interaction with the help of linear RNN. Then, we integrate the 3D spatial feature descriptor, which further produces an obvious performance improvement with 1.7 MAPH/L.2. This demonstrates the superiority of the 3D spatial feature descriptor in compensating for the lack of capturing spatial information of linear RNNs. Furthermore, we notice that the 3D spatial feature descriptor is very helpful to small objects (e.g., Pedestrians) thanks to its capability of extracting the local information of 3D objects. To address
8

---

Table 3: Performances on the nuScenes validation and test set. 'T.L', 'C.V.' 'Ped... 'M.T', T.C.., and 'B.R' are short for trailer, construction vehicle, pedestrian, motor, traffic cone, and barrier, respectively, All results are reported without any test-time augmentation and model ensembling.
Performances the validation set
Method
Present at
NDS
mAP
Car
Truck
Bus
T.I..
C.V.
Ped.
M.T.
Bike
T.C.
B.R.
CenterPoint (69]
CVPR 21
66.5
59.2
84,9
57.4
70.7
38.1
16.9
85.1
59.0
42.0
69.8
68.3
VoxeINeXt [9]
CVPR
66.7
60.5
83,9
70.5
38.1
50.0
69.4
69.4
Uni3DETR [61]
NeurIPS 23
68.5
61.7
TransFusion [2]
CVPR
70.1
86.9
60.8
73.1
72.9
57.3
77.2
70.3
QTNet [25]
NeurIPS 23
70.9
66.5
87.2
61.5
75.8
43.0
25.7
87.8
75.5
61.5
75.4
71.4
DSVT [60]
CVPR
71.1
87.4
62.6
75.9
42.1
25.3
74.8
58.7
77.9
71.0
HEDNet [71]
NeurIPS 23
71.4
66.7
87.7
60.6
77.8
50.7
28.9
87.1
74.3
56.8
76.3
66.9
LION-RetNet (Ours)
71.9
67.3
87.9
64.3
78.7
44.6
27.6
88.9
73.5
56.6
79.2
72.1
LION RWKY (Ours)
71.7
88.1
77.6
74.3
56.2
80.1
68.3
Mombh (Ours)
72,1
68.0
87.9
64,9
77.6
44.4
28.5
89.6
75.6
59.4
80.8
71.6
Performances on the testt set
TransFusion LiDAR [2]
CVPR 22
70.2
65.5
86.2
56.7
66.3
58.8
28.2
86.1
68.3
44.2
82.0
78.2
DSVT [60]
CVPR 23
72.7
68.4
86.8
58.4
67.3
63.1
37.1
88.0
73.0
47.2
84,9
78.4
HEDNee [71]
NewrIPS 23
72.0
67.7
87.1
56.5
70.4
63.5
33.6
87.9
70.4
44.8
85.1
78.1
Mam--Mamaa (Ours)
73.9
87.2
61.1
68.9
36.3
90.0
49.2
87.3
Table 4: Comparison with prior methods on Argoverse V2 validation set. 'Vehicle', 'C-Barrel', "MPC-Sign', 'A-Bus', 'C-Cone', 'V-Trailer', 'MBT', 'W-Device' and 'W-Rider' are short for regular vehicle, construction barrel, mobile pedestrian crossing sign, articulated bus, construction cone, vehicular trailer, message board trailer, wheeled device, and wheeled rider.
mAP
Voduete
Blus
Podesman
Stop Sign
Box Trock
Dollard
C-Darrel
Matorsycliss
MPC-Sign
Motorsyele
Bicycle
A-Bus
School Bus
Truek Cab
C-Oone
V-Traeer
Sign
Vahiclo
Skoller
BicyeEat
MBT
Dog
Whoslchair
W Deviiice
Method
CenterPoiat [09]
22.0
67.6 38.9
46.5 16.9 37 40.1
32.2 28.6
27.4
33.4
24.5
25,8
22.6
29.5
22
6.3
3.9
20,1
22
0.0
3.9
10.9
4.2
HEDNet (71]
37.1
48.7
42.5
23.3
27.5
27.9
6.9
27.2
21.6
0.0
30.7
8.7
NoxelNeXi [9]
30.7 72.7
38.8
63.2
40.
40.
53.9
61.9
44.7
29.4
42.4
80.6
20.1
25.2
19.9
44.9
20.9
14.9
6.8
15.7
32.4
16.9
0.0
144
0.1
17.4
6.6
FSD+1 [17]
28.2 68.1 40.9
59.0
29.0
58.5
41.8
12.6
39.7
26.2
49.0
38.6
20.4
30.5
11.
11.2
26,9
11.9
59
13.8
33.4
21.1
0.0
955
7.1
14,0
9.2
FSD+2 [18]
37.6 77.0 47.6
70.5
43.6
41.5
$3.9
58.5
56.8
39.0
60.7
49.4
28
41.9
30.2
44.9
33.4
16.6
7.3
32.5
459
24
1.0
12.6
17.1
26.3
17.2
SAFDNer [70]
19.7
70.7
44.7
54.3
50.0
RoNNeR
[40.7 74.7 41.0
72.7
47.5
44.2
66.9
77.0
57.1
48.3
63.7
5
27.0
42.5
25.2
57.9
29.7
22.0
6.9
39.3
47.3
19.9
0.0
28.8
12.8
37.7
12.6
LION-WWKV
41.1 76.3 44.6
74.0
52.1
46.0
68.1
75.8
55.8
19.1
62.8
55.3
27.1
12.9
25.9
60.1
30.9
22.2
9.3
36.5
55.3
23.2
0.0
27.8
7.1
37.6
11.4
LION-Mamba
41.5 75.1 43.6 73.9 53.9
45.1 66.4
74.7 61.3
48.7 65.1 56.2
21.7
42.7
25.3
58.
28.9
23.6
8.3
49,5
47.3
19.0
0.0
31,4
8.7
37.6
11.8
Table 5: Ablation study for each component in LION. Here, the large group size means that we set as (4096, 2048, 1024, 512) for four blocks (also refer to the section of our implementation details), otherwise, we set a small group size as (256, 256, 256, 256).
3D AP/APH (L2) Pedestrian
mAP/mAPH (L2)
Large Group Size
3D Spatial Feature Descriptor
Voxel Generation
Vehicle
Cyelist
65.6/65.2
72.3/6.0
68.3/67.2
68 8/65,8
66.2/65.7
73.7/67.2
68. 68.7667.6
69 09.5166.9
66.5/66.
74.8/69.6
70.9/70.0
70.8/68.6
66.4/66.0
73.5/67.4
70.4/69.3
70. 1/67.6
67.0/66.6
75.4/70.2
71.9/71.
71.4/9.3
the challenge of feature representation in highly sparse point clouds, we adopt voxel generation to enhance the features of foregrounds, which brings a promising gain of 0.7 MAPH/L22 (67.6 vs. 66.9). By combining all components, our LION achieves superior performance of 69.3 MAPH/I.2, which outperform outperforms the baseline of 3.5 mAPH/L2.
Superiority of 3D Spatial Feature Descriptor. To further verify the necessity of 3D spatial feature descriptor, we provide the comparison with two available manners including the MLP and linear RNN to replace our descriptor in Table 6. Here, we set our LION without 3D spatial feature descriptor as the baseline in this part. We observe that MLP even does not bring promising performance improvement in terms of mAPH/L2 since MLP lacks the ability to capture local 3D spatial information. Furthermore, considering the limited receptive field of MLP, we adopt a linear group RNN operator to replace MLP. We find that there is only slight performance improvement with 0.3 MAPH/L22, which indicates that the linear group RNN might not be good at modeling local spatial relationships although has the strong capability to establish long-range relationships. In contrast, our 3D spatial feature

---

Table 6: Ablation study for 3D Spatial Feature Descriptor (3D SFD) in LION.
APIAPH (1.2)
mAP/mAPH
Methods
Vehicle
Pedestrian
Cyclist
(L2)
Bascline
66.4/66.0 66.6/66.2
73.5167.4 4
70.4/69.3
70.1/67.6
ML.P
74.1/68.1
70.0/69.0
70.2/67.7
Lincar Group RNN
66.4/66.0
74.0/68.2
70.5/69.5
70.3/67.9
3D SFD (Ours)
67.0/66.6
75.4/70.2
71.9/71.0
71.4/69.3
Table 7: Ablation study for voxel generation LION, "Baseline" indicates no voxel generation. "Zero Feats" and "K-NN Feats" indicate initializing features to all zeros and K-NN features, respectively. "Auto-Regre" "eguto-Rereesvvee uses the LION block based on linear group RNN for its auto-regressive property. "Sparse-Con" maintains the same structure as the LION block but replaces the linear group RNN with 3D sub-manifold convolution.
AP/APH (1.2)
mAP/mAPH (L.2)
Index
Methods
Vehicle
Pedestrian
Cyclist
Baseline
66.5/66.
74,8/69 6
70.9/70.0
70. 70.8/68.6 6
Zero Feats Spurse-Conv
64.6/64.2
72.8/67.4
69.3/68.3
68. 68.9/66.6
III
K-NN Feats Gutto-Rerressive
66.5/66,
74.0/68.7
71.1/70.1
70.5/68.3
IV
Zero Feats + Auto-Regressive (Ours)
67.0/66.6
75.4/70.2
71.9/710
71.//69.3
descriptor brings obvious performance improvement, which boosts the baseline of 1.7 mAPH/L.2. This effectively illustrates the superiority of the 3D spatial feature descriptor in compensating for the lack of local 3D spatial-aware modeling in the linear group RNN.
Effectiveness of Voxel Generation. Voxel generation is applied to enhance the feature representation of objects in highly sparse point clouds for accurate 3D object detection. Therefore, to explore the effectiveness of our proposed voxel generation, manners in Table 7. First, we compare our results of IV with by only replacing the operator of linear group RNN in LION block with 3D sub-manifold convolution to generate the diffused features. We find that the manner IV (69.3 vs. 66.6) significantly outperforms the performance of II in terms of mAPH/L2. This benefits from the linear group RNN's 's ability model long-range feature interactions, generating a more reliable feature representation through its auto-regressive capacity, demonstrating the superiority of voxel generation with the linear group RNN. To further illustrate that the effectiveness of voxel generation is from its auto-regressive property of LION block rather than strong feature extractor, voxels by K-NN operation (III) instead of the manner of all-zeros features (IV) and then feed them to the same following LION block for voxel generation. In Table we find that the manner of III is inferior to IV by 1.0 MAPH/L2. This clearly illustrates that our voxel generation is benefiting from its nuto-regressive property of LION block. Finally, compared with the baseline (I), our voxel generation (IV) can obtain promising performance improvement, which verifies its effectiveness.
present the comparison with several available
initialize the diffused features of the foreground
Conclusion
In this paper, we have presented a simple and effective window-based framework termed LION, which can capture the long-range relationship by adopting linear RNN for large groups. Specifically, LION incorporates a proposed LION block to unlock the great potential of linear RNNs in modeling long-range relationship and voxel generation strategy to obtain more discriminative feature representation in sparse point clouds. Extensive ablation studies demonstrate the effectiveness of our proposed components. Additionally, the generalization of our LION is verified by performing different linear group RNN operators. Benefiting from our well-designed framework and the proposed superior components, LION-Mamba achieves state-of-the-art performance the challenging Waymo and nuScenes datasets.
Acknowledgements
This work was supported by the National Science Fund for Distinguished Young Scholars of China (Grant No. 62225603).
10

---

References
[1] Benedikt Alkin, Maximilian Beck, Korbinian Püppel. Sepp Hochreiter, and Johannes Brandstetter. Vision- Istm: xlstm as generic vision backbone. arXiv:2406. 04303, 2024.
[2] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew- Tai. Trans- fusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR, 2022. 7, 8,
[3] Maximilian Boek, Korbinian Pöppel, Markus Spanring, Andreas Auer, Olcksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv:2405.0451/ 045517 2024. 15
[4] Holger Caesar, Varun Bankiti, Alex H Lang. Sourahh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset autonomous driving.
In CVPR, 2020. 2, 3, 7
[5] Chen Chen, Zhe Chen, Jing Zhang, and Dacheng Tao. Sasa: Senantics-augnmented set abstraction for point-based 3d object detection. In AAAI, 2022. 3
[6] Yilun Chen, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Fast point r-cnn. In ICCV, 2019. 3
[7] Yilun Chen, Zhiding Yu. Yukang Chen, Shiyi Lan, Anima Anandkumar. Jiaya Jia, and Jose M Alvarez. Focalformer3d: Focusing hard instance for 3d object detection. In ICCV, 2023. 8
[8] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Largekemel3dd: Scaling up kernels in sparse cnns, CVPR, 2023.
[9] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for object detection and tracking. In CVPR, 2023. 7,8,9 9
[10] Bowen Cheng, Lu Sheng. Shaoshuai Shi, Ming Yang. and Dong Xu, Back Back-tracing representative points for voting- based 3d object detection in point clouds. In CVPR, 2021. 3
[11] Soham De, Samuel L.. Smith, Anushan Fernando, Aleksandar Botev, George Bristian-Murany Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient language models. MxXi::2402.194222 2024." 3
[12] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance bosel-based 3d object detection. In AAAI, 2021. 3
[13] Shaocong Dong, Lihe Ding, Haiyang Wang, Tingfa Xu, Xinli Xu, Jie Wang. Ziyang Bian, Ying Wang. and Jianan Li. Mssvt: Mixed-scale sparse voxel transformer for 3d object detection point clouds. In NewrIPS, 2022.
[14] Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang. Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures. axXiv:2403.02.000 2024.
[15] Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu- Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In CVPR, 2022. 1, 3,4, 7,
[16] Lue Fan, Ziqi Pang. Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In CVPR, 2022.
[17] Lue Fan, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Fully sparse 3d object detection. In NeurlPS. 2022. 6, 8,
[18] Lue Fan, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Fsd Improving fully sparse 3d object detection with virtual voxels. arXiv:2308.0373: 2023.
[19] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRS, 2013. 16
[20] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012. 7, 15

---

[21] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In ICASSP, 2013.
[22] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv:2312.00775 2023. 2,3, 15, 17
[23] Tianrui Guan, Jun Wang, Shiyi Lan, Rohan Chandra, Zuxuan Wu, Larry Davis, and Dinesh Manocha. M3detr; Mulli-repeeeentation, multi-scale, mutual-relation 3d object detection with transformers. In WACV, 2022. 3
[24] Chenhang He. Hui Zeng, Jianqiang Huang. Xian-Sheng Hua, and Lei Zhang. Structure aware single-stage object detection from point cloud. In CVPR, 2020. 3
[25] Jinghua Hou, Zhe Liu, Zhikang Zou, Xiaoqing Ye, Xiang Bai, al. Query-based temporal fusion with explicit motion for 3d object detection. In NeurlPS, 2023. 9
[26] Yihan Hu, Zhuangzhuang Ding, Runzhou Ge, Wenxin Shao, Li Huang, Kun Li, and Qiang Liu. Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds. AAAI, 2022.
[27] Alex H Lang, Sourahh Vora, Holger Caesar, Lubing Zhou, Jiong Yang. and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds, In CVPR, 2019. 8, 15, 16
[28] Jinyu Li, Chenxu Luo, and Xiaodong Yang. Pillarnext: Rethinking network designs for 3d object detection in lidar point clouds. In CVPR, 2023. 8
[29] Dingkang Liang, Xin Zhou, Xinyu Wang, Xingkui Zhu, Wei Xu, Zhikang Zou, Xiaoqing Ye, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv:2402. 10739, 2024.
[30] Jianhui Liu, Yukang Chen. Xiaoqing Ye. Zhuotao Tian, Xiao Tan, and Xiaojuan Qi. Spatial pruned sparse convolution for efficient 3d object detection. In NewrlPS, volume 35, pages 6735-6748, 2022.
[31] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv:2401. 10166, 2024. 5
[32] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In ICCV, 2021. 3
[33] Zhe Liu, Jinghua Hou, Xiaoqing Ye, Tong Wang, Jingdong Wang, and Xiang Bai. Seed: simple and effective 3d detr in point clouds. In ECCV, 2024. 8
[34] Zhe Liu, Xin Zhao, Tengteng Huang, Ruolan Hu, Yu Zhou, and Xiang Bai. Tanet: Robust 3d object detection from point clouds with triple attention. In AAAI, 2020. 3. 6, 15
[35] Zhijian Liu, Xinyu Yang, Haotian Tang, Shang Yang, and Song Han, Flatformer: Flattened window attention for efficient point cloud transformer. In CVPR, 2023. 3,4 4
[36] Tao Lu, Xiang Ding, Haisong Liu, Gangshan Wu, and Limin Wang. Link: Linear kernel for lidar-based 3d perception. In CVPR, 2023. 3
[37] Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, et al. One million scenes for autonomous driving: Once dataset. 110037 2021. 2, 7, 15
[38] Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In ICML, 2023. 3
[39] Xuran Pan, Zhuofan Xia, Shiji Song. Li Erran Li, and Gao Huang. 3d object detection with pointformer. In CVPR, 2021. 3
[40] Bo Peng, Eric Alcaide, Quentin Gregory Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Nguyen Chung, Leon Derezynski, et al. Rwkv: Reinventing mns for the transformer era. In EMNLP, 2023, 2. 4. 7, 15
[41] Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, Kranthi Kiran GV, Jan Koconi Bartlomiej Koptyra, Satyapriya Krishna, Rouald McClelland au2, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song. Haoqin Tu, Stanistaw Wožniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu. Eagle and finch: Rwky with matrix matrix-valued states and dynamic recurrence.
2024.
12

---

[42] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, 2019. 3
[43] Charles R Qi. Wei Liu. Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgh-d data. In CVPR, 2018. 3, 15
[44] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[45] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NeurlPS, 2017. 3
[46] Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. In NewrIPS, 2023. 3
[47] Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: High-performance pillar-based 3d object detection. In ECCV, 2022. 8
[48] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In CVPR, 2020. 3, 8, 16
[49] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang. Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. IJC, 2021. 8,8
[50] Shaoshuai Shi, Xiaogang Wang. and Hongsheng Li. Pointrenn: object proposal generation and detection from point cloud. In CVPR, 2019, 3, 15, 16
[51] Shaoshuai Shi, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE TPAMI, 2020. 3
[52] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla. Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 2, 3, 6, 15, 19
[53] Pei Sun, Mingxing Tan, Weiyue Wang. Chenxi Liu, Fei Xia, Zhaoqi Leng, and Drago Anguelov. Swformer: Sparse window transformer for 3d object detection point clouds. ECCV, 2022. 1, 2, 3, 6, 7, 8
[54] Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang. Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et Learning to (leam test time): Runs with expressive hidden states. arXiv:2407.0m 04620, 2024. 3, 15
[55] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv:2307.088. 08621, 2023.
3, 7, 15
[56] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrlPS, 2017. 1,3
[57] Sourabh Vora, Alex Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In CVPR, 2020. 16
[58] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmentation for object detection. In CVPR, 2021. 7
[59] Haiyang Wang, I. Liee Ding, Shancong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, and I Liwei Wang. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. In NeurlPS, 2022. 3
[60] Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang. Di He, Bernt Schiele, and Liwei Wang. Dsvt: Dynamic sparse voxel transformer with rotated sets. In CVPR, 2023. 1, 4, 7, 8, 9, 15, 17, 19
[61] Zhenyu Wang, Ya-Li Li, Xi Chen, Hengshuang Zhao, and Shengjin Wang. Uni3detr: Unified 3d detection transformer. In NeurlPS, 2024. 9
[62] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagject Singh, Siddhesh Khandelwal, Bowen Pan. Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting, In NeurlPS, 2023. 2, 3. 6
[63] Yan Yan. Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018 3,
4, 7, 8, 15, 16
13

---

[64] Honghui Yang, Wenxiao Wang. Minghao Chen, Binbin Lin, Tong He, Hua Chen. Xiaofei He, and Wanli
Ouyang. Pyt-ssd: Single-stage 3d object detector with point-voxel transformer. In CVPR, 2023.
[65] Jinrong Yang. Lin Song, Songtao Liu, Weixin Mao, Zeming Li, Xiaoping Li, Hongbin Sun, Jian Sun, and Nanning Zheng. Dbq-ssd: Dynamic ball query for efficient 3d object detection. In ICLR, 2022. 3
[66] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv:2312.06635, 2023.
[67] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia, 3dssd: Point-based 3d single stage object detector. In CVPR, 2020. 3
[68] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In CVPR, 2019. 3
[69] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, 2021. 3, 4, 8, 8,9,1 16
[70] Gang Zhang, Junnan Chen, Guohuan Gao, Jianmin Li, Si Liu, and Xiaolin Hu, Safdnet: A simple and effective network for fully sparse 3d object detection. In CVPR, 2024. 2, 3, 6, 7, 8, 16
[71] Gang Zhang, Chen Junnan, Guohuan Gao, Jianmin Li, and Xiaolin Hu. HEDNet: A hierarchical encoder- decoder network for 3d object detection in point clouds. In NeurlPS, 2023. 7,8,9
[72] Guowen Zhang, Lue Fan, Chenhang He, Zhen Lei, Zhaoxiang Zhang, and Lei Zhang. Voxel mamba: Group-free state space models for point cloud based 3d object detection. arXiv preprint arXiv:2406. 10700, 2024. 3
[73] Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points equal: Learning highly efficient point-hased detectors 3d lidar point clouds. CVPR, 2022.
[74] Chao Zhou, Yanan Zhang, Jiaxin Chen, and Di Huang. Octr: Octree-based transformer for 3d object detection. CVPR, 2023.
[75] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018.
[76] Zixiang Zhou, Xiangehen Zhao, Yu Wang. Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. ECCV, 2022.
[77] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. 1908: 09492, 2019. 7
[78] Benjin Zhu, Zhe Wang. Shaoshuai Shi, Hang Xu, Langing Hong. and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection, CVPR, 2023.
14

---

A Appendix
The appendix is organized as follows. First, in section A.1, we provide more types of linear RNN operators (e.g.. RetNet, RWKV, Mamba, xLSTM, and TTT) based on our LION framework for 3D detection on small but popular dataset KITTI for a quick experience. Second, we present extra experiments on the WOD [52] validation set section A.3, including the placement of the spatial feature descriptor, the impact of different window sizes and different group sizes in inference, and the ratio r in voxel generation. Third, we provide the comparison of computation cost and parameter size in section A.4, and detailed information of LION structure in section A.5. Forth, in section A.6, we visualize the feature maps of different LION blocks to illustrate the rationality of distinguishing foreground voxels based feature response. Finally, results with DSVT [60] and qualitative results of LION to demonstrate the superiority our LION in section A.7 and section A.8. Besides, discuss the broader impacts in section A.9.
provide the comparison of qualitative
A.1 Experiments on KITTI dataset
Table 8: Effectiveness on the KITTI validation set for Car, Pedestrian, and Cyclist. * represents our reproduced results by keeping the same configures except for their 3D backbones for a fair comparison. Our LION supports different representative linear RNN operators (TTT. xLSTM, RetNet, RWKV. and Mamba). mAP is calculated by all categories and all difficulties with recall 11.
Car Moderate
Pedestrian Moderate
Cyclist Moderate
Method
mAP
Easy
Hard
Easy
Hard
Easy
Hard
VoxeINet [75]
77.5
65.1
57.7
39.5
33.7
31.5
61.2
48.4
44.4
51.0
SECOND [63]
83.1
73.7
51.1
42.6
37.3
70.5
53.9
46.9
58.4
PointPillars [27]
79.1
75.0
52.1
41.5
75.8
$2.9
60.8
PointRCNN [50]
85.9
75.8
68.3
49.4
41.8
38.6
73.9
59.6
53.6
60.8
TANet [34]
83.8
75.4
67.7
54.9
46.7
42.4
73.8
59.9
53.5
62.0
DSVT-Pillar [60]
87.3
77.4
76.2
61.4
56.8
51.8
82.3
67.1
63.7
69.3
DSVT-Voxel* [60]
87.8
77.8
76.8
66.1
59.7
55.2
83.5
66.7
63.2
70.8
LION-TTT
87.9
78.0
76.7
63.4
58.6
53.7
84.0
69.6
64.5
70.7
LION-xLSTM
87.7
77.9
76.8
59.3
54.0
82.4
67.4
63.4
70.6
LION-ReINet
88.0
77.9
55.8
83.6
69.6
71.5
LION-Mamba
88.6
78.3
67.2
60.2
55.6
83.0
68.6
63.9
71.4
LION-RWKV
88.5
78.3
77.1
68.9
62.2
58.1
89.6
71.2
66.9
73.4
KITTI Dataset. KITTI [20] is popular benchmark dataset for autonomous driving, which consists of 7481 training frames and 7518 test frames for 3D object detection. We follow the dataset splitting protocol in [43] and further split the 7481 training frames into 3712 frames for training set and 3769 frames for validation set. For the 3D detection task, KITTI dataset mainly detects Car, Pedestrian, and Cyclist for three difficulty levels, i.e., Easy. Moderate, and Hard. And the mean Average Precision (mAP) using 11 recall positions is adopted the evaluation metric.
Results on KITTI We conduct experiments on the KITTI validation set to illustrate the generalization of LION for different linear RNN operators. We select
representative linear RNN operators (TTT [54], xLSTM [3], RetNet [55], Mamba [22], and RWKV [40]) for LION, We adopt the same training parameters (i.e., number of epochs, learning rate, optimizer) with SECOND [63]. Besides, we use the same BEV backbone and the detection head with SECOND [63]. For a fair comparison, keep the same configure of DSVT [60] and all our LION methods except 3D backbones. As shown in Table 8, LION-RetNet, LJON-Mamba, and LION-RWKV outperforms DSVT-VVoel by 0.7 mAP, 0.6 mAP, and 2.6 mAP. These experiments demonstrate the generalization and effectiveness of our linear RNN-based framework LION.
A.2 Experiments on ONCE dataset
ONCE Dataset. ONCE [37] is another representative autonomous driving dataset, which consists of 5000, 3000, and 8000 frames for training, validation, and testing set, respectively. Each frame is annotated with 5 classes (Car, Bus, Truck, Pedestrian, and Cyclist). Besides, ONCE merges the car, bus, and truck class into super-class called vehicle following WOD [52]. For the detection metric,
15

---

Table 9; Comparison with previous methods on ONCE validation set. For fair comparison, we replace the SECOND 3D backbone in the ONCE dataset with our LION 3D backbone, maintaining a grid size of (0.4m, 0.4m, 0.25m), and adopt the same training process as SAFDNet [70] and SECOND [63] while utilizing the center head of CenterPoint [69].
Method
overall
0-30m
Vahicle
30-50m
50m-inf
overall
0.30m
Pedestrian
30-50m
S0m-inf
overall
0-30m
Cyelist
30-50m
50m-inf
mAP
PoinRCNN
PointPillars [27]
62.1
47.0
17.6
19.7
15.2
10.2
46.8
58.3
SECOND [63]
71.2
$4.0
63.0
47.3
26.4
29.3
24.1
18.1
58.0
70.0
52.4
34.6
51.9
PV-RCNN [[8]
77.8
89.4
72.6
58.6
23.5
25.6
22.8
17.3
59.4
71.7
52.6
36.2
53.6
ConterPoins
PointPannting [57]
66.2
80.3
59.8
42.3
44.8
52.6
36.6
22.5
62.3
73.6
57.2
40.4
57.8
LiON-RetNet
78.
88.7
72.4
5X.5
12.4
60.5
43.6
26.3
68.1
79.4
62.9
46.1
66.3
LION-RWKV
78.3
89.2
72.6
56.7
50.6
60.0
24.2
68.4
79,4
65.8
LION-Mamba
78.2
89.1
72.6
57.5
53.2
62.4
24.5
68.5
79.2
ONCE extends [19] by taking the object orientations into special consideration and evaluating the final performance by mAP for three classes.
Results on ONCE. We also evaluated our LION on ONCE validation set to further verify the
effectiveness of our LION. As shown in Table 9, our LION-RetNet, LION-RWKV, and LION-Mamba produces advanced detection performance with 66.3 mAP, 65.8 mAP, and 66.6 mAP, respectively. worth mentioning that LION-Mamba outperforms the previous SOTA method CenterPoint [69] with 6,5 mAP, leading to new SOTA result. These experiments illustrate the superiority of our LION.
A.3 Extra Experiments
Table 10: The Placement of 3D Spatial Feature Descriptor.
3D AP/APH (L2) Pedestrian
mAP/mAPH (L.2)
Methods
Vehicle
Cyclist
Baseline
66.4/66.0
73.5/67.4
70.4/69.3
70.1/67.6
Placement
66.5/66.1
74.8/69.1
71.1/70.2
70.1/68.6
Placement (Ours)
67.0/66.6
75.4/70.2
71.9/71.0
71.4/69.3
Table 11: The ratio r in voxel genenration.
3D AP/APH (I.2)
mAP/mAPH (L2)
Ratio
Vehicle
Pedestrian
Cyclist
66.5/66.1
74.8/69.6 75.4/70.2
70.9/70.0
70.8/68.6
0.2 (Ours) 0.5
67. 67.0/66.6
71.9/71.0
71.4/69.3
67.2/66.8
75.3/70.0
72.1/71.1
71.5/69.3
Table 12: Comparison of group sizes WOD validation (train with 20% training data).
3D APIAPH (L2) Pedestrian
mAP/mAPH (L2)
Group Size
Vehicle
Cyclist
[256, 256, 256, 256) [1024, 512, 256, 256
65.6/65.2
72.3/65.0
68.3/67.2
68.8/65.8
II
66.9/66.5
74.9/69.6
70.8/69.8
70.9/68.6
III
[2048, 1021, 512, 256]
66.7/66.3
74.9/69.7
72.2/71.2
71.3/69.1
4096, 2048, 1024, 512)
67.0/66.6
75.4/70.2
71.9/710
71.4/69.3
V
1192, 4096, 2048, 1024)
66,5/66.1
74.6/69.5
71.6/70.6
70.9/68.7
The Placement of 3D Spatial Feature Descriptor. We conduct experiments about the placement of the 3D spatial feature descriptor, shown in Table 10. We regard the manner that does not adopt 3D SFD of our LION as the baseline. Here, we provide two available manners: Placement and Placement 2. For Placement 1, we place the 3D SFD after voxel merging. For Placement 2, we place 3D SFD before the voxel merging- Compared the baseline, Placement brings 1.0 mAPH/I.2 improvement, which demonstrates the effectiveness of 3D SFD in compensating for the lack of local
16

---

Table 13: Comparison of different window and group sizes in inference on WOD validation set (train
with 100% training data). Bold denotes the result of LION with the default settings the main paper.
Window Size
Group Size
mAP/mAPH (L2)
77. 7, 3,3) [7, 7, 16). [7. 7, 8). 77, 7,44
[4096, 2048, 1024, 512]
73.24
[13, 13, 32].
[13, 13, 16),
[13, 13, 8). [13, 13, 4]
[4096, 2048, 1024, 512]
73.24
[25, 25, 32).
[25, 25, 16).
[25, 25, 8). [25. 25, 4]
[4096, 2048, 1024, 512]
73.25
[13, 13, 32).
[13, 13, 16), [13, 13, 8J. [13, 13, 4]
[2048, 1024, 512, 256]
73.18
[13, 13, 32).
[13, 13, 16). [13, 13, 8). [13, 13, 4]
[4096, 2048, 1024, 512]
73.24
[13, 13, 32]. [13, 13, 16). [13, 13, 8). [13, 13,4 4]
[8192, 4096, 2048, 1024]
73.02
3D spatial-aware modeling in linear RNNs. Moreover, Placement 2 further brings 0.7 mAPH/L.2 improvement over Placement which demonstrates the effectiveness 3D SFD for reducing spatial information loss in the process of voxel merging.
The Ratio in Voxel Generation. We conduct the ablation study for foreground selection ratio in voxel generation. As shown in Table 11, compared with baseline (r = 0), the manner of setting 0.2 brings 0.7 mAPH/L2 performance improvement. When the performance is improved slightly. Therefore, we set r = 0.2 to balance the performance and computation cost.
set larger ratio 0.5.
Different Group Sizes in Training. We provide a comparison of different group sizes in Table 12. We set minimum group size of 256 for four LION blocks as the baseline. We could find that the manners with large group sizes (i.e., II, III. IV) bring consistent performance over the small ones. However, there is drop in performance when the group size is increased from IV to V. This might lead to less effective retention important information in excessively long sequences due to the limited memory capacity of linear RNNs.
Different Window Sizes and Group Sizes in Inference. To analyze the impact of window size and group size in the inference process, we evaluate the results under the cases of different window sizes and group sizes with the same trained model of LION-Mamba (i.e., window size= 13, 32), (13, 13, 16), (13, 13, 8), and (13, 13, 4)] and group bize=(4096 2048, 1024, 512)) on WOD 100% training data. As shown in Table 13, surprisingly, we find that using different window sizes or group sizes during inference still does not significantly affect performance. This indicates that LION might decrease the strong dependence on hand-crafted priors and have good extrapolation ability.
A.4 Comparison of Computation Cost, Parameter Size and Latency
Table 14: Comparison of computation cost, parameter size and latency of different methods on the WOD validation set.
Method
Operator
mAP/mAPH (L.2)
FLOPs
Params (M)
Latency (ms)
DSVT-Voxel [60]
Transformer
74.0/72.1
100.8
2.7
136.7
LION (Ours)
Mambu
75.1/73.2
58.5
1.4
146.2
We compare our LION with the representative transformer- method DSVT- [60] in terms of computation cost, parameter size and latency the WOD validation set. As shown in Table 14, LION with Mamba [22] achieves a more superior performance (73.2 mAPH/L2), less computational cost (58.5 GFLOPs), fewer parameters than DoVT- However, our LION still has more latency in inference stage. These properties clearly illustrate the superiority of our proposed LION and might have great potential to further optimize the speed of inference in the future.
A.5 Architecture Specifications
As shown Table 15, the architecture specifications of the LION models (IJON-RWKV, LION- RetNet, LION- and LION-Mamba-L) Waymo Open dataset are detailed in terms of
17

---

Table 15: Detailed architecture specifications Waymo Open dataset.
LION-RWKV
LION-RetNet
LION-Mamba
LION-Mamba-L.
Window Shape
Window Shape
Window Shape
Window Shape
Block
Dim, Group Size
Dim, Group Size
Dim, Group Size
Dim, Group Size
[13, 13, 13] 64. 4096
[13, 13,32]
[13, 13, 33)
[13, 13, 32]
Block
64, 4096
64, 4096
128, 4096
[13, 13, 16]
[13, 13, 16)
[13, 13, 16)
[13, 13, 16]
Block
64, 2048
64, 2048
64, 2048
128, 2048
[13, 13, 8]
[13, 13, 8]
[13, 13, 8)
[13, 13,
Block
64, 1024
64. 1024
64, 1024
128, 1024
[13, 13, 4]
[13, 13, 4]
[13, 13, 4]
[13, 13, 4]
Block
64, 512
64, 512
64, 512
128, 512
window shape, dimension, and group size. For LION-Mamba-L. set the dimension to 128 to double the channel of LION.
High
Low
Block
Block
Block
Block 4
Figure 6: Visualization of feature map of different blocks. We highlight the foreground annotated by red GT boxes. The color map represents the magnitude of the feature response.
A.6 Visualization for Feature Map
As shown in Figure 6, we visualize feature maps of different LION blocks. We can observe that the features pass through more blocks, the magnitude the foreground's feature response becomes larger, demonstrating the rationality of distinguishing foreground voxels by feature response. Besides, we find that the foreground features become more dense and more distinguished, which also demonstrates the effectiveness of the voxel generation operation.
18

---

(a)
ROOD DNT on
Exso uelky g0
(b)
Figure 7: Comparison of DSVT and LION the WOD validation set from the BEV perspective. Blue and green boxes are the prediction and ground truth boxes. It can be seen that LION can achieve better results compared DSVT, demonstrating the superiority of LION.
A.7 Comparison of Qualitative Results with DSVT
To illustrate the superiority of LION, we present the visualization of the qualitative results of DSVT [60] (a) and LION (b) the WOD [52] validation set, shown in Figure 7. Specifically, in the first and third columns, our LION can reduce more false positives compared with DSVT. In the second column, LION can achieve more accurate localization. These qualitative results demonstrate the superior performance of our LION.
LION even detects some hard objects distance. In the last column,
A.8 Qualitative Results
As shown in Figure 8, we visualize the qualitative results of LION on the WOD validation set. As shown in the first column, LION still achieve satisfactory results even in crowded 3D scenes. However, as shown in the second and third columns, LION misses some objects at distance with sparse point clouds. Therefore, will further improve the performance of distant objects by fusing the image features in the future.
(a)
(b)
(c)
Figure 8: Qualitative results of LION on the WOD validation set. Green and blue boxes denote
ground truth and predicted bounding boxes, respectively.
A.9 Broader Impacts
LION achieves promising performance for 3D object detection, enhancing the safety of autonomous driving. However, LION has relatively high requirements on computing resources to achieve faster running speed, which puts forward higher requirements for the hardware of autonomous driving. Future research could focus on optimizing LION to improve bottlenecks in running speed while maintaining high detection accuracy, making it more accessible and practical for autonomous driving.
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The main claims in the abstract and introduction accurately reflect the our contributions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification:
We have discussed limitations in paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper primarily introduces architecture and proposes several modules based on this architecture. does not include theoretical assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information to reproduce the main experimental results of the paper.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We plan to provide all code for reproducing the results after the manuscript is accepted. Additionally, all datasets used in our experiments are publicly available, ensuring that the main experimental results can be faithfully reproduced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.ca
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: hwwbbiic/uuiies / /CodeSubmi ssi assionoolicn for more details.
/
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
22

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experimental details in paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper provides detailed experimental setups and procedures to ensure the reproducibility of the results. However, it does not include error bars or other statistical significance information.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide experiments compute resources in paper.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: i crchicsseilllees
Answer:
[Yes]
Justification:
Our research conducted in the paper conform with the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: These impacts are discussed in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]

---

The paper focuses on foundational research and does not release any data or models that pose high risk for misuse. Therefore, no specific safeguards are necessary.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing assets used in the paper, such as code and datasets, are properly credited. The licenses and terms of use for these assets are explicitly mentioned and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new models introduced in the paper are thoroughly documented. Compre- hensive documentation is provided alongside the models to ensure clarity and reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
25

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects. Therefore, it does not include participant instructions, screenshots, or compensation details.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects. Therefore, IRB approval equivalent review is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "YAADDrJ9Bz", "reviewer": "Reviewer_ehGN", "summary": "This paper presents the LION block, a neural component that builds up a backbone to extract 3D features with linear group RNNs for 3D object detection. The authors introduce a 3D spatial feature descriptor to extract point features, and a novel auto-regressive voxel generation method to density the foreground feature in sparse point clouds.", "questions": "Since objects-of-interest in autonomous driving is relatively small comparing to the whole scene, why long-range relation ($K=4096, 2048, …$) is important (line 133 - 135) in this scenario?", "limitations": "The limitations are properly discussed in the main paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation is clear and easy to follow.\n2. The proposed method surpasses previous state-of-the-art methods.", "weaknesses": "1. Figure 3 (a) is confusing. Since the 3D Spatial Feature Descriptors are neural layers with learnable parameters, it would be better to represent these layers with blocks like “LION Layer”.\n2. It would be beneficial to discuss the rationale for positioning the 3D Spatial Feature Descriptor after the Linear Group RNN, as placing it beforehand could potentially better preserve spatial information.\n3. An ablation study on Voxel Merging and Expanding would help quantify their contributions to the framework's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: Figure 3 (a) is confusing. Since the 3D Spatial Feature Descriptors are neural layers with learnable parameters, it would be better to represent these layers with blocks like “LION Layer”.\n\nThanks for your nice suggestion! We will revise \"3D Spatial Feature Descriptors\" as \"3D Spatial Description Layer\" to make it clearer in the final version. \n\n**W2**: It would be beneficial to discuss the rationale for positioning the 3D Spatial Feature Descriptor (3D SFD) after the Linear Group RNN, as placing it beforehand could potentially better preserve spatial information.\n\nThanks! We provide the corresponding results of different placements as following table. For Placement 1, we place the 3D SFD after voxel merging. For Placement 2, we place 3D SFD before the voxel merging. We agree with your explanation that placing 3D Spatial Feature Descriptor beforehand could potentially better preserve spatial information.\nFinally, we will add this discussion of the rationale for positioning the 3D Spatial Feature Descriptor in the revised version. \n\n| Method      |  Vehicle  | Pedestrian |  Cyclist  | mAP/mAPH (L2) |\n|:------------|:---------:|:----------:| :-------: |:-------------:|\n| Baseline    | 66.4/66.0 | 73.5/67.4  | 70.4/69.3 |   70.1/67.6   |\n| Placement 1 | 66.5/66.1 | 74.8/69.1  | 71.1/70.2 |   70.1/68.6   |\n| **Placement 2** | 67.0/66.6 | 75.4/70.2  | 71.9/71.0 |  **71.4/69.3**    |\n\n**W3**: An ablation study on Voxel Merging and Expanding would help quantify their contributions to the framework's performance.\n\nThanks! Note that removing the voxel merging and voxel expanding operations means that the input voxels of the linear group RNN are not processed by any downsampling or upsampling operations, which results in some additional computational cost. In the following table, we provide the experiment of our method with/without the voxel merging and voxel expanding operations.\nIt can be observed that our method with adopting the voxel merging and voxel expanding operations could effectively reduce the computational costs while maintaining superior performance.\n\n|                    Method                    | Latency (ms) |  Vehicle  | Pedestrian |  Cyclist  | mAP/mAPH (L2) |\n| :------------------------------------------| :---------: | :-------: | :--------: | :-------: | :----------: |\n| LION-Mamba (w/o Voxel Merging and Expanding) |    180.1    | 66.8/66.4 | 75.1/70.1  | 72.1/71.1 |  71.3/69.2   |\n|                  LION-Mamba                  |    **146.2**    | 67.0/66.6 | 75.4/70.2  | 71.9/71.0 |  **71.4/69.3**   |\n\n**Q1**: Since objects-of-interest in autonomous driving is relatively small comparing to the whole scene, why long-range relation (K=4096, 2048, ...) is important (line 133 - 135) in this scenario?\n\nGood question! In fact, capturing the long-range relationship is helpful to obtain richer context information for better understanding the whole scene. Therefore, it is not only important for detecting large objects, but also for detecting small objects.  \nIn this paper, we build the long-range relationship by the linear group RNN operations. We provide the ablation studies of different group sizes in the following table. Here, we set a minimum group size of 256 for all our four LION blocks (Baseline: [256, 256, 256, 256]). We could observe that the manners with larger group size (i.e, II, III, IV, V) bring consistent performance improvement over the baseline (I). However, we carefully find that there is a drop in performance from IV to V by further enlarging the group size. This might lead to less effective retention of important information in excessively long sequences due to the limited memory capacity of linear RNNs.\nFinally, to better understand the long-range relationship, we provide the visualization in Figure 1 (Please refer to our uploaded PDF file). We will add these discussion and experiments in the final version.\n\n|  #   |  Group Size |  Vehicle  | Pedestrian |  Cyclist  | mAP/mAPH (L2) |\n|:----|:------------------------|:-------: | :--------: | :-------: | :----------: |\n|  I   |   [256, 256, 256, 256]   | 65.6/65.2 | 72.3/65.0  | 68.3/67.2 |  68.8/65.8   |\n|  II  |  [1024, 512, 256, 256]  | 66.9/66.5 | 74.9/69.6  | 70.8/69.8 |  70.9/68.6   |\n| III  |  [2048, 1024, 512, 256]  | 66.7/66.3 | 74.9/69.7  | 72.2/71.2 |  71.3/69.1   |\n|  IV  | **[4096, 2048, 1024, 512] (Ours)**  | 67.0/66.6 | 75.4/70.2  | 71.9/71.0 |  **71.4/69.3**   |\n|  V   | [8192, 4096, 2048, 1024] | 66.5/66.1 | 74.6/69.5  | 71.6/70.6 |  70.9/68.7   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OliWipJYUo", "reviewer": "Reviewer_wvps", "summary": "This paper targets the problem of long-range feature interactions for point cloud detection. It proposes a window-based 3D backbone based on linear group RNN and sparse convolution. In contrast to existing Transformers methods, this work increases the group size by leveraging the linear complexity of recent Mamba and RWKV. The paper also introduces the 3D spatial feature descriptor to capture 3D local spatial information and 3D voxel generation strategy to address the sparsity of point clouds. Experiments demonstrate the efficacy of this proposed method.", "questions": "Please see the weaknesses section. Additionally,  \n$\\bullet$\tPlease provide a comparison with HEDNet in Waymo, including latency, accuracy, and parameters.  \n$\\bullet$\tIt would be better to illustrate visualization results to support the claim LION can model long-range dependencies.  \n$\\bullet$\tWhat is the performance of replacing the LION Layer with DSVT block (use the setting in the original paper) in the LION Block? Moreover, please provide a comparison of the latency between the two variants.", "limitations": "Limitations have been included", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "$\\bullet$  The problem studied in this paper is important, as the long-range relationship is critical in point cloud detection.\n\n$\\bullet$  LION-Mamba achieves strong performance with low GFLOPs on widely used outdoor datasets, including the Waymo Open Dataset and nuScene.\n\n$\\bullet$  The proposed 3D spatial feature descriptor and voxel generation are simple and easy to follow.", "weaknesses": "$\\bullet$  [Novelty] My main concern is the overall limited technical contribution. The paper does not show significant differences from previous works. Additionally, it lacks a thorough discussion comparing it with existing approaches.  \n$\\qquad$(1) Model Structure. The proposed LION block uses the same encoder-decoder structure as SED block in HEDNet [1]. Furthermore, the 3D spatial feature descriptor is identical to SSR block, and the voxel merging and expanding operations are similar to RS conv. Besides, the LION layer has the same structure with DSVT block. This paper appears to merely integrate the DSVT block into SED block and replace Transformers with linear RNNs.  \n$\\qquad$(2) Window Partitions. The equal-size window partition along X/Y-axis has been widely adopted in voxel-based detectors, such as Flatformer and DSVT.   \n$\\qquad$(3) Voxel Generation: The way to “distinguishing foreground voxels without supervision” has been widely used in point cloud detection, such as SPSS-Conv [2] and VoxelNeXt [3].   \n$\\bullet$  [Latency] While there is some analysis of computation cost (GFLOPs), it lacks a comparison of latency with state-of-the-art algorithms, such as HEDNet and DSVT. For 3D object detection in outdoor scenarios like autonomous driving, the real-time application makes the latency more critical. Besides, in Figure 1, the comparison with LION-Mamba-L is missing.  \n$\\bullet$  In line 42-44, the authors claim LION can support thousands of voxel features to establish long-range relationships. However, in Line 153-157, this paper proposes the spatial information loss issue and need additional local sparse convolution to address. In my point of view, a sequence of thousands based on window partitions must include the voxel and all its neighbors. This inconsistency raises concerns about the robustness and effectiveness of LION.  \n$\\bullet$ The linear RNN, such as Mamba, is a unidirectional model. Is it reasonable to use two single-layer RNNs in a LION Layer? Will the bidirectional Mamba, such as Vision Mamba, enhance performance?  \n$\\bullet$  [Motivation] The X/Y-axis window partition is proposed to address the feature interactions problem with group size limitation. The motivation to use X/Y-axis window partition is not clear, given the large group size.  \n$\\bullet$  [Motivation] The motivation for choosing only four different offsets in voxel generation with auto-regressive property is unclear. A voxel has many neighbors in 3D space, so why did the author choose only these four offsets? Further explanation is needed.  \n\n[1] HEDNet: A hierarchical encoder-decoder network for 3d object detection in point clouds. NIPS’23  \n[2] Spatial Pruned Sparse Convolution for Efficient 3D Object Detection. NIPS’22   \n[3] VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking. CVPR’23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and valuable suggestions. Here, we further address our contributions in this paper:\n\n1) **Linear RNN-based 3D detection framework**: support kinds of linear RNNs (e.g., Mamba, RWKV, RetNet) to allow long-range feature interaction. \n\n2) **3D spatial feature descriptor**: compensate for the lack of linear RNNs in capturing local 3D spatial information.\n\n3) **Voxel generation**: utilize auto-regressive property of the linear RNN for voxel diffusion and obtain a more discriminative feature representation in highly sparse point clouds.\n\n**[Novelty]**\n\n1) Model Structure.\n   \n   - The encoder-decoder structure is not our contribution and we adopt it to reduce the computational costs while maintaining superior performance. We provide the experiments in the following table.\n   \n    |Method|Latency (ms)|Vehicle|Pedestrian|Cyclist|mAP/mAPH (L2)|\n    |:---|:---:|:---:|:---:|:---:|:---:|\n    |LION-Mamba(w/o encoder-decoder structure)|180.1|66.8/66.4|75.1/70.2|72.1/71.1|71.3/69.2|\n    |LION-Mamba|146.2|67.0/66.6|75.4/70.2|71.9/71.0|71.4/69.3|\n     \n   - Although the 3D spatial descriptor consists of a common and simple spconv-based module, it is crucial to help our linear RNN-based network to better capture 3D local spatial information. Therefore, the combination of 3D spatial descriptor and linear RNN operators could make each voxel feature perceive the local spatial information and long-range relation, which is important to improve detection performance ((Refer to the Table 3 and 4 in the main paper)).\n   \n   - The structure of LION layer is a widely used structure (e.g., Swin Transformer, FlatFormer, and DSVT). We keep consistent with DSVT for convenience. \n\n2) Window Partition.\n\n    We follow the previous work FlatFormer (Refer to L130-133 in the main paper). Besides, we do not claim that the equal-size window partition is not our contribution. \n\n3) Voxel Generation.\n\n    Thanks for your valuable feedback. We would like to emphasize that our primary contribution in voxel generation lies in the first attempt of the auto-regressive capacity of the linear RNN. The part of “distinguishing foreground voxels without supervision\" only serves it. We appreciate your point regarding the distinction of foreground voxels without supervision, and we will revise this part in the final version.\n\n**[Latency]**\n\nThanks. We provide the comparison of different methods. We evaluate the latency on single 3090 with a batch size of 1. LION is an early attempt to adopt linear RNN to 3D object detection, and the running time needs to be optimized compared with the highly-optimized operator (e.g., sparse conv). We will improve the speed in future work for the real-time application. Please refer to the limitation (L323-325) in the main paper. For Figure 1 in the main paper, we will revise it in the final version.  For a more detailed discussion of the Transformer-based method DSVT, please refer to W4 of Reviewer 9fJS.\n\n|Method|mAPH (L2)|Latency (ms)|params (M)|\n|:---|:---:|:---:|:---:|\n|HEDNet|73.4|74.8|11.9|\n|DSVT|72.1|136.7|8.7|\n|LION-Mamba|73.2|146.2|8.6|\n|LION-Mamba-L|74.0|191.3|16.1|\n\n**[Motivation]**\n\n1) Motivation to use X/Y-axis window partition.\n\n    For linear RNN models, different sequence arrangements lead to varying feature interactions. More arrangements can improve feature richness, so we use X/Y-axis window partitioning to generate diverse sequences.\n\n2) Why did the author choose only these four offsets?\n\n    Setting more different offsets usually results in producing more voxels, which brings more computation costs for the following LION layers. To reduce computation cost, we only consider the offset of each voxel for diffusion in BEV space and ignore the offset along the Z axis. Here, we simply choose these four offsets along the two diagonals in BEV space.\n\n**[Weakness]**\n\n1) A sequence of thousands based on window partitions must include the voxel and all its neighbors ...\n\n    Although a sequence with thousands of voxels usually contains the voxel features of their neighbors, two adjacent voxels might be far apart in this sequence (please refer to Figure 4 in the main paper) since the linear RNN extract features along the sequence. Therefore, we adopt an additional local sparse convolution (or 3D spatial feature descriptor) to address this problem.\n\n2) Bidirectional modeling.\n\n    Sorry for missing this detail! In this paper, all linear RNNs (Mamba, RWKV and RetNet) adopt the bidirectional manner for better feature interaction. We will add the details in the revised version.\n\n\n**[Question]**\n1) Comparison with HEDNet\n\n    Thanks. We provide the comparison with HEDNet in **[Latency]**.\n\n2) Visualization of long-range dependencies.\n\n    Thanks. We provide the visualization for long-range dependencies in Figure 1 in our uploaded PDF file.\n\n3) Performance of replacing the LION Layer with DSVT block.\n\n    We replace the LION layer with DSVT block to validate the effectiveness of LION, as shown in the following table. * represents the results without our 3D spatial feature descriptor and voxel generation. All models are trained with 20% data and 12 epochs. We report the official results of DSVT (I). We can observe that the performance of integrating DSVT into LION (II and IV) is lower than LION (III and V). Finally, the manners with our 3D spatial feature descriptor and voxel generation (IV and V) produce much better performance than the methods (II and III), which effectively demonstrates the importance of our proposed components.\n    \n    |#|Method|Latency (ms)|Vehicle|Pedestrian|Cyclist|mAP/mAPH (L2)|\n    |:---|:---|:---:|:---:|:---:|:---:|:---:|\n    |I|DSVT|136.7|67.2/66.8|72.5/66.4|70.1/69.1|69.9/67.4|\n    |II|LION-DSVT*|122.7|64.8/64.3|71.0/63.6|67.4/66.2|67.7/64.7|\n    |III|LION-Mamba*|123.2|66.2/65.7|73.7/67.2|68.7/67.6|69.5/66.9|\n    |IV|LION-DSVT|157.7|66.1/65.7|74.4/68.8|70.7/69.8|70.4/68.1|\n    |V|LION-Mamba|146.2|67.0/66.6|75.4/70.2|71.9/71.0|71.4/69.3|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kpg2dwsec0", "reviewer": "Reviewer_5dKF", "summary": "This paper proposes a linear Group RNN-based backbone for 3D object detection tasks. It can achieve a larger window size compared to previous transformer-based methods. A 3D spatial feature descriptor is also introduced to capture 3D spatial information. Furthermore, to address the sparsity of point clouds, this paper leverages the auto-regressive property of RNNs to generate voxels for distinguishing foreground features. Experiments on the large-scale nuscenes and waymo datasets validate the effectiveness, as well as the generalization across different linear compositional RNN operators such as Mamba, RWKV, and RetNet.", "questions": "1. L124, I’m not clear about the Window Partition. As far as I understand, non-empty voxels are extracted line by line into the window along the X or Y axes. Every time a sequence of 4096 lengths is filled, the remaining non-empty voxels are continued to be placed in the next window. So, how do you deal with it if the final window is not enough for 4096 voxels. Regarding the division and description of windows, it should be clearer. The current description is difficult for me to understand.\n\nOther questions pleae see the weakness. Will raise up the score after solving my concerns.", "limitations": "The author claim the corresponding limitation in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.While transformer-based backbone networks have demonstrated superior performance, their quadratic complexity has limited their application scenarios. This paper explores the potential of linear group RNNs as feature extraction backbones for 3D detection tasks, and presents state-of-the-art performance on large-scale datasets, which is interesting.\n2. The proposed method has been shown to be effective across multiple linear group RNN operators such as Mamba, RWKV, and RetNet, demonstrating the generalization ability.\n3. The paper is well-written, with a clear expression of the motivation and is easy to read.", "weaknesses": "1.In this paper, the author has transformed the irregular point cloud into regular voxel representation. However, L172-175，the author claim max or average-pooling operations is not suitable to achieve downsampling or upsampling operations. This is conflicted.  Furthmore, I think the motivation provided for the voxel generation approach is not sufficient, making it difficult to accept the motivation behind the proposed voxel generation method.\n\n2.The author should report the test set results on the waymo dataset of the experimental results. In addition, the authors should also report the results on the multi-frame setting on the waymo dataset to prove the effectiveness of the method in the multi-frame setting. I want to the know the test set results on waymo dataset and multi-frame setting results.\n\n3.The authors should report the inference time of LION so that we have a clearer understanding of the latency and resource consumption of the proposed method.\n\n4.When will release the code for the community?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1-1**: In this paper, the author has transformed the irregular point cloud into regular voxel representation. However, L172-175, the author claim max or average-pooling operations is not suitable to achieve downsampling or upsampling operations. This is conflicted.\n\nSorry for this misleading! Since the distribution of voxels in 3D space is sparse, the regular max or average pooling operations on a dense  2D image are not suitable for performing voxel downsampling or upsampling. \n\n**W1-2**: Furthmore, I think the motivation provided for the voxel generation approach is not sufficient, making it difficult to accept the motivation behind the proposed voxel generation method.\n\nOur motivation of voxel generation has two folds: 1) Voxel generation can densify some key voxel features to enhance the feature representation in highly sparse point clouds; 2) Voxel generation can mitigate the information loss from voxel merging operation, which is an effective operation to reduce the computation cost in our LION.\nWe will make them clearer in the revised version.\n\n**W2**: The author should report the test set results on the waymo dataset of the experimental results. In addition, the authors should also report the results on the multi-frame setting on the waymo dataset to prove the effectiveness of the method in the multi-frame setting. I want to the know the test set results on waymo dataset and multi-frame setting results.\n\nThanks for your nice suggestion. We provide the results with 3 frames on the val set in the following table. It could be found that our LION-Mamba-L even outperforms DSVT with 2.2 mAPH/L2, which effectively illustrates the superiority of our LION.\n\n|    Method    | Frames  |   Vehicle   |  Pedestrian  |  Cyclist  | mAP/mAPH (L2) |\n|:------------|:-------:|:-----------:|:------------:|:---------:|:------------:|\n|     SST      |    3    |  68.5/68.1  |  75.1/70.9   |    -/-    |     -/-      |\n|     DSVT     |    3    |  73.6/73.2  |  78.2/75.4   | 77.2/76.4 |  76.3/75.0   |\n| **LION-Mamba-L** |    3    |  73.9/73.5  |  81.0/78.3   | 80.7/79.8 |  **78.5/77.2**   |\n\nFurthermore, for submitting Waymo test benchmark, it is a common trick to unify the training and validation datasets to train the model for better performance. Therefore, we need to reorganize our dataset and re-train our model for the test set. Considering the limited time, we train our LION-Mamba-L with 3 frames for only 12 epochs to save the training time.\nOur LION-Mamba-L achieves state-of-the-art performance on Waymo test dataset, whose corresponding results are as follows (We also provide the screenshot for results on the Waymo official website with 3 frames in the uploaded global PDF file):  \n\n|    Method     | Epochs | Frames  |   Vehicle   | Pedestrian  |  Cyclist  |   mAP/mAPH (L2)    |\n|:-------------|:------:|:-------:|:-----------:|:-----------:|:---------:|:-----------------:|\n| CenterPoint++ |   30   |  3 |  75.5/75.1  |  75.1/72.4  | 72.0/71.0 |     74.2/72.8     |\n|  PillarNeXt   |   36   |  3 |  76.2/75.8  |  78.8/76.0  | 71.6/70.6 |     75.5/74.1     |\n| **LION-Mamba-L**  |   12   |  3 |  77.2/76.9  |  82.0/79.3  | 76.8/75.9 |     **78.7/77.4**     |\n\n**W3**: The authors should report the inference time of LION so that we have a clearer understanding of the latency and resource consumption of the proposed method.\n\nThanks for your suggestion. We provide the detailed latency of our LION-Mamba and LION-Mamba-L inference time in the following table. We evaluate the latency on one NVIDIA GeForce RTX 3090 with a batch size of 1. For a more detailed discussion of the Transformer-based method DSVT in terms of latency and performance, please refer to W4 of Reviewer 9fJS.\n\n|    Method     | Voxel Extraction (ms)| 3D Backbone (ms)| BEV Backbone (ms)| Detection Head (ms) | Total Latency (ms) | mAPH (L2) |\n|:-------------|:----------------:|:-----------:|:------------:|:--------------:|:------------------:|:-------:|\n|  LION-Mamba   |       4.3        |    97.1     |     27.1     |      17.7      |       146.2        |  73.2   |\n| LION-Mamba-L  |       5.6        |    136.0    |     32.2     |      17.5      |       191.3        |  74.0   |\n\n\n**W4**: When will release the code for the community?\n\nWe will release all codes and models by September 30, 2024. \n\n**Q1**: L124, I’m not clear about the Window Partition. As far as I understand, non-empty voxels are extracted line by line into the window along the X or Y axes. Every time a sequence of 4096 lengths is filled, the remaining non-empty voxels are continued to be placed in the next window. So, how do you deal with it if the final window is not enough for 4096 voxels. Regarding the division and description of windows, it should be clearer. The current description is difficult for me to understand.\n\nWe apologize for the unclear presentation. For the final window that is not enough for 4096 voxels, we repeat these remained voxels in this window up to 4096 voxels. We will make it clearer in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0JTtEa5r4t", "reviewer": "Reviewer_9fJS", "summary": "This paper proposes to leverage linear networks such as RWKV and Mamba to capture long-range dependencies in LiDAR-based outdoor 3D object detection, leading to relatively larger group sizes of the voxel partition. The proposed techniques include voxel merging/expanding and voxel generation. Experiments are conducted on the Waymo Open dataset and nuScenes dataset, achieving state-of-the-art performance.", "questions": "- It seems like the multiple diffused voxels can occupy a single position. How do you handle this situation?\n- How many new voxels will be generated? Does the voxel generation reduce the efficiency?", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is an early attempt to utilize linear RNN for outdoor LiDAR-based 3D object detection.\n- LION achieves state-of-the-art performance on the mainstream datasets.\n- LION can be built upon multiple linear RCNNs such as mamba, RetNet, and RWKV, showcasing the universality of the proposed framework.", "weaknesses": "- Some claims are obscure and not well supported by evidence, such as: \"However, effectively applying linear group RNN to 3D object detection in highly sparse point clouds is not trivial due to its limitation in handling spatial modeling.\" Why is it not trivial? What is the limitation? What is spatial modeling?\n- The proposed techniques are not novel. The spatial descriptor is a common spconv-based module. Voxel merging and expanding are trivial. The voxel generation is new but also similar to some methods, such as the \"virtual voxel\" in \"fsdv2: Improving fully sparse 3d object detection with virtual voxels\". It would be better to conduct a discussion.\n- The ablation for larger group size is not sufficient. The authors should conduct experiments with different group sizes to showcase the workings of group sizes since it is a main claim.\n- Since the most significant advantage of utilizing linear RCNN is efficiency, the authors are encouraged to conduct a more detailed runtime evaluation to reveal the latency of each component and how the designs affect the efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: Some claims are obscure and not well supported by evidence ...\n\nBefore feeding voxel features into linear RNN, we need to flatten 3D voxel features into 1D sequence features. Unlike the common 3D sparse convolution operation that directly deals with 3D voxel features in 3D space, the linear RNN only processes 3D voxel in 1D space, bringing limitations in handling spatial modeling. Here, the limitation is that the linear RNN is a sequence model, which is less effective in perceiving 3D spatial information (e.g. two adjacent voxels). Spatial modeling means capturing the local 3D geometric information for each voxel. \n\n**W2**: The proposed techniques are not novel. \n\n- Although the 3D spatial descriptor consists of a common spconv-based module, it is crucial to help our linear RNN-based network to better capture 3D local spatial information. Therefore, the combination of 3D spatial descriptor and linear RNN operators could make each voxel feature perceive the local spatial information and long-range relation, which is important to improve detection performance.\n- Voxel merging and expanding operations are applied to reduce the computation cost without harming the detection performance. But we do not claim these operations are our contributions (Refer to L63-71 of the main paper).\n- For voxel generation, we make the first attempt to leverage the auto-regressive properties of linear RNNs (refer to Table 5 in the main paper) to generate new voxel features for 3D detection. This is different from the mentioned method FSDv2.\nSpecifically, FSDv2 first votes center points and converts these points to virtual voxels. For virtual voxel features, FSDv2 aggregates these point features by MLP. Therefore, we could categorize it as a KNN manner for voxel diffusion (Refer to the experiments in Table 5 of the main paper). Finally, we will add this discussion in the revised paper.\n\n**W3**: Ablation for larger group size. \n\nThanks. We provide the ablation studies of different group sizes in the following table. Here, we set a minimum group size of 256 for all our four LION blocks (Baseline: [256, 256, 256, 256]). We could observe that the manners with larger group size (i.e, II, III, IV, V) bring consistent performance improvement over the baseline (I). However, we carefully find that there is a drop in performance from IV to V by further enlarging the group size. This might lead to less effective retention of important information in excessively long sequences due to the limited memory capacity of linear RNNs.\n\n|#|Group Size|Vehicle|Pedestrian|Cyclist|mAP/mAPH (L2)|\n|:---|:---|:---:|:---:|:---:|:---:|\n|I|[256, 256, 256, 256]|65.6/65.2|72.3/65.0|68.3/67.2|68.8/65.8|\n|II|[1024, 512, 256, 256]|66.9/66.5|74.9/69.6|70.8/69.8|70.9/68.6|\n|III|[2048, 1024, 512, 256]|66.7/66.3|74.9/69.7|72.2/71.2|71.3/69.1|\n|IV|**[4096, 2048, 1024, 512]**|67.0/66.6|75.4/70.2|71.9/71.0|**71.4/69.3**|\n|V|[8192, 4096, 2048, 1024]|66.5/66.1|74.6/69.5|71.6/70.6|70.9/68.7|\n\n**W4**: More detailed runtime evaluation.\n\nWe provide the detailed latency of each part of our LION-Mamba. Considering that some linear RNN-based operators (Mamba, Retnet, or RWKV) are usually applied to replace Transformer in modeling long sequences thanks to their efficiency. Therefore, we sufficiently compare the latency with transformer-based backbone (DSVT) to illustrate the efficiency of our LION. Here, we evaluate the latency on one NVIDIA GeForce RTX 3090 with a batch size of 1. \nDue to the quadratic complexity of the transformer, DSVT adopts a small group size of 48 in their paper. When we increase the group size of DSVT to 256, the latency obviously increases and the corresponding memory is even unacceptable (produces about 20G GPU memory for inference and fails to train this model due to OOM). \nIn contrast, benefiting from the high efficiency of the linear RNN in modeling long sequences, our LION could adopt a larger group size (4096) for feature interaction while maintaining acceptable latency (146.2 ms) and low GPU memory (about 3G during the inference).\n\n|Component|Voxel Extraction (ms)|3D Backbone (ms)|BEV Backbone (ms)|Detection Head (ms)|Total Latency (ms)|mAPH (L2)|\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|\n|DSVT(official paper)|7.5|82.8|28.6|17.8|136.7|72.1|\n|DSVT(256 group)|7.3|164.4|28.6|17.8|218.1|OOM|\n|LION-Mamba|4.3|97.1|27.1|17.7|146.2|73.2|\n|LION-Mamba-L|5.6|136.0|32.2|17.5|191.3|74.0|\n\nFurthermore, we provide a detailed ablation study on the Waymo validation set with 20% training data to analyze the effect of each component on the efficiency and performance.\n\n|3D Spatial Descriptor|Voxel Generation|Latency (ms)|mAPH (L2)|\n|:---:|:---:|:---:|:---:|\n| | |123.2|65.8|\n|√| |131.3|68.6|\n|√|√|146.2|69.3|\n\n**Q1**: It seems like the multiple diffused voxels can occupy a single position.\n\nWe apologize for the unclear presentation. We merge the diffused voxels and raw voxels by summing these voxels at the same position in the voxel merging operation to deduplicate voxels. We will make it clearer in the revised version.\n\n**Q2**: How many new voxels will be generated? Does the voxel generation reduce the efficiency?\n\nActually, the number of new generated voxels depends on the number of input voxels and the diffusion ratio. Moreover, the voxel generation does reduce the efficiency of the whole network. To better illustrate the effect of voxel generation, we provide the ablation study on the important hyper-parameter of diffusion ratio in the following table. We find that a larger diffusion ratio brings more latency but better performance. In this paper, we set the diffusion ratio as 0.2 to trade off the performance and latency.\n\n|ratio|Latency (ms)|Vehicle|Pedestrian|Cyclist|mAP/mAPH (L2)|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|0|131.3|66.5/66.1|74.8/69.6|70.9/70.0|70.8/68.6|\n|0.1|141.0|66.9/66.5|75.0/69.8|71.5/70.5|71.1/68.9|\n|**0.2**|146.2|67.0/66.6|75.4/70.2|71.9/71.0|**71.4/69.3**|\n|0.5|158.9|67.2/66.8|75.3/70.0|72.1/71.1|71.5/69.3|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful for the valuable suggestion and feedback of all reviewers, which will greatly improve the quality of our paper.  We will carefully revise our paper according to your suggestions.

To Reviewer ehGN and wvps: We provide the visualization of long-range dependencies in the uploaded PDF file.

To Reviewer 5dKF: We provide the anonymous screenshot for testing results on the Waymo official website in the uploaded PDF file.

Meta-review: The paper introduces a linear Group RNN-based backbone for 3D object detection, offering an advantage in achieving a larger window size over previous transformer-based methods. It initially received three positive and one negative scores, with the major concerns including the importance of long-range relations, the novelty of the approach, the need for results on multi-frame data from the Waymo dataset, and the necessity of ablation studies for larger group sizes. After the authors' careful response and multi-round discussion, most concerns were addressed satisfactorily, and most of the reviewers reached a consensus of acknowledging the paper's value to the community. 

Therefore, the meta-reviewer believes the work meets the conference's standards and recommends it for acceptance.
Decision: Accept (poster)