SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Range-view-based 3D Detection. In light of the compactness of the RV representation, certain approaches endeavor to perform detection based on RV. VeloFCN [39] was the first work to perform 3D object detection using RV, which involves transforming the point cloud to a range image and then applying 2D convolution to detect 3D objects. Subsequent research endeavors [29, 30, 9, 10] are put forth to improve the efficacy of RV-based detectors. For instance, LaserNet [29] models the distribution of 3D box corners to capture their uncertainty, resulting in more accurate detections. RCD [30] introduces the range-conditioned dilation mechanism to dynamically adjust the dilation rate based on the measured range, thereby alleviating the scale-sensitivity issue of RV-based detectors. Moreover, RangeDet [9] proposes the Range Conditioned Pyramid to mitigate the scale-variation issue and utilizes the Meta-Kernel convolution to better exploit the 3D geometric information of the points. Despite outperforming all previous RV-based methods, RangeDet lags behind the widely-used BEV-based method CenterPoint by 2.77 average AP, evaluated on WOD. Recently, FCOS-LiDAR [10] proposes a novel range view projection mechanism, and demonstrates the benefits of fusing multi-frame point clouds for a range-view-based detector. Although FCOS-LiDAR is  $90\%$  faster than CenterPoint, its overall validation AP is 3.32 lower, evaluated with multi-frame setting on nuScenes dataset.
Table 1: Detection performance measured by 3D AP/APH on WOD validation set, along with inference speed measured by FPS. All experiments are conducted under single-frame setting.
Method	View	Stage	Vehicle	Pedestrian	Cyclist	Average	FPS
L1	L2	L1	L2	L1	L2	L1	L2
Second	B	one	72.25/71.67	63.84/63.32	68.69/58.12	60.73/51.26	60.61/59.27	58.35/57.03	67.18/63.02	60.97/57.20	29.51
PointPillar	B	one	71.76/71.25	63.50/62.93	69.65/47.41	61.68/41.27	59.03/52.95	56.81/51.38	66.81/57.20	60.66/51.65	41.64
CenterPoint	B	one	74.58/74.04	66.42/65.93	76.17/69.98	68.33/62.63	72.00/70.80	69.39/68.23	74.25/71.61	68.05/65.60	34.73
PV-RCNN	B+P	two	76.70/76.11	68.44/67.95	73.95/63.20	65.68/55.86	67.95/66.25	65.54/63.79	73.47/68.52	66.55/62.53	3.17
Part-A2-anchor	B+P	two	77.03/76.49	68.46/67.98	75.26/66.89	66.19/58.63	68.61/67.37	66.13/64.93	73.63/70.25	66.93/63.85	8.25
RangeDet	R	one	72.85/72.33	64.03/63.57	75.94/71.94	67.60/63.89	65.67/64.39	63.33/62.08	71.48/69.55	64.99/63.18	34.88
RangePerception	R	one	73.62/73.11	66.47/66.00	80.24/76.12	72.29/68.54	70.33/68.93	68.75/67.43	74.73/72.72	69.17/67.32	45.85
Dataset. The experiments in this study utilize the WOD dataset, which is the only dataset that provides native range images. WOD consists of 798 training sequences and 202 validation sequences, with each sequence containing approximately 200 frames. The 64-beam top LiDAR signals are utilized to train and evaluate the RangePerception and baseline models. Scan per cycle of WOD's top LiDAR is 2650, resulting in range image represented by  $I \in \mathbb{R}^{64 \times 2650 \times 8}$ . The metrics of L1/L2 3D AP/APH are calculated and reported following the official evaluation protocol of WOD.
Data Augmentation. Data augmentation techniques are employed during training to improve the model's generalization capabilities. Range images and point clouds are randomly flipped along both the x and y axes and rotated within the range of  $[- \pi / 4, \pi / 4]$ . Additionally, a random global scaling factor between [0.95, 1.05] is applied. The ground-truth copy-paste data augmentation [1] approach is also utilized.
Implementation Details. The RangePerception framework is implemented on top of OpenPCDet codebase [40]. Since OpenPCDet only supports voxel-based and point-based models, range-view data pipeline and detection models are built from scratch. For Vision Restoration Module, restoration angle  $\delta$  is predefined as  $0.086\pi$ , generating  $I^r \in \mathbb{R}^{64 \times 2880 \times 8}$ . For Range Aware Kernel, six Perception Windows are adopted:  $W = \{[0,15], [10,20], [15,30], [20,40], [30,60], [45,\infty)\}$ , resulting in Subspaces  $K \in \mathbb{R}^{6 \times 64 \times 2880 \times 8}$  and transformed Subspaces  $K' \in \mathbb{R}^{64 \times 2880 \times 48}$ . DLA-34 network is adopted as Range Backbone, by updating the input convolution kernel's fan in to 48 channels. Models are learned using Adam optimizer with an initial learning rate of  $3e-3$ , scheduled with the one-cycle learning rate policy. The decay weight is set to 0.01, and momentum range is  $[0.95, 0.85]$ . All models are trained with 30 epochs on WOD training set, where batch size is 32 and frame sampling rate is  $100\%$ . Inference speed is examined with one NVIDIA A100 GPU with batch size set to 1.
Baseline Methods. As shown in Table 1, state-of-the-art BEV-based [1, 2, 6], MV-based [7, 8], and RV-based [9] detectors are selected as baseline methods. All BEV-based and MV-based baselines are trained and evaluated with OpenPCDet's official PyTorch implementation. Since official open-sourced version of RangeDet is coded with MxNet [41], we reimplement RangeDet with PyTorch and integrate RangeDet-PyTorch into OpenPCDet framework. We train RangeDet-PyTorch according to settings presented in their paper and measure its inference speed under OpenPCDet framework. The detection AP/APH of RangeDet is listed according to experimental results in their paper.
Main Results. Detection performance measured by 3D AP/APH is reported in Table 1, where RangePerception is compared against state-of-the-art BEV-based (B), RV-based (R), and MV-based  $(\mathbf{B} + \mathbf{P})$  methods. Inference speed measured by frame per second (FPS) is also presented. It is evident that RangePerception's average AP/APH surpass all baselines, which highlights the strong detection functionality of RangePerception framework. Furthermore, RangePerception achieves state-of-the-art performance in pedestrian AP/APH. We attribute this to the fact that range images better preserve visual features of small objects, while voxelization introduces quantization errors to originally sparse foreground points. The results also demonstrate that RangePerception has the fastest inference speed among all methods. Specifically, RangePerception is 1.32 times as fast as CenterPoint, which is already a highly efficient BEV-based detector. Additionally, RangePerception is the first RV-based detector to achieve higher average AP/APH compared to CenterPoint, outperforming the previous state-of-the-art RV-based method by a large margin.
Ablation Study. Presented in Table 2, an ablation study is conducted to assess the effectiveness of our proposed designs, specifically RAK and VRM. Firstly, the study investigates the impact of RAK
Table 2: Ablation study of RangePerception, measured by 3D AP/APH on WOD validation set.
Setting	Vehicle	Pedestrian	Cyclist	Average
L1	L2	L1	L2	L1	L2	L1	L2
A1	without RAK	70.46/70.01	63.45/62.89	72.86/68.88	64.56/60.72	63.63/62.43	61.44/60.25	68.98/67.11	63.18/61.29
A2	2D Convolution	70.48/70.03	63.46/62.90	72.86/68.87	64.56/60.71	63.64/62.43	61.45/60.26	68.99/67.11	63.16/61.29
A3	Meta-Kernel	72.95/72.43	65.98/65.37	75.95/71.96	67.63/63.90	66.76/65.46	64.57/63.38	71.88/69.95	66.06/64.21
A4	4 Perception Windows	73.13/72.62	65.97/65.51	80.15/76.03	72.17/68.42	70.15/68.85	68.42/67.10	74.48/72.50	68.85/67.01
A5	8 Perception Windows	73.59/73.08	66.45/65.98	80.25/76.13	72.30/68.56	70.31/68.92	68.74/67.42	74.72/72.71	69.16/67.32
A6	without VRM & RP	72.50/71.97	66.40/65.91	80.20/76.08	72.27/68.53	70.31/68.92	68.74/67.42	74.34/72.32	69.13/67.29
RangePerception	73.62/73.11	66.47/66.00	80.24/76.12	72.29/68.54	70.33/68.93	68.75/67.43	74.73/72.72	69.17/67.32
by removing it from our framework (A1) and replacing it with a  $1 \times 1$  Convolution layer of equal dimension (A2). The results show a significant decrease of more than 5 average L1 AP for both cases, highlighting the crucial role of RAK in achieving strong detection performance. Secondly, we compare the performance of RAK and Meta-Kernel by replacing RAK with Meta-Kernel in our framework (A3). Comparison reveals that the RAK setting outperforms the Meta-Kernel setting by an improvement of 2.85 average L1 AP, further validating the efficacy of RAK in our approach. Thirdly, we investigate the optimality of the Perception Window setting in RAK by varying the number of Perception Windows to 4 and 8, respectively (A4, A5). Analysis indicates that reducing the number of Perception Windows leads to a subtle decrease in detection AP, while increasing the number of Perception Windows brings no further performance gain. Lastly, to evaluate the efficacy of VRM, we disable VRM and RP during training and inference processes (A6). Notably, decreases in detection AP are observed for all classes, particularly for vehicles. We attribute this observation to the fact that vehicles, being relatively large objects, are more susceptible to Vision Corruption.
(e)
Figure 6: Qualitative detection results of RangePerception, on a validation frame of WOD. (a) Input range image. (b,c) Dense class predictions for vehicles and pedestrians. (d) Predicted boxes from BEV. (e) Ground-truth boxes from BEV. For (a-c), ego vehicle heads towards the middle of the range image. For (d,e), ego vehicle is visually highlighted in red.
(a) Intensity of input range image
(b) Dense class predictions for vehicles
Figure 7: Qualitative vehicle detection results from RangePerception, on a validation frame of WOD. Notably, the vehicles in this frame are significantly occluded by trees and barriers. Despite these challenges, RangePerception exhibits remarkable detection performance by effectively extracting features from the range view.
Qualitative Results. Detection results on a WOD validation frame are visualized in Fig. 6. Notably, the sparse pedestrian foreground points are accurately preserved in RV, which eases the detection process of RangePerception. Additionally, Fig. 7 showcases RangePerception's capability of handling highly occluded foreground objects.
RangePerception is designed to be highly compatible with point clouds generated from a single viewpoint. Similar to other existing RV-based detectors, RangePerception may not be suitable for perception tasks involving point clouds generated from multiple viewpoints, such as those obtained from multiple autonomous vehicles. However, it is important to emphasize that the occurrence of such multi-viewpoint point clouds is rare in autonomous driving scenarios. Therefore, this limitation does not affect the real-world deployment and application of RangePerception framework.
This paper presents RangePerception, an RV-based 3D detection framework that effectively addresses Spatial Misalignment and Vision Corruption challenges. By introducing RAK and VRM, RangePerception achieves superior detection performance on WOD, showcasing its potential for efficient and accurate real-world deployment.
The research was supported by Shanghai Artificial Intelligence Laboratory, the National Key R&D Program of China (Grant No. 2022ZD0160104) and the Science and Technology Commission of Shanghai Municipality (Grant No. 22DZ1100102).
The authors express their gratitude to Zhongxuan Hu from Huazhong University of Science and Technology for the valuable assistance in refining mathematical representations in this paper.
[36] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[37] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[38] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2017.
[39] Bo Li, Tianlei Zhang, and Tian Xia. Vehicle detection from 3d lidar using fully convolutional network. arXiv preprint arXiv:1608.07916, 2016.
[40] OpenPCDet Development Team. Openpcdet: An open-source toolbox for 3d object detection from point clouds. https://github.com/open-mmlab/OpenPCDet, 2020.
[41] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274, 2015.