SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yeqi Bai<sup>1</sup> Ben Fei<sup>1,2</sup> Youquan Liu<sup>1</sup> Tao Ma<sup>1,3</sup>
Yuenan Hou<sup>1</sup> Botian Shi<sup>1,*</sup> Yikang Li<sup>1</sup>
$^{1}$ Shanghai AI Laboratory  $^{2}$ Fudan University
<sup>3</sup>Multimedia Laboratory, The Chinese University of Hong Kong
{baiyeqi,feiben,liuyouquan,matao}@pjlab.org.cn
{houyuenan,shibotian,liyikang}@pjlab.org.cn
LiDAR-based 3D detection methods currently use bird's-eye view (BEV) or range view (RV) as their primary basis. The former relies on voxelization and 3D convolutions, resulting in inefficient training and inference processes. Conversely, RV-based methods demonstrate higher efficiency due to their compactness and compatibility with 2D convolutions, but their performance still trails behind that of BEV-based methods. To eliminate this performance gap while preserving the efficiency of RV-based methods, this study presents an efficient and accurate RV-based 3D object detection framework termed RangePerception. Through meticulous analysis, this study identifies two critical challenges impeding the performance of existing RV-based methods: 1) there exists a natural domain gap between the 3D world coordinate used in output and 2D range image coordinate used in input, generating difficulty in information extraction from range images; 2) native range images suffer from vision corruption issue, affecting the detection accuracy of the objects located on the margins of the range images. To address the key challenges above, we propose two novel algorithms named Range Aware Kernel (RAK) and Vision Restoration Module (VRM), which facilitate information flow from range image representation and world-coordinate 3D detection results. With the help of RAK and VRM, our RangePerception achieves 3.25/4.18 higher averaged L1/L2 AP compared to previous state-of-the-art RV-based method RangeDet, on Waymo Open Dataset. For the first time as an RV-based 3D detection method, RangePerception achieves slightly superior averaged AP compared with the well-known BEV-based method CenterPoint and the inference speed of RangePerception is 1.3 times as fast as CenterPoint.
In recent years, LiDAR-based 3D perception [1, 2, 3, 4, 5] has made tremendous advances in the field of autonomous driving. One primary task in this area is 3D detection [1, 2, 6, 7, 8, 9, 10], which involves predicting the 3D locations of objects of interest, as well as their geometric and motional properties, including categories, 3D sizes, headings, and velocities. Despite sharing some similarities, there is a fundamental difference between LiDAR-based 3D detection and image-based 2D detection [11, 12, 13, 14, 15, 16]: RGB images are inherently well-structured matrices, whereas LiDAR signals are sets of sparse and unordered points in 3D space. Considering modern computer vision techniques, such as Convolutional Neural Networks (CNNs) [17, 18] and Vision Transformers
(ViTs) [19, 20] require well-structured matrices as inputs, an essential process in LiDAR-based 3D detection is to effectively organize the unstructured LiDAR points.
To effectively organize LiDAR points and facilitate the use of established computer vision techniques, two major representations have been adopted: bird's-eye view (BEV) and range view (RV). Shown in Fig. 1(a-d) is a frame of top LiDAR signal from Waymo Open Dataset (WOD) [21], represented in RV and BEV accordingly. BEV-based 3D detectors [1, 2, 6] convert sparse LiDAR points into 3D voxel grids, extract features with a 3D convolution backbone, and perform classification and regression after mapping the extracted 3D features to the BEV. With the aid of well-organized voxel representation and compatibility with well-developed detection paradigms, BEV-based methods exhibit the highest detection accuracy among contemporary LiDAR-based 3D object detectors. However, it is worth pointing out that a LiDAR sensor in autonomous driving scenarios [22, 23, 24] uniformly samples from a spherical coordinate system, and the cartesian-coordinated voxel representation has certain incompatibility with LiDAR sensors, as evidenced by two factors. First, as the distance from LiDAR grows longer, the distribution of voxels becomes a lot sparser, despite that LiDAR beams are uniformly distributed in the spherical coordinate. Second, when the horizontal length of farthest beam extends to the time of  $l$ , the space of the voxel representation has to increase to the time of  $l^2$  to contain complete information. As a result, BEV-based methods have certain drawbacks in their applications, such as the requirement for complex and time-consuming sparse 3D convolutions as their backbones, limiting their efficiency. Moreover, if users aim to augment horizontal perception range, the computational complexity of BEV-based detectors must increase quadratically, which poses a challenge for real-time implementation.
The range-view (RV) representation, on the other hand, is naturally generated from the scanning mechanism of the LiDAR sensor [25, 26, 27, 28]. Each pixel in the range image corresponds to an optical response of the LiDAR beam, making the range view the most compact and informative way to represent LiDAR signals. The compactness of range images also enables RV-based 3D detectors [29, 30, 9, 10] to enjoy more compact feature spaces and higher inference speeds, compared to the BEV-based 3D detectors. However, in the aspect of detection accuracy, pioneering RV-based 3D detectors significantly lag behind the top-performing BEV-based detectors, with a performance gap of more than 10 average L1 AP on WOD validation set. More recently, RangeDet [9] proposed several modules with stronger representation power, narrowing the performance gap to 2.77 average L1 3D AP on WOD, compared with state-of-the-art BEV-based method CenterPoint [6]. On top of the arts above, FCOS-LiDAR [10] develops a multi-frame RV-based detection pipeline on nuScenes dataset [31, 32]. Despite being  $90\%$  faster than CenterPoint, FCOS-LiDAR's overall validation AP is 3.32 lower, evaluated with multi-frame setting on nuScenes dataset.
To better exploit the potential of the range-view representation, a detailed analysis of existing RV-based detection methods is conducted, which reveals two critical unsolved challenges.
Spatial Misalignment. Existing RV-based detectors treat range images the same way as RGB images, by directly feeding them into 2D convolution backbones. This workflow neglects the nature that range images contain rich depth information, and even two range pixels are adjacent in range coordinate, their actual distance in 3D space could be more than 30 meters. As visualized in Fig. 1(e), foreground pixels on the margins of vehicles and pedestrians are often far from their neighboring background pixels in 3D space. Directly processing such 3D-space-uncorrelated pixels with 2D convolution kernels can only produce noisy features, hindering geometric information extraction from the margins of foreground objects. This phenomenon will be termed as Spatial Misalignment in further discussion of this paper.
Vision Corruption. When objects of interest are located on the margins of range images, as shown in Fig. 1(c,f), their corresponding foreground range pixels are separately distributed around the left and right borders of the range image. Since CNNs have limited receptive fields, features around the left border cannot be shared with features around the right border and vice versa, when 2D convolution backbones are used as feature extractors. This phenomenon, called Vision Corruption, can significantly impact the detection accuracy of objects on the margins of range images. Previous RV-based detection methods have overlooked this issue and directly processed range images with 2D convolution backbones without compensating for the corrupted areas.
In this paper, we demonstrate an efficient and accurate RV-based 3D detection framework, termed RangePerception. To overcome the key challenges above, two novel algorithms named Range Aware Kernel (RAK) and Vision Restoration Module (VRM) are proposed and integrated into RangePer
(a) Range of LiDAR points (RV)
(b) Intensity of LiDAR points (RV)
(c) Foreground LiDAR points of Vehicles (RV)
(d) LiDAR points (BEV)
Figure 1: (a-d) A sample frame of top LiDAR signal, represented in RV and BEV respectively. (e) Spatial Misalignment phenomena. (f) Vision Corruption phenomena.
(e) Range distribution of foreground points (RV)
(f) Vision Corruption phenomena (RV)
ception framework, both facilitating information flow from range image representation and world-coordinate 3D detection results. With the help of RAK and VRM, our RangePerception presents 73.62 & 80.24 & 70.33 L1 3D AP for vehicle & pedestrian & cyclist, on WOD, achieving state-of-the-art performance as a range-view-based 3D detection method. The contributions of this paper are presented as follows.
RangePerception Framework. A novel high-performing 3D detection framework, named RangePerception, is introduced in this paper. RangePerception is the first RV-based 3D detector to achieve 74.73/69.17 average L1/L2 AP on WOD, outperforming the previous state-of-the-art RV-based detector RangeDet, which has average L1/L2 APs of 71.48/64.99, presenting an improvement of 3.25/4.18. RangePerception also demonstrates slightly superior performance compared to widely-used BEV-based method CenterPoint [6], which has average L1/L2 APs of 74.25/68.04. Notably, RangePerception's inference speed is 1.3 times as fast as CenterPoint, justifying better fitness for real-time deployment on autonomous vehicles.[2]
Range Aware Kernel. As part of RangePerception's feature extractor, Range Aware Kernel (RAK) is a trailblazing algorithm tailored to RV-based networks. RAK disentangles the range image space into multiple subspaces, and overcomes the Spatial Misalignment issue by enabling independent feature extraction from each subspace. Experimental results show that RAK lifts the average L1/L2 AP by 5.75/5.99, while incurring negligible computational cost.
<sup>2</sup>Project website is available at https://rangeperception.github.io, to enhance the accessibility and comprehension of this study.
Figure 2: The RangePerception framework takes a range image  $I$  as input and generates dense predictions. To improve representation learning, the framework sequentially integrates the VRM and RAK modules before the Range Backbone. Subsequently, a specially devised Redundancy Pruner is used to remove redundancies in the deep features, thereby mitigating the computational cost in the subsequent Region Proposal Network and post-processing layers.
Vision Restoration Module. To resolve the Vision Corruption issue, Vision Restoration Module (VRM) is brought to light in this study. VRM extends the receptive field of the backbone network by restoring previously corrupted areas. VRM is particularly helpful to the detection of vehicles, as will be illustrated in the experiment section.
2 Preliminary of Range-view Representation
This section provides a brief overview of the range view representation of LiDAR data. Specifically, LiDAR data can be represented as an  $m \times n$  matrix, known as a range image, where  $m$  represents the number of beams emitted and  $n$  represents the number of measurements taken during one scan cycle. Each column of the range image corresponds to a shared azimuth, while each row corresponds to a shared inclination, indicating the relative vertical and horizontal angles of a returned point with respect to the LiDAR's original point. Each pixel in the range image contains at least three geometric values, namely range  $r$ , azimuth  $\theta$ , and inclination  $\phi$ , which define a spherical coordinate system. The widely-used point cloud data with Cartesian coordinates is derived from the spherical coordinate system:  $x = r\cos (\phi)\cos (\theta), y = r\cos (\phi)\sin (\theta), z = r\sin (\phi)$ , where  $x, y, z$  denote the Cartesian coordinates of the points. Considering modern LiDAR sensors [33, 34, 35] often measure magnitude of the returned laser pulse named intensity  $\eta$  and elongation  $\rho$ , range view of LiDAR signal can be engineered as  $I \in \mathbb{R}^{m\times n\times 8}$ , with each pixel being  $I_{j,k} = \{r,x,y,z,\theta ,\phi ,\eta ,\rho \}$ . This study additionally defines range matrix as  $R \in \mathbb{R}^{m\times n}$ , with each pixel being  $R_{j,k} = r$ , for the sake of further discussion. To provide a better illustration, Fig. 1(a) presents a sample frame of range matrix  $R$  and Fig. 1(b) shows intensity values in the corresponding range image  $I$ .
The RangePerception framework, depicted in Fig. 2, takes range image  $I$  as input and produces dense predictions. To enhance representation learning, VRM and RAK are sequentially incorporated before the Range Backbone. Afterwards, a carefully designed Redundancy Pruner is employed to eliminate redundancies in deep features, which minimizes computational costs in subsequent Region Proposal Network (RPN) and post-processing layers. This section begins with comprehensive explanations of RAK and VRM, followed by an in-depth elaboration of RangePerception architecture.
(a) Perception Windows for RAK
(c) Range of LiDAR points in range image  $I$
(d) Intensity of LiDAR points in range image  $I$
(e) Intensity of LiDAR points in subspace  $K_{1}$
(b) Remedy for Spatial Misalignment
Figure 3: Range Aware Kernel disentangles the range image space into multiple subspaces, and overcomes the Spatial Misalignment issue by enabling independent feature extraction from each subspace.
(f) Intensity of LiDAR points in subspace  $K_{4}$
3.1 Range Aware Kernel
As a key component of RangePerception's feature extractor, Range Aware Kernel is an innovative algorithm specifically designed for RV-based networks. RAK disentangles the range image space into multiple subspaces, and overcomes the Spatial Misalignment issue by enabling independent feature extraction from each subspace. Shown in Fig. 3(a), Range Aware Kernel comes with a set of  $l$  predefined Perception Windows, formulated as  $W = \{w_{1}, w_{2}, \dots, w_{l-1}, w_{l}\}$ , where each Perception Window is a range-conditioned interval  $w_{i} = [r_{i1}, r_{i2}]$ .
Given a frame of range image  $I \in \mathbb{R}^{m \times n \times 8}$ , RAK first calculates binary mask  $M \in \mathbb{Z}^{l \times m \times n \times 8}$  according to Perception Windows  $W = \{w_{1}, w_{2}, \dots, w_{l-1}, w_{l}\}$ , where each  $M_{i} \in \mathbb{Z}^{m \times n \times 8}$  is a pixel-wise mask for range image  $I$ , indicating whether each range value  $R_{j,k} = I_{j,k,1}$  stays in current Perception Window  $w_{i}$ . Subsequently, RAK defines a tensor  $K \in \mathbb{R}^{l \times m \times n \times 8}$  representing  $l$  subspaces, and derives each subspace  $K_{i} = M_{i} \odot I$ . Detailed computing logic of RAK is illustrated in Algorithm 1, note that though inference process of RAK seems to incur  $O(lmn)$  time complexity, GPU implementation of RAK can readily achieve  $O(1)$  with proper parallelism.
As elaborated above, RAK divides range image  $I$  into multiple subspaces  $K$ , where each subspace  $K_{i}$  contains LiDAR points that belong to Perception Window  $w_{i}$ . To provide a clearer visualization, Fig. 3(c,d) displays range and intensity values of a frame of input range image  $I$ . By further processing range image  $I$  with RAK, tensor  $K$  is computed, from which intensity values of subspaces  $K_{1}$  and  $K_{4}$  are presented in Fig. 3(e,f). RAK effectively separates foreground vehicle points from their background counterparts, thus minimizing Spatial Misalignment and facilitating feature extraction from range view. Fig. 3(b) provides further evidence of the efficacy of RAK, by clearly disentangling previously indistinguishable vehicles from noisy background points.
In the architecture of RangePerception, Range Aware Kernel is positioned directly before backbone network. Subspaces  $K \in \mathbb{R}^{1 \times m \times n \times 8}$ , generated from Range Aware Kernel, is subsequently fed into the backbone network for non-linear feature extraction. Experimental results demonstrate that RAK increases the average L1/L2 AP by 5.75/5.99, while incurring negligible computational cost.
3.2 Vision Restoration Module
As described in Sec. 2, each column in range image  $I$  corresponds to a shared azimuth  $\theta \in [0,2\pi ]$  indicating the spinning angle of LiDAR. Specifically,  $\theta = 0$  at left margin of range image and  $\theta = 2\pi$  at right margin of range image. Due to the periodicity of LiDAR's scanning cycle, azimuth values  $0$  and  $2\pi$  correspond to beginning and end of each scanning cycle, both pointing in the opposite
Algorithm 1 Range Aware Kernel
1: input range image  $I \in \mathbb{R}^{m \times n \times 8}$   
2: init range matrix  $R \in \mathbb{R}^{m \times n}$   
3: init  $i, j, k \in \mathbb{Z}$   
4: for  $j \in [1, m], k \in [1, n]$  do  
5:  $R_{j,k} \gets I_{j,k,1}$   
6: init tensor  $K \in \mathbb{R}^{l \times m \times n \times 8}$   
7: init binary mask  $M \in \mathbb{Z}^{l \times m \times n \times 8}$   
8: for  $w_i \in W$  do  
9: for  $R_{j,k}$  in  $R$  do  
10: if  $R_{j,k} \in w_i$  then  
11:  $M_{i,j,k} \gets 1$   
12: else if  $R_{j,k} \notin w_i$  then  
13:  $M_{i,j,k} \gets 0$   
14:  $K_i \gets M_i \odot I$   
15: output tensor  $K$
Figure 4: Spherical Coordinate of VRM.
direction of the ego vehicle. As illustrated in Fig. 4, objects located behind ego vehicle are often separated by ray with  $\theta = 0$ , resulting in Vision Corruption phenomena elaborated in Fig. 1(c,f).
To resolve Vision Corruption issue, Vision Restoration Module is brought to light in this study. By predefining a restoration angle  $\delta$ , VRM builds an extended spherical space with azimuth  $\theta \in [-\delta, 2\pi + \delta]$ . In this way, visual features originally corrupted by LiDAR's sampling process are restored on both sides of range image  $I$ , significantly easing the feature extraction from the margins of  $I$ . This restoration process is clearly visualized in Fig. 5(a,b), where VRM-extended range images can be termed as  $I^r$ . In the architecture of RangePerception, restored range images  $I^r$  are subsequently processed by RAK and range backbone for deeper pattern recognition. Note that in Sec. 3.1, input range image of RAK is still denoted as  $I$  instead of  $I^r$ , for the sake of simplicity.
It is straightforward to observe from Fig. 5(a,b) that VRM introduces redundancies to range view: region with azimuth  $\theta \in [-\delta, \delta]$ , appears twice in restored range image  $I^r$ . Though this duplication helps information extraction from the margins of  $I$ , redundancies persist in feature space  $F^r$  learnt by RAK and range backbone, leading to unnecessary computational costs for subsequent region proposal network and post-processing layers. To address this issue and improve efficiency, Redundancy Pruner (RP) is designed and equipped with RangePerception framework. Operating on feature space  $F^r$ , RP performs inverse function of VRM, by pruning  $F^r$  from azimuth interval  $[- \delta, 2\pi + \delta]$  back to  $[0, 2\pi]$ .
To better explain the process above, a pseudo VRM-extended image  $I^r$  is rendered in Fig. 5(c). Vanilla pixels in  $I$  are filled with zeros, while pixels generated by VRM are filled with ones. A pseudo feature space  $F^r$  is subsequently computed, via inputting pseudo image  $I^r$  to RAK and backbone. Finally, RP drops redundancies in pseudo  $F^r$ , resulting in pruned feature space  $F$ . As observed from lower part of Fig. 5(c), visual information that belongs to VRM-restored spaces readily flows to vanilla space, thanks to the layered convolution kernels. This justifies that Redundancy Pruner solely decreases computational costs, without causing any loss in visual features.
3.3 RangePerception Framework
For the sake of efficiency, RangePerception is designed as an anchor-free single-stage detector, as presented in Fig. 2. Given a frame of range image  $I$ , RangePerception framework first compensates the corrupted regions with VRM, resulting in restored image  $I^r \in \mathbb{R}^{m \times \mu \times 8}$ , where  $\mu = \frac{m(\delta + \pi)}{\pi}$ . Subsequently, RAK converts  $I^r$  into subspaces  $K \in \mathbb{R}^{l \times m \times \mu \times 8}$ , disentangling misaligned pixels. Range Backbone, derived from DLA-34 [36], is adopted to extract non-linear features  $F^r \in \mathbb{R}^{m \times \mu \times c}$  from subspaces. Further, RP eliminates redundant features, generating pruned features  $F \in \mathbb{R}^{m \times n \times c}$ . RPN learns dense predictions  $\{C, B, U\}$  on top of deep features  $F$ , representing class predictions, box predictions, and IoU predictions accordingly. Finally, Weight NMS is employed to aggregate dense predictions, where IoU predictions  $U$  are treated as per-box confidence scores.
(a) Vision Restoration of range matrix
(b) Vision Restoration of foreground points
(c) Pseudo feature propagation
Figure 5: Vision Restoration Module. By predefining a restoration angle  $\delta$ , VRM builds an extended spherical space with azimuth  $\theta \in [-\delta, 2\pi + \delta]$ . As a result, Vision Corruption issues are resolved on both sides of range image  $I$ , significantly easing the feature extraction from the margins of  $I$ .
BEV-based 3D Detection. The majority of the highest-performing LiDAR-based 3D detectors are categorized as BEV-based detection, where the initial step involves the conversion of the point cloud into BEV (Bird's Eye View) images. VoxelNet [37] is the pioneering end-to-end BEV-based detector that utilizes PointNet [38] for inner-voxel representation and 3D convolutions for high-level feature generation, where high-level features are further processed by region proposal network (RPN). To reduce the computational burden of 3D convolutions, SECOND [1] introduces the usage of sparse convolutions. Another common approach is to eliminate voxelization along the elevation axis and convert the point cloud into pillars instead of voxels, as proposed by [2]. By leveraging either voxel-based or pillar-based BEV representation, CenterPoint [6] achieves state-of-the-art performance levels in a center-based anchor-free manner. However, complex and time-consuming sparse 3D convolutions in these methods hinder their practical applications, and lightweight 3D detectors urgently need to be developed.
Multi-view-based 3D Detection. Most top-performing multi-view-based (MV-based) 3D detectors [7, 8] adopt a two-stage approach, where the first stage typically employs a BEV-based detector, and point-view features are subsequently utilized in the second stage for proposal refinement. PV-RCNN [7] combines 3D voxel CNN and PointNet-based [38] set abstraction to learn discriminative features from point clouds. Part- $A^2$  [8] introduces part-awareness multi-view aggregation, achieving outstanding performance by sequentially predicting coarse 3D proposals and accurate intra-object part locations. However, despite the excellent detection performance achieved by MV-based methods, their inference speed remains impractical for real-time deployment on vehicles due to the computational complexity of their structure.