SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiakang Yuan $^{1,\ast}$ , Bo Zhang $^{2,\dagger}$ , Xiangchao Yan $^{2}$ , Tao Chen $^{1,\dagger}$ , Botian Shi $^{2}$ , Yikang Li $^{2}$ , Yu Qiao $^{2}$
$^{1}$ School of Information Science and Technology, Fudan University
$^{2}$ Shanghai Artificial Intelligence Laboratory
jkyuan22@m.fudan.edu.cn, zhangbo@pjlab.org.cn, eetchen@fudan.edu.cn
It is a long-term vision for Autonomous Driving (AD) community that the perception models can learn from a large-scale point cloud dataset, to obtain unified representations that can achieve promising results on different tasks or benchmarks. Previous works mainly focus on the self-supervised pre-training pipeline, meaning that they perform the pre-training and fine-tuning on the same benchmark, which is difficult to attain the performance scalability and cross-dataset application for the pre-training checkpoint. In this paper, for the first time, we are committed to building a large-scale pre-training point-cloud dataset with diverse data distribution, and meanwhile learning generalizable representations from such a diverse pre-training dataset. We formulate the point-cloud pre-training task as a semi-supervised problem, which leverages the few-shot labeled and massive unlabeled point-cloud data to generate the unified backbone representations that can be directly applied to many baseline models and benchmarks, decoupling the AD-related pre-training process and downstream fine-tuning task. During the period of backbone pre-training, by enhancing the scene- and instance-level distribution diversity and exploiting the backbone's ability to learn from unknown instances, we achieve significant performance gains on a series of downstream perception benchmarks including Waymo, nuScenes, and KITTI, under different baseline models like PV-RCNN++, SECOND, CenterPoint. Project page: https://jiakangyuan.github.io/AD-PT.github.io/.
LiDAR sensor plays a crucial role in the Autonomous Driving (AD) system due to its high quality for modeling the depth and geometric information of the surroundings. As a result, a lot of studies aim to achieve the AD scene perception via LiDAR-based 3D object detection baseline models, such as CenterPoint [32], PV-RCNN [17], and PV-RCNN++ [19].
Although the 3D object detection models can help autonomous driving recognize the surrounding environment, the existing baselines are hard to generalize to a new domain (such as different sensor settings or unseen cities). A long-term vision of the autonomous driving community is to develop a scene-generalizable pre-trained model, which can be widely applied to different downstream tasks. To achieve this goal, researchers begin to leverage the Self-Supervised Pre-Training (SS-PT) paradigm recently. For example, ProposalContrast [31] proposes a contrastive learning-based approach to enhance region-level feature extraction capability. Voxel-MAE [6] designs a masking and reconstructing task to obtain a stronger backbone network.
* This work was done when Jiakang Yuan was an intern at Shanghai Artificial Intelligence Laboratory.
† Corresponding to Tao Chen and Bo Zhang.
(a) Previous pre-training paradigm.
(b) AD-PT pre-training paradigm.
Figure 1: Differences between previous pre-training paradigm and the proposed AD-PT paradigm.
However, as illustrated in Fig.1, it should be pointed out that there is a crucial difference between the above-mentioned SS-PT and the desired Autonomous Driving Pre-Training (AD-PT) paradigm. The SS-PT aims to learn from a single set of unlabeled data to generate suitable representations for the same dataset, while AD-PT is expected to learn unified representations from as large and diversified data as possible, so that the learned features can be easily transferred to various downstream tasks. As a result, SS-PT may only perform well when the test and pre-training data are sampled from the same single dataset (such as Waymo [20] or nuScenes [1]), while AD-PT presents better generalized performance on different datasets, which can be continuously improved with the increase of the number of the pre-training dataset.
Therefore, this paper is focused on achieving the AD-related pre-training which can be easily applied to different baseline models and benchmarks. By conducting extensive experiments, we argue that there are two key issues that need to be solved for achieving the real AD-PT: 1) how to build a unified AD dataset with diverse data distribution, and 2) how to learn generalizable representations from such a diverse dataset by designing an effective pre-training method.
For the first item, we use a large-scale point cloud dataset named ONCE [14], consisting of few-shot labeled  $(e.g., \sim 0.5\%)$  and massive unlabeled data. First, to get accurate pseudo labels of the massive unlabeled data that can facilitate the subsequent pre-training task, we design a class-wise pseudo labeling strategy that uses multiple models to annotate different semantic classes, and then adopt semi-supervised methods (e.g., MeanTeacher [21]) to further improve the accuracy on the ONCE validation set. Second, to get a unified dataset with diverse raw data distribution from both LiDAR beam and object sizes, inspired by previous works [29, 24, 33, 34], we exploit point-to-beam playback re-sampling and object re-scaling strategies to diversify both scene- and region-level distribution.
For the second item, we find that the taxonomy differences between the pre-training ONCE dataset and different downstream datasets are quite large, resulting in that many hard samples with taxonomic inconsistency are difficult to be accurately detected during the fine-tuning stage. As a result, taxonomy differences between different benchmarks should be considered when performing the backbone pretraining. Besides, our study also indicates that during the pre-training process, the backbone model tends to fit with the semantic distribution of the ONCE dataset, impairing the perception ability on downstream datasets having different semantics. To address this issue, we propose an unknown-aware instance learning to ensure that some background regions on the pre-training dataset, which may be important for downstream datasets, can be appropriately activated by the designed pre-training task. Besides, to further mine representative instances during the pre-training, we design a consistency loss to constrain the pre-training representations from different augmented views to be consistent.
Our contributions can be summarized as follows:
1. For the first time, we propose the AD-PT paradigm, which aims to learn unified representations by pre-training a general backbone and transfers knowledge to various benchmarks.
2. To enable the AD-PT paradigm, we propose a diversity-based pre-training data preparation procedure and unknown-aware instance learning, which can be employed in the backbone pre-training process to strengthen the representative capability of extracted features.
3. Our study provides a more unified approach, meaning that once the pre-trained checkpoint is generated, it can be directly loaded into multiple perception baselines and benchmarks. Results further verify that such an AD-PT paradigm achieves large accuracy gains on different benchmarks (e.g.,  $3.41\%$ ,  $8.45\%$ ,  $4.25\%$  on Waymo, nuScenes, and KITTI).
Figure 2: The overview of the proposed AD-PT. By leveraging the proposed method to train on the unified large-scale point cloud dataset, we can obtain well-generalized pre-training parameters that can be applied to multiple datasets and support different baseline detectors.
2.1 LiDAR-based 3D Object Detection
Current prevailing LiDAR-based 3D object detection works [11, 16, 17, 19, 27, 18, 4] can be roughly divided into point-based methods, voxel-based methods, and point-voxel-based methods. Point-based methods [16, 30] extract features and generate proposals directly from raw point clouds. PointRCNN [16] is a prior work that generates 3D RoIs with foreground segmentation and designs an RCNN-style two-stage refinement. Unlike point-based methods, voxel-based methods [27, 32] first transform unordered points into regular grids and then extract 3D features using 3D convolution. As a pioneer, SECOND [27] utilizes sparse convolution as 3D backbone and greatly improves the detection efficiency. CenterPoint [32] takes care of both accuracy and efficiency and proposes a one-stage method. To take advantage of both point-based and voxel-based methods, PV-RCNN [17] and PV-RCNN++ [19] propose a point-voxel set abstraction to fuse point and voxel features.
2.2 Autonomous Driving-related Self-Supervised Pre-Training
Inspired by the success of pre-training in 2D images, self-supervised learning methods have been extended to LiDAR-based AD scenarios [6, 13, 12, 28, 26, 10, 31]. Previous methods mainly focus on using contrastive learning or masked autoencoder (MAE) to enhance feature extraction. Contrastive learning-based methods [12, 31, 7] use point clouds from different views or temporally-correlated frames as input, and further construct positive and negative samples. STRL [7] proposes a spatial-temporal representation learning that employs contrastive learning with two temporally-correlated frames. GCC-3D [12] and ProposalContrast constructs a consistency map to find the correspondence between different views, developing contrastive learning strategies at region-level. CO3 [2] utilizes LiDAR point clouds from the vehicle- and infrastructure-side to build different views. MAE-based [6, 10] methods utilize different mask strategies and try to reconstruct masked points by a designed decoder. Voxel-MAE [6] introduces a voxel-level mask strategy and verifies the effectiveness of MAE. BEV-MAE [13] designs a BEV-guided masking strategy. More recently, GD-MAE [28] and MV-JAR [26] explore masking strategies in transformer architecture. Different from the existing works that use a designed self-supervised approach to pre-train on unlabeled data and then fine-tune on labeled data with the same dataset, AD-PT aims to pre-train on a large-scale point cloud dataset and fine-tune on multiple different datasets (i.e., Waymo [20], nuScenes[1], KITTI [5]).
Table 1: Performance using different detectors on ONCE validation set. We report mAP using ONCE evaluation metric.
Detector	Head Choice	Vehicle	Pedestrian	Cyclist
ONCE Benchmark (Best)	Center Head	66.79	49.90	63.45
CenterPoint (ours)	Center Head	-	56.01	-
PV-RCNN++ (ours)	Anchor Head	82.50	-	71.19
Figure 3: Overall dataset preparation procedure.
Table 2: Statistics on the number of pseudolabeled instances per frame. We compare it with ONCE labeled data set.
ONCE labeled set	Pseudo label set
Vehicle	Ped.	Cyclist	Vehicle	Ped.	Cyclist
19.01	4.52	5.63	15.67	1.63	1.90
Figure 4: Statistics of object re-scaling.
To better illustrate our AD-PT framework, we first briefly describe the problem definition and overview of the proposed method in Sec. 3.1. Then, we detail the data preparation and model design of AD-PT in Sec. 3.2 and Sec. 3.3, respectively.
3.1 Preliminary
Problem Formulation. Different from previous self-supervised pre-training methods, AD-PT performs large-scale point cloud pre-training in a semi-supervised manner. Specifically, we have access to  $N$  samples which are composed of a labeled set  $D_{L} = \{(x_{i},y_{i})\}_{i = 1}^{N_{L}}$  and a unlabeled set  $D_{U} = \{x_{i}\}_{i = 1}^{N_{U}}$ , where  $N_{L}$  and  $N_{U}$  denote the number of labeled and unlabeled samples. Note that  $N_{L}$  can be much smaller than  $N_{U}$  (e.g.,  $\sim 5\mathrm{K}$  v.s.  $\sim 1\mathrm{M}$ , about  $0.5\%$  labeled samples). The purpose of our work is to pre-train on a large-scale point cloud dataset with few-shot  $D_{L}$  and massive  $D_{U}$ , such that the pre-trained backbone parameters can be used for many down-stream benchmark datasets or 3D detection models.
Overview of AD-PT. As shown in Fig. 2, AD-PT mainly consists of a large-scale point cloud dataset preparation procedure and a unified AD-focused representation learning procedure. To initiate the pre-training, a class-aware pseudo labels generator is first developed to generate the pseudo labels of  $D_U$ . Then, to get more diverse samples, we propose a diversity-based pre-training processor. Finally, in order to pre-train on these pseudo-labeled data to learn their generalizable representations for AD purposes, an unknown-aware instance learning coupled with a consistency loss is designed.
3.2 Large-scale Point Cloud Dataset Preparation
In this section, we detail the preparation of the unified large-scale dataset for AD. As shown in Fig. 3, our data creation consists of a class-aware pseudo label generator and a diversity-based pre-training processor to be introduced below.
3.2.1 Class-aware Pseudo Labels Generator
It can be observed from Tab. 7 that, pseudo-labels with high accuracy on the pre-training dataset are beneficial to enhance the detection accuracy on downstream datasets such as Waymo [20] and nuScenes [1]. Therefore, to get more accurate pseudo labels, we design the following procedure.
Class-aware Pseudo Labeling. ONCE benchmark<sup>3</sup> is utilized to evaluate the pseudo labeling accuracy and more results are shown in the supplementary material. We find that different baseline
<sup>3</sup>https://once-for-auto-driving.github.io/benchmark.html
(a) Original point clouds
(b) Upsampled point clouds
(c) Downsampled point clouds
Figure 5: Visualization of point-to-beam playback re-sampling.
models have different biased perception abilities for different classes. For example, the center-based method (i.e., CenterPoint [32]) tends to detect better for small-scale targets (e.g., Pedestrian), while anchor-based methods perform better for other classes. According to the observation, we utilize the PV-RCNN++ [19] with anchor-head to annotate Vehicle and Cyclist classes on ONCE, where the accuracy on Vehicle and Cyclist is found to be much better than the methods listed in the ONCE benchmark. On the other hand, CenterPoint [32] is employed to label the Pedestrian class on ONCE. Finally, we use PV-RCNN++ and CenterPoint to perform a class-wise pseudo labeling process.
Semi-supervised Data Labeling. We further employ the semi-supervised learning method to fully exploit the unlabeled data to boost the accuracy of the pseudo labels. As shown in ONCE benchmark, MeanTeacher [21] can better improve the performance on ONCE, and therefore, we use the MeanTeacher to further enhance the class-wise detection ability. Tab. 1 shows that the accuracy on ONCE validation set can be improved after leveraging massive unlabeled data, significantly surpassing all previous records.
Pseudo Labeling Threshold. To avoid labeling a large number of false positive instances, we set a relatively high threshold. In detail, for Vehicle, Pedestrian, and Cyclist, we filter out bounding boxes with confidence scores below 0.8, 0.7, and 0.7. As a result, it can be seen from Tab. 2 that, compared with ONCE labeled data, some hard samples with relatively low prediction scores are not annotated.
3.2.2 Diversity-based Pre-training Processor
As mentioned in [15, 9], the diversity of data is crucial for pre-training, since highly diverse data can greatly improve the generalization ability of the model. The same observation also holds in 3D pre-training. However, the existing datasets are mostly collected by the same LiDAR sensor within limited geographical regions, which impairs the data diversity. Inspired by [33, 34], discrepancies between different datasets can be categorized into scene-level (e.g., LiDAR beam) and instance-level (e.g., object size). Thus, we try to increase the diversity from the LiDAR beam and object size, and propose a point-to-beam playback re-sampling and an object re-scaling strategy.
Data with More Beam-Diversity: Point-to-Beam Playback Re-sampling. To get beam-diverse data, we use the range image as an intermediate variable for point data up-sampling and down-sampling. Specifically, given a LiDAR point cloud with  $n$  beam (e.g., 40 beam for ONCE dataset) and  $m$  points per ring, the range image  $R^{n \times m}$  can be obtained by the following equation:
$$
r = \sqrt {x ^ {2} + y ^ {2} + z ^ {2}}, \theta = \arctan (x / y), \phi = \arcsin (z / r), \tag {1}
$$
where  $\phi$  and  $\theta$  are the inclination and azimuth of point clouds, respectively, and  $r$  denotes the range of the point cloud. Each column and row of the range image corresponds to the same azimuth and inclination of the point clouds, respectively. Then, we can interpolate or sample over rows of a range image, which can also be seen as LiDAR beam re-sampling. Finally, we reconvert the range image to point clouds as follows:
$$
x = r \cos (\phi) \cos (\theta), y = r \cos (\phi) \sin (\theta), z = r \sin (\phi), \tag {2}
$$
where  $x, y, z$  denote the Cartesian coordinates of points, and Fig. 5 shows that point-to-beam playback re-sampling can generate scenes with different point densities, improving the scene-level diversity.
Data with More RoI-Diversity: Object Re-scaling. According to the statement in [34, 29, 23] that different 3D datasets were collected in different locations, the object size has inconsistent distributions. As shown in Fig. 4, a single dataset such as ONCE cannot cover a variety of object-size distributions,
resulting in that the model cannot learn a unified representation. To overcome such a problem, we propose an object re-scaling mechanism that can randomly re-scale the length, width and height of each object. In detail, given a bounding box and points within it, we first transform the points to the local coordinate and then multiply the point's coordinates and the bounding box size by a provided scaling factor. Finally, we transform the scaled points with the bounding box to the ego-car coordinate. It can be seen from Fig. 4 that after object re-scaling, the produced dataset contains object sizes with more diversified distributions, further strengthening the instance-level point diversity.
3.3 Learning Unified Representations under Large-scale Point Cloud Dataset
By obtaining a unified pre-training dataset using the above-mentioned method, the scene-level and instance-level diversity can be improved. However, unlike 2D or vision-language pre-training datasets which cover a lot of categories to be identified for downstream tasks, our pseudo dataset has limited category labels (i.e., Vehicle, Pedestrian and Cyclist). Besides, as mentioned in Sec. 3.2.1, in order to get accurate pseudo annotations, we set a high confidence threshold, which may inevitably ignore some hard instances. As a result, these ignored instances, which are not concerned in the pre-training dataset but may be seen as categories of interest in downstream datasets (e.g., Barrier in the nuScenes dataset), will be suppressed during the pre-training process.
To mitigate such a problem, it is necessary that both pre-training-related instances and some unknown instances with low scores can be considered when performing the backbone pre-training. From a new perspective, we regard the pre-training as an open-set learning problem. Different from traditional open-set detection [8] which aims at detecting unknown instances, our goal is to activate as many foreground regions as possible during the pre-training stage. Thus, we propose a two-branch unknown-aware instance learning head to avoid regarding potential foreground instances as the background parts. Further, a consistency loss is utilized to ensure the consistency of the calculated corresponding foreground regions.
Overall Model Structure. In this part, we briefly introduce our pre-training model structure. Following the prevailing 3D detectors [17, 3, 27], as shown in Fig. 2, the designed pre-training model consists of a voxel feature extractor, a 3D backbone with sparse convolution, a 2D backbone and the proposed head. Specifically, given point clouds  $\mathbf{P} \in \mathcal{R}^{N \times (3 + d)}$ , we first transform points into different views through different data augmentation methods  $\Gamma_{1}$  and  $\Gamma_{2}$ . Then, voxel features are extracted by a 3D backbone and mapped to BEV space. After that, dense features generated by a 2D backbone can be obtained, and finally, the dense features are fed into the proposed head.
Unknown-aware Instance Learning Head. Inspired by previous open-set detection works [8, 35], we consider background region proposals with relatively high objectness scores to be unknown instances that are ignored during the pre-training stage but may be crucial to downstream tasks, where the objectness scores are obtained from the Region Proposal Network (RPN). However, due to that these unknown instances contain a lot of background regions, directly treating these instances as the foreground instances during the pre-training will cause the backbone network to activate a large number of background regions. To overcome such a problem, we utilize a two-branch head as a committee to discover which regions can be effectively presented as foreground instances. Specifically, given the RoI features  $\mathbf{F}^{\Gamma_1} = [f_1^{\Gamma_1}; f_2^{\Gamma_1}; \ldots; f_N^{\Gamma_1}] \in \mathcal{R}^{N \times C}$ ,  $\mathbf{F}^{\Gamma_2} = [f_1^{\Gamma_2}; f_2^{\Gamma_2}; \ldots; f_N^{\Gamma_2}] \in \mathcal{R}^{N \times C}$  and their corresponding bounding boxes  $\mathbf{B}^{\Gamma_1} \in \mathcal{R}^{N \times 7}$ ,  $\mathbf{B}^{\Gamma_2} \in \mathcal{R}^{N \times 7}$ , where  $N$  is the number of RoI features and  $C$  denotes the dimension of features, we first select  $M$  features  $\tilde{\mathbf{F}}^{\Gamma_1} \in \mathcal{R}^{M \times C}$ ,  $\tilde{\mathbf{F}}^{\Gamma_2} \in \mathcal{R}^{M \times C}$  and its corresponding bounding boxes  $\tilde{\mathbf{B}}^{\Gamma_1} \in \mathcal{R}^{N \times 7}$ ,  $\tilde{\mathbf{B}}^{\Gamma_2} \in \mathcal{R}^{N \times 7}$  with the highest scores. Then, to obtain the positional relationship corresponding to the activation regions of the two branches, we calculate the distance of the box center between  $\tilde{\mathbf{B}}^{\Gamma_1}$  and  $\tilde{\mathbf{B}}^{\Gamma_2}$ , and the feature correspondence can be obtained by the following equation:
$$
\left(\hat {\mathbf {F}} ^ {\Gamma_ {1}}, \hat {\mathbf {F}} ^ {\Gamma_ {2}}\right) = \left\{\left(\tilde {f} _ {i} ^ {\Gamma_ {1}}, \tilde {f} _ {j} ^ {\Gamma_ {2}}\right) \right| \sqrt {\left(c _ {i , x} ^ {\Gamma_ {1}} - c _ {j , x} ^ {\Gamma_ {2}}\right) ^ {2} + \left(c _ {i , y} ^ {\Gamma_ {1}} - c _ {j , y} ^ {\Gamma_ {2}}\right) ^ {2} + \left(c _ {i , z} ^ {\Gamma_ {1}} - c _ {j , z} ^ {\Gamma_ {2}}\right) ^ {2}} <   \tau \}, \tag {3}
$$
where  $(c_{i,x}^{\Gamma_1}, c_{i,y}^{\Gamma_1}, c_{i,z}^{\Gamma_1})$  and  $(c_{j,x}^{\Gamma_2}, c_{j,y}^{\Gamma_2}, c_{j,z}^{\Gamma_2})$  denote  $i$ -th and  $j$ -th box center of  $\tilde{\mathbf{B}}^{\Gamma_1}$  and  $\tilde{\mathbf{B}}^{\Gamma_2}$ ,  $\tau$  is a threshold. Once the correspondence features from different input views are obtained, these unknown instances will be updated as the foreground instances that can be fed into their original class head.
Consistency Loss. After obtaining the corresponding activation features of different branches, a consistency loss is utilized to ensure the consistency of the corresponding features as follows: