AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset
Authors: Jiakang Yuan
Bo Zhang
Xiangchao Yan
Botian Shi
Tao Chen
Yikang LI
Yu Qiao
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D Object Detection
3D Pre-training
Autonomous Driving
TL;DR: We are committed to building a large-scale pre-training dataset with diverse point-cloud data distribution and learning generalizable representations from such a diverse pre-training dataset.

Abstract: It is a long-term vision for Autonomous Driving (AD) community that the perception models can learn from a large-scale point cloud dataset, to obtain unified representations that can achieve promising results on different tasks or benchmarks. Previous works mainly focus on the self-supervised pre-training pipeline, meaning that they perform the pre-training and fine-tuning on the same benchmark, which is difficult to attain the performance scalability and cross-dataset application for the pre-training checkpoint.  In this paper, for the first time, we are committed to building a large-scale pre-training point-cloud dataset with diverse data distribution, and meanwhile learning generalizable representations from such a diverse pre-training dataset. We formulate the point-cloud pre-training task as a semi-supervised problem, which leverages the few-shot labeled and massive unlabeled point-cloud data to generate the unified backbone representations that can be directly applied to many baseline models and benchmarks, decoupling the AD-related pre-training process and downstream fine-tuning task. During the period of backbone pre-training, by enhancing the scene- and instance-level distribution diversity and exploiting the backbone's ability to learn from unknown instances, we achieve significant performance gains on a series of downstream perception benchmarks including Waymo, nuScenes, and KITTI, under different baseline models like PV-RCNN++, SECOND, CenterPoint.

AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset
Jiakang Yuan
Bo Zhang Xiangchao Yan², Tao Chen 1i, Botian Shi?, Yikang Li,, Yu Qiao
School of Information Science and Technology, Fudan University Shanghai Artificial Intelligence Laboratory jkyuan220m www.n.uuaaeoo. .du. cn, zhangbo@pjlab www.laa.................... cn, eetchenCfudan. edu. cn
Abstract
It is a long-term vision for Autonomous Driving (AD) community that the per- ception models can learn from a large-scale point cloud dataset, obtain unified representations that can achieve promising results on different tasks or benchmarks. Previous works mainly focus the self-supervised pre-training pipeline, mean- ing that they perform the pre-training and fine-tuning on the same benchmark, which difficult attain the performance scalability and cross-dataset application for the pre-training checkpoint. In this paper, for the first time, we are commit- ted building large-scale pre-training point-cloud dataset with diverse data distribution, and meanwhile learning generalizable representations from such a diverse pre-training dataset. We formulate the point-cloud pre-training task as a semi- cuni-sueeevieed problem, which leverages the few-shot labeled and massive unlabeled point-cloud data to generate the unified backbone representations that can be directly applied to many baseline models and benchmarks, decoupling the AD-related pre-training process and downstream fine-tuning task. During the period of backbone pre-training, by enhancing the scene- and instance-level distribution diversity and exploiting the backbone's s ability to learn from unknown instances, we achieve significant performance gains series of downstream perception benchmarks including Waymo, nuScenes, and KITTI, under differ- ent baseline models like PV-RCNN++, SECOND, CenterPoint. Project page: //jak/////a github. 1ow..-o2aDDDD PT. github.io/.
Introduction
LiDAR sensor plays erucial role in the Autonomous Driving (AD) system due to high quality for modeling the depth and geometric information of the surroundings. As result, lot of studies aim object detection baseline models, such as CenterPoint [32], PV-RCNN [17], and PV-RCNN+ [19].
to achieve the scene perception via LiDAR-base
Although the 3D object detection models can help autonomous driving recognize the surrounding environment, the existing baselines are hard to generalize to a new domain (such as different sensor settings or unseen cities). A long-term vision of the autonomous driving community is to develop generaeenell pre-trained model, which can be widely applied to different downstream tasks. To achieve this goal, researchers begin to leverage the Self-Supervised Pre-Traine rre-Training (SS- PT) paradigm recently. For example, ProposalContrast [31] proposes contrastive learning based approach enhance region-level feature extraction capability. Voxel-MAE [6] designs a masking and reconstructing task to obtain a stronger backbone network.
This work was done when Jiakang Yuan was an intern at Shanghai Artificial Intelligence Laboratory. Corresponding Tao Chen and Bo Zhang.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Single delaset
Fine- tuning on the same dataset
Build a unified dataset
Fine buning on multiple dalaeet
Dolast
Dalagt
Datask
Dulaset
Daluze
Daloset
(a) Previous pre-training paradigm.
(b) AD-PT pre-training paradigm.
Figure 1: Differences between previous pre-training paradigm and the proposed AD-PT paradigm.
However, as illustrated in 1ig. it should be pointed out that there is a crucial difference between the above-mentioned SS-PT and the desired Autonomous Driving Pre-Training (AD-PT) paradigm. The SS-PT aims to learn from a single set of unlabeled data to generate suitable representations for the same dataset, while AD-PT is expected to learn unified representations from as large and diversified data as possible, so that the learned features can be easily transferred to various downstream tasks. As result, SS-PT may only perform well when the test and pre-training data are sampled from the same single dataset (such as Waymo [20] or nuScenes [I]), while AD-PT presents better generalized performance on different datasets, which can be continuously improved with the increase of the number of the pre-training dataset.
Therefore, this paper is focused achieving the AD-related pre-training which can be easily applied to different baseline models and benchmarks, By conducting extensive experiments, we argue that there are two key issues that need be solved for achieving the real AD-PT: 1) how build a unified AD dataset with diverse data distribution, and 2) how to learn generalizable representations from such a diverse dataset by designing an effective pre- pre-training method.
For the first item, we use a large-scale point cloud dataset named ONCE [14], consisting of few-shot labeled (e.g., ~0.5%) and massive unlabeled data. First, to get accurate pseudo labels of the massive unlabeled data that can facilitate the subsequent pre-training task, we design a class-wise pseudo labeling strategy that uses multiple models to annotate different semantic classes, and then adopt semi-supervised methods (e.g., MeanTTechhrr [21]) to further improve the accuracy on the ONCE validation set. Second, to get unified dataset with diverse raw data distribution from both LiDAR beam and object sizes, inspired by previous works [29, 24, 33,34) 34], we exploit point-to-beam playback re-sampling and object re-scaling strategies diversify both scene- and region-level distribution.
For the second item, we find that the taxonomy differences between the pre- pre-training ONCE dataset and different downstream datasets are quite large, resulting in that many hard samples with taxonomic inconsistency are difficult to be accurately detected during the fine-tuning stage. As a result, taxonomy differences between different benchmarks should be considered when performing the backbone pre- training. Besides, our study also indicates that during the pre-training process, the backbone model tends fit with the semantic distribution of the ONCE dataset, impairing the perception ability on downstream datasets having different semantics. To address this issue, we propose an unknown-aware instance learning ensure that some background regions on the pre-training dataset, which may be important for downstream datasets, can be appropriately activated by the designed pre-training task. Besides, to further mine representative instances during the pre-training, we design consistency loss to constrain the pre-training representations from different augmented views to be consistent.
Our contributions can be summarized as follows:
1. For the first time, we propose the AD-PT paradigm, which aims to learn unified representa- tions by pre-training a general backbone and transfers knowledge to various benchmarks.
2. To enable the AD-PT paradigm, we propose a diiersity-basse pre-training data preparation procedure and unknown-aware instance learning, which can be employed in the backbone pre-training process to strengthen the representative capability of extracted features.
3. Our study provides more unified approach, meaning that once the pre-trained checkpoint is generated, it can be directly loaded into multiple perception baselines and benchmarks. Results further verify that such an AD-PT paradigm achieves large accuracy gains different benchmarks (e.g., 3.41%, 8.45%, 4.25% on Waymo, nuScenes, and KITTI).

---

Build unified largesscale point cloud dataset
Filler
Pre-training
parado
shared
shared
Cross- view matchinng
I Large-scole datinee with diverdtty
View
Locsist
F
pre-training
Deuse
Filter
Fine mine-tuning
Multiple other datasets
Different detecters
3D backbone
200 baokbooe
Selected features
CrnterPoint
veiate
Consistent ROI features
FV-RCNN++
Vizw flww
Waymo
Figure 2: The overview of the proposed AD-PT. By leveraging the proposed method to train on the unified large-scale point cloud dataset, we can obtain well-generalized pre-training parameters that can be applied to multiple datasets and support different baseline detectors.
2 Related Works
2.1 LiDAR-based 3D Object Detection
Current prevailing LiDAR-based 3D object detection works [11, 16, 17, 19, 27, 18, can be roughly divided into point-based methods, voxel-based methods, and point-voxel-based methods. Point- based methods [16, 30] extract features and generate proposals directly from raw point clouds. PointRCNN [16] is a prior work that generates 3D Rols with foreground segmentation and designs an RCNN-style two-stage refinement. Unlike point-based methods, voxel-based methods [27, 32] first transform unordered points into regular grids and then extract 3D features using 3D convolution. As pioneer, SECOND [27] utilizes sparse convolution 3D backbone and greatly improves the detection efficiency. CenterPoint [32] takes care of both accuracy and efficiency and proposes a one-stage method. To take advantage of both point-based and voxel- based methods, PV-RCNN [17] and PV-RCNN+ [19] propose a poxnt-voxel set abstraction to fuse point and voxel features.
2.2 Autonomous Driving-related Self-Supervieed Pre- Pre-Training
Inspired by the success of pre-training in 2D images, self-supervised learning methods have been
extended to LiDAR-based AD scenarios [6. 13, 12, 28, 26, 10, 31]. Previous methods mainly focus on using contrastive learning or masked autoencoder (MAE) to enhance feature extraction. Contrastive learning-based methods [12, 31, $1,77 7] use point clouds from different views or temporally- correlated frames input, and further construct positive and negative samples. STRL [7] proposes a
representation learning that employs contrastive learning with two temporally- correlated frames. GCC-3D [12] and ProposalContras. constructs consistency map to find the correspondence between different views, developing contrastive learning strategies at region-level. CO3 [2] utilizes LiDAR point clouds from the vehicle- and infrastructure-side to build different views. MAE-based [6, 10] methods utilize different mask strategies and try to reconstruct masked points by designed decoder. Voxel-MAE [6] introduces voxel-level mask strategy and verifies the effectiveness of MAE, BEV-MAE [13] designs a BEV-guided masking strategy. More recently, GD-MAE [28] and MV-JAR [26] explore masking strategies in transformer architecture. Different from the existing works that use designed self-supervised approach to pre-train on unlabeled data and then fine-tune labeled data with the same dataset, AD-PT aims to pre-train large-scale point cloud dataset and fine-tune on multiple different datasets (i.e., Waymo [20], nuScenes[]] []]
KITTI [5]).
3

---

Table I: Performance using different detectors on ONCE validation set. We report mAP using ONCE evaluation metric.
Table 2: Statistics on the number of pseudo- labeled instances per frame. We compare with ONCE labeled data set.
Detector
Choice
Vahicle
Podestrian
Cycllst
ONCE labeled set
Pscudo label set
Benchmark (Best)
66.79
Vchicle
Ped.
Cyclist
Vehicle
Ped.
Cyelist
CenterPoint (ouns)
Conter
PV-RCNN=4 (ours)
Anchor Head
82.50
71.19
19.01
4.52
5.63
15.67
1.63
1.90
awcre Pacudo Labds Generaoor
65
28
Labeded sot
Model
1
Poeaady
Class 1
Pseudo Labsled
2.6
55-
24-
Model
Panado
Class K
4.5
Diversty- besed Pre tramiig Processor
14
3,5
16
Some--eeul ban-resirplay
25
122
Large-scale datase with diversity
Length
Width
Height
KITT
naSocno
Wayner
ONCE
OXCEmoocaing
Figure 3: Overall dataset preparation procedure.
Figure 4: Statistics of object re-scaling.
3 Method
To better illustrate our AD-PT framework, we first briefly describe the problem definition and overview of the proposed method in Sec. 3.1. Then, we detail the data preparation and model design of AD-PT in Sec. 3.2 and Sec. 3.3, respectively.
3.1 Preliminary
Problem Formulation. Different from previous self-supervised pre-training methods, AD-PT performs large-scale point cloud pre-training in semi-supervised manner. Specifically, we have access to N samples which are composed of a labeled set Dr = {(0,, y1) y:)) No, and a unlabeled set Du { No where Nr and Ny denote the number of labeled and unlabeled samples. Note that Nr can be much smaller than No (e.g., <55 v.s. ~M, about 0.5% labeled samples). The purpose of our work is to pre-train that the pre-traine treitra backbone parameters can be used for many down-stream benchmark datasets or 3D detection models.
large-scale point cloud dataset with few-shot Dr and massive Du such
Overview AD-PT. As shown in Fig. 2, AD-PT mainly consists of large-scale point cloud dataset preparation procedure and unified AD-focused representation learning procedure. To initiate the pre-training, a class-aware pseudo labels generator is first developed to generate the pseudo labels of Du. Then, to get more diverse samples, we propose diversity-basee pre-training processor. Finally, in order to pre-train on these pscudo-labeled data learn their generalizable representations for AD purposes, unknown-aware instance learning coupled with consistency loss is designed.
3.2 Large-scale Point Cloud Dataset Preparation
In this section, we detail the preparation of the unified large-scale dataset for AD. As shown Fig. 3, data creation consists of a class-aware pseudo label generator and diversity- bassed pre-training processor to be introduced below.
3.2.1 Class-aware Pseudo Labels Generator
It can be observed from Tab. 7 that, pseudo-lahels with high accuracy the pre-training dataset are beneficial to enhance the detection accuracy on downstream datasets such as Waymo [20] and nuScenes [1]. Therefore, get more accurate pseudo labels, we design the following procedure.
Class-aware Pseudo Labeling. ONCE benchmark is utilized to evaluate the pseudo labeling accuracy and more results are shown in the supplementary material. We find that different baseline
https://nce-forr for-nnce---o---ueeee wwwtwootiooooeeeeeeee httpr/iigggiiuubbb///enmnnciiii html

---

(a) Original point clouds
(b) Upsampled point clouds
(c) Downsampled point clouds
Figure 5: Visualization of point-to-beam playback re-sampling.
models have different biased perception abilities for different classes. For example, the center-based
method (i.e., CenterPoint (32]) tends to detect better for small-scale targets (e-g., Pedestrian), while anchor-based methods perform better for other classes. According to the observation, we utilize the PV-RCNN+ [19] with anchor-head
annotate Vehicle and Cyclist classes on ONCE, where the accuracy on Vehicle and Cyclist is found to be much better than the methods listed in the ONCE benchmark. On the other hand, CenterPoint [32] is employed to label the Pedestrian class on ONCE, Finally, we use PV-RCNN+ and CenterPoint to perform a class-wise pseudo labeling process.
Semi-supervised Data Labeling. We further employ the semi-supervised learning method to fully exploit the unlabeled data to boost the accuracy of the pseudo labels. As shown in ONCE benchmark, Mean'Teache [21] can better improve the performance ONCE, and therefore, the MeanTeacher to further enhance the class-wind wise detection ability. Tab. shows that the accuracy ONCE validation set can be improved after leveraging massive unlabeled data, significantly surpassing all previous records.
Pseudo Labeling Threshold. To avoid labeling a large number of false positive instances, we set relatively high threshold. In detail, for Vehicle, Pedestrian, and Cyclist, we filter out bounding boxes with confidence scores below 0.8, 0.7, and 0.7. As result, it can be seen from Tab. 2 that, compared with ONCE labeled data, some hard samples with relatively low prediction scores are not annotated.
3.2.2 Diversity-base Pre-training Processor
As mentioned in [15, 9], the diversity of data is crucial for pre-training, since highly diverse data can greatly improve the generalization ability of the model. The same observation also holds in 3D pre-training. However, the existing datasets are mostly collected by the same LiDAR sensor within limited geographical regions, which impairs the data diversity. Inspired by [33, 34], discrepancies between different datasets can be categorized into scene-level (e.g.. LiDAR beam) and instance-level (e.g., object size). Thus, we try increase the diversity from the LiDAR beam and object size, and propose a point-to-beam playback re-sampling and an object re-scaling strategy.
Data with More Beam-Diversity: Point-to-Beam Playback Re-sampling. To get beam-divers data, we use the range image as an intermediate variable for point data up-sampling and down- sampling. Specifically, given a LiDAR point cloud with n beam (e.g., 40 beam for ONCE dataset) and m points per ring, the range image paxm can be obtained by the following equation:
2 22 +2, 0 arrcta(x/v))
arcsin(z/r),
(1)
where á and / are the inclination and azimuth of point clouds, respectively, and denotes the range of the point cloud. Fach column and row of the range image corresponds to the same azimuth and inclination of the point clouds, respectively. Then, we can interpolate sample over rows of range image, which can also be seen as LiDAR beam ra-sanp. Finally, we reconvert the range image to point clouds follows:
rcoss())oss(() y rcos(()sin(())
rsin(())
(2)
where x. y. z denote the Cartesian coordinates of points, and Fig. 5 shows that point-to-beant playback re-sampinng can generate scenes with different point densities, improving the scene-level diversity.
Data with More Rol-Diversity: Object Re-scaling. According to the statement in 29,,29,22 different 3D datasets were collected in different locations, the object size has inconsistent distributions. As shown in Fig. 4, single dataset such as ONCE cannot cover variety of object-size distributions,
29, 23] that
5

---

resulting in that the model cannot learn a unified representation. To overeome such problem, we propose an object re-scaling mechanism that can randomly re-scale the length, width and height of each object. In detail, given a bounding box and points within it, we first transform the points to the local coordinate and then multiply the point's coordinates and the bounding box size by a provided scaling factor. Finally, we transform the scaled points with the bounding box to the ego-car coordinate. can be seen from Fig. 4 that after object re-scaling, the produced dataset contains object sizes with more diversified distributions, further strengthening the instance-level point diversity.
3.3 Learning Unified Representations under Large-scale Point Cloud Dataset
By obtaining a unified pre- training dataset using the above-mentioned method, the scene-level and instance-level diversity can be improved. However, unlike 2D or laiiuag pre-training datasets which cover lot of categories to be identified for downstream tasks, our pseudo dataset has limited category labels (i.e., Vehicle, Pedestrian and Cyclist). Besides, as mentioned in Sec. 3.2.1, in order to get accurate pseudo annotations, we set a high confidence threshold, which may inevitably ignore some hard instances. As a result, these ignored instances, which are not concerned in the pre-training dataset but may be seen categories of interest in downstream datasets (e.g., Barrier in the nuScenes dataset), will be suppressed during the pre-training process.
To mitigate such problem, it is necessary that both pre- tre-training treatninnn-rrlttedd related instances and some unknown instances with low scores can be considered when performing the backbone pre-training. From a new perspective, we regard the pre-training as an open-set learning problem. Different from traditional open-set set detection [8] which aims at detecting unknown instances, our goal is activate as many foreground regions as possible during the pre- veetrraiing stage. Thus, we propose two-branch unknown-awar instance learning head to avoid regarding potential foreground instances as the background parts. Further, consistency loss is utilized to ensure the consistency of the calculated corresponding foreground regions.
Overall Model Structure. In this part, we briefly introduce our pre-training model structure. Following the prevailing 3D detectors [17, 27], as shown in Fig. 2, the designed pre-trane pre-training model consists of a voxel feature extractor, a 3D backbone with sparse convolution, a 2D backbone and the proposed head. Specifically, given point clouds P R Rx++dd) we first transform points into different views through different data augmentation methods I and F2. Then, voxel features are extracted by a 3D backbone and mapped to BEV space. After that, dense features generated by a 2D backbonn can be obtained, and finally, the dense features are fed into the proposed head.
Unknown-aware Instance Learning Head. Inspired by previous open-set detection works [8, 35], we consider background region proposals with relatively high objectness scores to be unknown instances that are ignored during the pre-training stage but may be crucial to downstream tasks, where the objectness scores are obtained from the Region Proposal Network (RPN). However, due to that these unknown instances contain lot of background regions, directly treating these instances as the foreground instances during the pre-training will cause the backbone network to activate large number of background regions. To overcome such a problem. we utilize two-branch head as a committee to discover which regions can be effectively presented as foreground instances. Specifically, e
given the RoI features F = [ff f :...; [') E R Rxe. F²2 = [ff²; 12 ::... JKJ] E R-NxC and their corresponding bounding boxes B" R B R Nx7, where N is the number of Rol features and C denotes the dimension of features, first select M features T" MxC FT MMC and its corresponding bounding boxes B" Nx7 B r² R with the highest scores. Then, obtain the positional relationship corresponding to the activation regions of the two branches, we calculate the distance of the box center between B" and T and the feature correspondence can be obtained by the following equation:
JT - G2y)a + (Ch.) - -,,)2= + (ch: - -5,))² < T}
(3)
where (c... c.)) and cf denote i-th and j-th box center of B and B is threshold. Once the correspondence features from different input views are obtained, these unknown instances will be updated as the foreground instances that can be fed into their original class head.
Consistency Loss After obtaining the corresponding activation features of different branches, a consistency loss is utilized to ensure the consistency of the corresponding features as follows:

---

Table 3: Fine-tuning performance Waymo benchmark (LEVEL_2 metric). Note that we only use single checkpoint parameter to initialize all downstream baselines including SECOND, CenterPoint, PV-RCNN++. Semi denotes the semi-supervisee method training on unlabeled ONCE split.
Method
Paradigm
Data amount
L.2 AP/ APH
Overall
Vehicle
Pedestrian
Cyclist
From scratch (SRCOND)
3%
52.00/ /3.00/37.70 37.70
58.11 / 57.44
51.34/27.88 27.38
46.57 / 22828
From scratch (SECOND)
20%
60.62 / 56.86
64.26 63.73
59.72/ / 50.38
57.87 56
ProposalContrast (SECOND) [3]]
SS-PT
20%
60.91 / 57.16
64,50/ 63.90
60.33 /5.00
57.90/56
BEV-MAE (SECOND) [13]
SS-PT
20%
61.03 / 57.30
64.42/ 63.87
59.97 / 50.65
58.69 /57.39
MeanTeacher (SECOND) [2]]
Semi
20%
60.93 / 55.311
64.22 63.73
59.54 50.80
58.66 55.44
Ours (SECOND)
AD-PT
3%
55.41 51.7
60.53/ 59.93
54.91 /45.77
50.79 49..65
Ouns (SECOND)
AD-PT
20%
61.26 $7.69
64.54/ / 64.00
60.25 / 51.21
59.00 / 57.86
From scratch (CenterPoint)
3%
$9.00 56.29
$7.12/ 56.57
$8.66 52.44
61.24 / 59.89
From scrateh (CenterPoint)
20%
66.47 64.01
64,91 / 64.42
66.03 / 60.34
68.49 66.
GCC-3D (CenterPoint) (12]
SS-PT
20%
65.29 / 62.79
63.97 / 63.47
64.23 / 68.47
67.68 / 66.44
ProposalContrast (CenterPoint) [31]
SS-PT
20%
66.67 / 64.20
65.22 / 64.80
66.40 60.49
68.48 / 6
BEV-MAE (CenterPoint) [13]
SS-PT
20%
66.92/64.45
64.78/ / 64.29
66.25 / 60.53
69.73 68.52
MeanTeacher (C (CenterPoint) [21]
Semi
20%
66.66 / 64.23
64.94/ 64.43
66.35 60.61
68.69 / 67.65
Ours (CenterPoint)
AD-PT
3%
61.21 58.46
60.35 59,79
60.57 / 54.02
62.73 61.57
Ours (CenterPoint)
AD-PT
20%
67.17 / 64.65
65.33 / 64.83
67.16 / 61.20
69.39 / 68.25
From scratch (PV-RCNN++)
63.81 / 61.10
64.42/ 63.93
64.33 / 57.79
62.69 66.59
From scrutch (PV-RONN++)
20%
69.97 / 67.58
69.18/ 68.75
70.88 / 65.21
69.84 68.77
ProposalConirast (PV-RCNN+ [31]
SS-PT
20%
70.30/ / 67.78
69 69.45/ 69:000
71.42 65.68
70.04 / 64.05
BEV-MAE (PV- RCNN++ [13]
SS-PT
20%
70.54 / 68.11
69.53 69.07
71.50/ 65.69
70,60 / 69.56
MeanTeacher (PV-RCN+++) [2]]
Semi
20%
70.62 / 68.14
69. 68.81
71.96 66.42
70.17 69.21
Ours (PV-RCNN++)
AD-PT
3%
68. 58.33/ / 65 69
68.17 67.70
68. 82 62. 29
68.00 / /67.00
Ours RCNN+)+++)
AD-PT
20%
71.55 / 69.23
70.62/ / 70.19
72.36 / 66.82
71.69 / 70.70
K
Lconsist
ΣΣ
- -f
BK
where B is the batch size and K is the number of the corresponding activation features. Overall Function. The overall loss function can be formulated as:
Lintal Lets + Lreg Leonsist::
(5)
where Lots and Greg represent classification loss and regression loss of the dense head, respectively, and Lconsist is the activation consistent loss as shown in Eq. 4.
4 Experiments
4.1 Experimental Setup
Pre-training Dataset. ONCE [14] is large-scale dataset collected in many scenes and weather conditions. ONCE contains 581 sequences composed of 20 labeled (~~19 frames) and 561 unlabeled s
sequences(~1M frames). The labeled set divides into train set with 6 sequences (~5K samples),
validation set with 4 sequences (~33k frames), and a test set with 10 sequences (~8k frames). We merge Car, Bus, Truck into a unified category (i.e., Vehicle) when performing the pre- pre-training. Our main results are based on ONCE small split and use a larger split to verify the pre-training scalability.
Description of Downstream Datasets. 1) Waymo Open Dataset [20] contains ~550k frames. 2) nuScenes Dataset [l] provides point cloud data from a 32-beam LiDAR consisting of 28130 training samples and 6019 validation samples. 3) KITTI Dataset [5] includes 7481 training samples and is divided into a train set with 3712 samples and a validation set with 3769 samples.
Description of Selected Baselines. In this paper, we compare our method with several baseline methods including both Self- Supervieed Pre- Trainiinn (SS-PT) and supervise learning methods. SS-PT methods: as mentioned in Sec. 2.2, We mainly compare with contrastive learning based (i.e., GCC-3D [12], ProposalContrast [31]) and MAE-based SS-PT methods (i.e., Voxel-MAE [6], BEV- MAE [13]). All these methods are pre-trained and fine-tuned on the same dataset. Semi-supervised learning methods: to verify the effectiveness of our method under the same experimental setting, we also compare with commonly-use semi-supervised technique (i.e., MeanTeacher [21]).
7

---

Table 4: Fine-tuning performance on nuScenes benchmark. C.P. denotes that CenterPoint is employed as the baseline detector and D.A. represents the Data Amount. We fine-tune on 5% and 100% data for 20 epochs. Compared with other works [12, 13], our pre-training process is performed on unified dataset rather than nuScenes.
Method
Setting
D.A.
mAP
NDS
Car
Truck
CV.
Bus
Trailer
Barrier
Motor.
Bicycle
Pod.
TC.
From scratch (SECOND)
5%
29.24
39.74
67.69
33.02
7.15
45.91
17.67
25.23
11.92
0.00
53.00
30.74
From scratch (SECOND)
100%
50.59
Ours (SECOND)
AD-PT AD-PT
5%
37.69
47.95
74.89
41
12.05
54,77
28.91
34.4]
23.63
3.19
63.61
39.54
Ours (SECOND)
100%
52.23
63.04
83.12
52.86
15.24
68.58
37.54
59.48
46.01
20.44
78.96
60.05
From scratch (C.P)
5%
42.68
43.61
10.65
44.01
18.71
52.95
36.26
16.76
67.62
54.52
From scratch (C.P)
100%
56.2
64.5
84.8
53.9
16.8
67.0
35.9
64.8
55.8
36.4
83,1
63.4
GCC-3D (C.P. [12]
SS-PT
100%
57.3
65.0
85.0
54.7
17.6
67.2
35.7
65.0
56.2
36.0
82.9
63.7
BEV-MAF (C.P)
SS.PT
100%
57.2
65.1
84.9
54.9
16.5
67.2
35.9
65.2
56.0
36.2
81,2
63.5
Ours (C.P.)
AD-PT
5%
44.99
52.99
78.90
43.82
11.13
55.16
21.22
55.10
39.03
17.76
72.28
55.43
Ours (C.P.)
AD-PT
100%
57.17
65.48
84.86
54.37
16:09
67.34
36.06
64.31
58.50
40.58
83,53
66.05
Table 5: Fine-tuning performance (AP3)) on KITTI benchmark.
Method
Setting
Data
mAP
Car
Pedestian
Cyclist
(Mod.)
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
From seratch (SECOND)
20%
61.70
89.78
78.83
76.21
52.08
47.23
43.37
76.35
$9.06
55. 55.24
From scratch (SECOND)
100%
66,70
89.63
80.78
78.21
58.05
52.61
48.24
84.25
66.71
62.50
(SECOND)
65.55 $5.95
90.23
55.63
83.78
67.50
Ours (SECOND)
AD-PT
100%
67.58
90.36
81.39
78.41
58.30
$3.58
48.72
$6,04
67.78
63.95
From scratich (PV-RCNN)
20%
66.71
91.81
K2,52
80.11
58.78
$3.33
47,61
86.74
64.28
59.53
ProposalC (PV-RCNN) [31]
8S-PT
68
55.05
66.88
62.
From sermecl (PV.RCNN)
70:57
57.06
70.14
GCC-3D (PV-RCNK) [12]
SS-PT
100%
71.26
STRI (PV-RUNN)
SS-PT
10055
71.
84.70
$7.80
71,88
PointConirast (PVRRCNN)
SS-PT
71.55
82:25
65.73
57.74
$2.46
91.47
72.72
67
ProposalContrast (PV- RCNN) [3]]
SS-PT
100%
72.92
92.45
84.72
82.47
68:43
60.36
55.01
92.77
73.69
69.51
Ours (PVRCNNN)
AD-PT
209
69.
92.18
K2.75
82.12
65.30
57.59
51.84
84
67.96
64
Ours (PV:RCNN)
AD-PT
10055
73.01
91.96
84.75
82.53
68.87
60.79
55.42
91.81
73.49
09.21
Implementation Details. We fine-tune the model on several different detectors including SEC- OND [27], CenterPoint [32] and PV-RCNN+ [19]. Note that the transformer- based detectors are not used, since we consider directly applying the pre-trained backbone parameters to the most commonly used downstream baselines. For different views generation, we consider 3 types of data augmentation methods, including random rotation ([-180", 180"), random scaling ([0.7, 1.2]), and random flipping along X-axis and Y-axis. We use Adam optimizer with one-cycle learning rate schedule and the maximum learning rate is set to 0.003. We pre-train for 30 epochs on ONCE small split and large split using NVIDIA Tesla A 100 GPUs. The number of selected Rol features is set to 256 and the view matching threshold is set to 0.3. Our code is based on 3DTrans [22].
Evaluation Metric. We use dataset-specific evaluation metries to evaluate fine-tuning performance on each downstream dataset. For Waymo, average precision (AP) and average precision with heading (APH) are utilized for three classes (i.e., Vehicle, Pedestrian, Cyclist). Following [13, 31], we mainly focus on the more difficult LEVEL_2 metric. For nuScenes, we report mean average precision (mAP) and NuScenes Detection Score (NDS). For KITTI, we use mean average precision (mAP) with 40 recall to evaluate the detection performance and report AP3D results.
Table 6: Ablation study on data preparation.
Method
Enhancement
Waymo AP/APH
nuScenes
Overall
Vehiele
Pedestrian
Cyclist
mAP
NDS
Baseline Baseline-re-scalinn
None
67.12 64.55
67.45 66.97
66.19/65225 67.74761.15
65.24
36.26
45.04
Object-size
67.39/ 64.68
67.52 6743
62
66.83/ 06.33/5570 65.79
39.72
49,93
Baseline re sampling
LiDAR-heam
67.37/64.70 64.70
67.70/ 67.21
68.21 /6.77
66.15/65.11
41.35
51.03
Baselinctre-sealingg samplinng
Both
67.77/ 65.09
68.01 / 67.61
68.32 /6.62/61.99
66.99/ 65.98
43.11
52.41
4.2 Main Results
Results on Waymo. Results on Waymo validation set are shown in Tab. 3. We first compare the proposed method with previous SS-PT methods. It can be seen that all three detectors achieve the best results using AD-PT initialization, surpassing previous SS-PT methods even using a smaller pre-training dataset (i.e., 100k frames on ONCE small split). For example, the improvement achieved by PV-RCNN+ is 1.58% 1.65% in terms of L2 AP/ APH. Note that the compared SS-PT
8

---

Table 7: The impact of pseudo-labeling methods on downstream datasets.
ONCH
Waymo 1.2 APAPH
nuScenes
Pscudo-labeling Method
Overall
Overall
Vehicle
Pedestrian
Cyclist
mAP
NDS
SECOND (Low Performance)
57.10
65.96/ 63.29
65.95 65.46
66.87 60.36
65.07 64.06
41,49
50.82
CenterPoint (Middle Performance)
60,84
66.79/ 64.10
67.09 66.60
65.51 64455
41.91
51.64
Ours (High Performance)
69.90
67.77 / 65.09
68.01 67.61
68.32 61.69
66.99 / 65.98
43.11
52.41
Table 8: The performance scalability using KITTI and ONCE.
Waymo L2 AP/APH
Pre-training dataset
Overall
Vehicle
Pedestrian
Cyclist
APIAD-F1
Afti disc ansak (ADa
KITTI (-44k
16..16
64.73/
64.43/57.30
62.60
ONCE (~4k)
64.28 44.28/61.36
66.11 65.64
66.26 59.51
65.39 64.35
ONCE (~10k)
66.94/644 /4,224
67.41 66.91
67.97/61.39
65.45 64.43
Number buning
ONCE 100k)
68.33 65.69
68.17 67.70
68.82/62.39
68.00 67.00
ONCE (0000))
69,04/ 66.52
68.69 / 68.23
69.81 63.74
68.61 67.60
Figure 6: Different budgets
methods are pre-trained on Waymo 100% unlabeled train set (~150k frames), which has smaller domain gap with fine-tuning data. Further, to verify the effectiveness of fine-tuning with small number of samples, we conduct experiments of fine-tuning on 3% Waymo train set (~5K frames). We can observe that, with the help of pre-trate pre-trained prior knowledge, fine-tuning with small amount of data can achieve much better performance than training from scratch (e.g., 3.41% / 14.08% 1.2 using SECOND as baseline). In addition, to ensure the fairness of the experiments, we compare our method with semi-supervised learning methods which are identical to our pre-training setup.
Results on nuScenes. Due to the huge domain discrepancies, few works can transfer the knowledge obtained by pre-training on other datasets nuScenes dataset. Thanks constructing unified large-scale dataset and considering taxonomic differences in the pre-training stage, our methods can also significantly improve the performance on nuScenes. As shown in Tab. 4, AD-PT improves the performance of training from scratch by 0.93% and 0.98% in mAP and NDS.
relatively small dataset (i.e., KITTI). Note that previous methods often use models pre-trained on Waymo as an initialization since the domain gap is quite small. It can be observed that using AD>PPT initialization can further improve the accuracy when fine-tuning on both 20% and 100% KITTI training data under different detectors. For instance, the AP3 in moderate level can improve 2.72% and 1.75% when fine-tuning on 20% and 100% KITTI data using PV-RCNN+ as the baseline detector.
Results KITTL As shown in Tab. 5, we further fine-tune
4.3 Insight Analyses
In this part, we further discuss the 3D pre-training, Note that in ablation studies, we fine-tune on 3% Waymo data and 5% nuScenes data under the PV-RCNN+ and CenterPoint baseline setting.
4.3.1 Insight Analyses on the Data Preparation
Discussion on Diversity-based Pre-training Processor. From Tab. 6, we observe that both beam re-sampling and object re-scaling during the pre-training stage can improve the performance for other downstream datasets. For example, the overall performance can be improved by 0.65% / 0.54% on Waymo and 7.37% on nuScenes. Note that when pre-training without our data processor, the fine-tuning performance nuScenes will drop sharply compared with training from scratch, due to the large domain discrepancy. Since our constructed unified dataset can cover diversified data distributions, the backbone can learn more general representations.
Pseudo-labeling Performance. Tab. indicates the impact of peeud--labelng operation on the downstream perception performance. We use three types of lseudo-labeling methods to observe the low, middle, and high performance on ONCE, We find that the performance of fine-tuning positively correlated with the accuracy of pseudo-labell with relatively high accuracy can guide the backbone to activate more precise foreground features and meanwhile suppress background regions.
on ONCE. This is mainly due to that pseudo labels

---

Scalability, To verify the scaling ability of the AD-PT paradigm, we conduct experiments in Tab 8 to show the Waymo performance initialized by different pre-trained checkpoints, which are obtained using pre-training datasets with different scales. Please refer to Appendix for more results.
4.3.2 Insight Analyses on the Unified Representations Learning
Discussion on Unknown-aware Instance Learning Head. can be seen from Tab. that, Unknown- aware Instance Learning (UII.) head and Consistency Loss (CI.) can further boost the perception accuracy on multiple datasets. It can be observed that the gains are larger on the Pedestrian and Cyclist categories. The reason is that the UIL head can better capture downstream-sensitee instances, which are hard to be detected during the pseudo-labeling pre-training process.
Table 9: Ablation study on the designed UIL and CL.
Waymo 1.2 AP/APH
nuScenes
Method
Overall
Vehicle
Pedestrian
Cyclist
mAP
NDS
Baseline
67.77 / 65.09
68.01 / 67.61
68.32 61.69
66.99 / 65.98
43.11
52.41
Baseline+UUD UIL
67.97 / 65.35
67.99/ / 67.58
68.62 / 62.12
67.32 / 66.35
43.92
52.65
Baseline+UlL UIL+CCL
68.33 65.69
68.17 67.70
68.82/ / 62.39
68.00 / 67.00
44.99
52.99
Training-efficient Method. In the main results, we verify that our methods can bring performance gains under different amounts of fine-tuning data (e.g., 1%, 5%, 10% budgets). As shown in Fig. 6, results demonstrate that our method can consistently improve performance under different budgets.
4.3.3 Insight Analysis on Different Types of Baseline Detectors
To verify the generalization of our proposed method, we further conduct experiments on different types of 3D object detection backbones (i.e., pillar-based and point-based). As shown in Tab. 10 and Tab. 11, the performance of multiple types of detectors can improve when initialized by AD-PT pre-trained checkpoints which further shows that AD-PT is general pre-training pipeline that can be used on various types of 3D detectors.
Table 10: Ablation study the pillar baseed backbone. We conduct experiments Waymo dataset.
Waymo L. AP/APH
Method
Data amount
Overall
Vehicle
Pedestrian
Cyclist
From scratch (PointPillar)
20%
48.56 / 39.30
54.28 / 53.51
47.11 / 25.50
44.29/ / 38.89
AD-PT (PointPillar)
20%
52,01 43.99
58.51 57.85
50.22 32.52
47.31 41,559
From scratch (PointPillar)
100%
57.85/ / 50.69
62.18/ /1 64
58.18 40.64
53.18 49.80
AD-PT (PointPillar)
100%
59.71 / 53.49
64.10/63. 63.54
59.00 / 43.13
56.04/ / 53.80
Table 11: Ablation study on the point-based backbone. We conduct experiments on KITTI dataset.
Method
Data amount
mAP
Car
Pedestrian
Cyclist
(Mod.)
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
From sonatch (CNimRCNN)
20%
64.12
75.30
52.52
AD-PT (PointRCNN)
20%
67.67
88.75
77.20
75.18
64.58
54.16
48.24
84.25
71.86
62.50
From scratch (PoiaIRCNN)
100%
68.40
78.70
54.41
72.11
AD-PT (PainRCNN)
100%
70.47
90.90
80.25
78.05
64.80
57.13
50.37
92.45
74.04
69.45
5
Conclusion
In this work, we have proposed the AD-PT paradigm, aiming to pre-train on a unified dataset and transfer the pre-trained checkpoint to multiple downstream datasets. We comprehensively verify the generalization ability of the built unified dataset and the proposed method by testing the pre-trained model on different downstream datasets including Waymo, nuScenes, and KITTI, and different 3D) detectors including PV-RCNN, PV-RCNN++ CenterPoint, and SECOND.
Limitation
Although the AD-PT pre-trained backbone can improve the performance on multiple downstream datasets, it needs to be verified in more actual road scenarios. Meanwhile, training a backbone with more generalization capabilities through data from different sensors is also future direction.
10

---

Acknowledgement
This work is supported by National Natural Science Foundation of China (No. 62071127), National Key Research and Development Program of China (No. 2022ZD0160100) Shanghai Natural Science Foundation (No. 232R 1402900) and Shanghai Artificial Intelligence Laboratory, in part by Science and Technology Commission of Shanghai Municipality under Grant 22DZ1100102, in part by National Key R&D Program of China under Grant No. 20222D0160104.
References
[1] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11621
11631, 2020. 2, 3,
[2] Runjian Chen, Yao Mu, Runsen Xu, Wenqi Shao, Chenhan Jiang, Hang Xu, Zhenguo Li, and Ping Luo.. Co" 3: Cooperative unsupervised 3d representation learning for autonomous driving. arXiv preprint arXiv:2206.04P0 2022. 3
[3] Jiajun Deng, Shaoshuai Shi, Peiwei Li. Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cmn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1201 1209, 2021. 6
[4] Shengheng Deng, Zhihao Liang, Lin Sun, and Kui Jia. Vista: Boosting 3d object detection via dual cross-view spatial attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages #448-8457, 2022.
[5] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3354-3361, 2012. 3, 7, 17
[6] Georg Hess, Johan Jaxing. Elias Svensson, David Hagerman. Christoffer Petersson, and Lennart Svensson. Masked autoencoder for sele-super pre-training on lidar point clouds. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 350-359, 2023. 1, 3,7
[7] Siyuan Huang. Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6535 6545, 2021. 3,
[8] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5830-5840, 2021. 6
[9] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.0266 02644443,,, 2023.
[10] Georg Krispel, David Schinagl, Christian Fruhwirth- Reiiingh,, Horst Possegger, and Horst Bischof. Macli-masked autoencoder for large-scale lidar point clouds. arXiv preprint 2rXiv:2212.07207, 2022. 3
[11] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12705- 2019.
[12] Hanxue Liang, Chenhan Jiang, Dapeng Feng, Xin Chen, Hang Xu, Xiandan Liang, Wei Zhang, Zhenguo Li. and Lue Van Gool. Exploring geometry aware contrast and clustering harmonization for self-supervised 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3293-330 1202-3302, 2021. 7,8
[13] Zhiwei Lin and Yongtao Wang. Bev-mac: Bird's eye view masked autoeneoders for outdoor point cloud pre-training. arXiv preprint arXiv.22/2.05758, 2022. 3, 7, 8, 18
[14] Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang. Jingheng Chen, Xiaodan Liang. Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, et al. One million scenes for autonomous driving: Once dataset. arXiv preprint 106.:2206.110377 2021. 2.

---

[15] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proceedings of International Conference on Machine Learning, pages 8748-8763, 2021.
[16] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-779, 2019. 3
[17] Shaoshuai Shi, Chaoxu Guo, Li Jiang. Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-remn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529-10550 2020. 1,3,6
[18] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang. and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE transactions on pattern analysis and machine intelligence, 43(8):264722644, 2020.
[19] Shaoshuai Shi, Li Jiang, Jiajun Deng. Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn++: Point-voxel feature set abstraction with local vector representation for 3d object detection, International Journal of Computer Vision, 131(2)55115551 2023. 1, 3, 5,8
[20] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patniik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2446-2454, 2020. 2, 2,
[21] Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Proceedings of Advances in Neural Information Processing Systems, volume 30, 2017. 2,5,7
[22] 3D'raan Development Team. 3dtrans: An open-source codebase for exploring transfcrable autonomous driving perception task. https:
cwm/P JLab- ADG/ /0G/3Drrans, 2023. 8
[23] Yan Wang, Xiangyu Chen, Yurong You, Li Erran Li, Bharath Hariharan, Mark Camphell, Kilian Q Weinberger, and Wei-Lun Chao. Train in germany, test in the usa: Making 3d object detectors generalize. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11713-
11723, 2020. 5
[24] Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu. Lidar distillation: Bridging the beam-induced domain gap for 3d object detection. arXiv preprint arXiv:2203. 11003..49555 14956, 2022. 2
[25] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Proceedings of European Conference on Computer Vision, pages 7744591,7 2020. 8
[26] Runsen Xu, Tai Wang, Wenwei Zhang, Runjian Chen, Jinkun Cao, Jiangmiao Pang, and Dahua Lin. Mv- jar: Masked voxel jigsaw and reconstruction for lidar-based self-supervised pre-training. arXiv preprint arXiv:2303. 135100 2023. 3
[27] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10): 3337, 2018. 3, 6,
[28] Honghui Yang. Tong He. Jiaheng Liu, Hua Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wanli Ouyang. Gd-mae: Generative decoder for mae pre- pre-trainin on lidar point clouds. arXiv preprint 2xXix:2212.03010 2022. 3
[29] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition. pages 13368-03788. 2021. 2,
[30] Zetong Yang, Yanan Sun. Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1951-1960, 2019.
[31] Junbo Yin, Dingfu Zhou, Liangjun Zhang. Jin Fang. Cheng ZZong Xu, Jianbing Shen, and Wenguan Wang. Proposalcontrast: Unsupervised pre-training for lidar based 3d object detection. In Proceedings of European Conference Computer Vision, pages 17-33, 2022. 1, 3, 7,
12

---

[32] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11784- 11793, 2021. 8,8
[33] Jiakang Yuan. Bo Zhang, Xiangchao Yan, Tao Chen, Botian Shi, Yikang Li, and Yu Qiao. Bi3d: Bi-domain active learning cross-domain object detection. arXiv preprint arXiv:2303.05880 2023. 2,
[34] Bo Zhang, Jiakang Yuan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Uni3d: A unified baseline for 3d object detection. arXiv preprint arXiv:2303, 06880, 2023, 2,5
[35] Jiyang Zheng, Weihao Li, Jie Hong, Lars Petersson, and Nick Barnes. Towards open-set object detection and discovery. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 3961-3970. 2022. 6
13

---

Supplementary Material
In this supplementary material, we provide more details and experimental results not included in our main text.
Outlines:
Sec. A: More details about large-scale pre-training dataset preparation.
Sec. A.I: Preliminary experiments on class-aware pseudo label generator.
Sec. A.2: Analysis on pseudo label threshold on different classes.
Sec. A.3; Visualization results of pseudo labels.
Sec. A.4: Details of object re-scaling.
Sec. A.5; Taxonomy difference between different datasets.
Sec. B: Detailed dataset description and evaluation metries.
- Sec. B.1: Dataset description.
Sec. B.2: Evaluation metrics.
Sec. C: More implementation details.
Sec. D: More experimental results.
Sec. D.1: Ablation studies on unknown-aware instance learning head.
Sec. D.2: More results of pre-training scalability.
Sec. D.3: Results of fine-tuning on ONCE.
Sec. D.4: Visualization results.
A More Details about Large-scale Pre-training Dataset Preparation.
In this section, give some preliminary experimental results and analysis on large-scale pre-training dataset preparation.
A.1 Preliminary Experiments on Class-aware Pseudo Label Generator
As mentioned in Sec. 3.2.1 in submission, we explore how to improve the performance ONCE. We first analyze the results in the ONCE benchmark and find that CenterPoint reaches the SOTA performance
pedestrian and eyclist while PV-RCNN achieves the best performance vehicle. To use a stronger baseline to further improve the performance, we conduct experiments using PV-RCNN+ as the baseline detector. As shown in Tab. 12, PV-RCNN+ with center head can obtain a satisfactory performance on ONCE while PV-RCNN+ with anchor head can achieve better accuracy on vehicle and pedestrian.
Further, to obtain more accurate pseudo labels, we use a semi-supervised learning method to further improve the performance as shown in Tab. 13. Finally, individually train pedestrian using CenterPoint and other classes using PV-RCNN++
Table 12: Effects of using different heads on PV-RCNN++. We report mAP using the ONCE evaluation metric.
Detector PV-RCNN+ Center Head
Head Choice
Vehicle
Pedestrian
Cyclist
71.61
45,27 43.86
61.15
PV-RCNN+
Anchor Head
81.72
66.17
A.2 Analysis on Pseudo Label Threshold on Different Classes
Fig. shows the precision under different IoU thresholds. The precision can be calculated by Precision TP/(FP + TP), where FP and TP denote false positive and true positive, respectively. We can observe that when loU thresholds are more than 0.8, 0.7, 0.7 for vehicle, pedestrian and eyclist, the number of TP instances is significantly more than that of FP instances.
14

---

Table 13: Effects of using MeanTeacher. We report mAP using the ONCE evaluation metric.
Detector
McaheTacher
Vehicle
Pedestrian
Cyclist
CenterPoint
46.22
CenterPoint
56.01
PV-RCNN+
81.72
66.17
PV-RCNN+
82.50
71.19
Vehicle
Pedestrian
Cyclist
Precision
Precision
Precision
66
0.6
0.6
844
0.4
04
02
02
0001
0142
0283
0304
044.
asea
0607
Beas
88
8601
81a3
8203
0.04
asas
asue
6607
03ad
0&8a
8840
8801
6142
0243
0.se
0607
Gand
061d
Figure 7: The Precision under different IoU thresholds.
The visualization of the pseudo label results under different thresholds in Fig. 8, we can see that some FP pseudo labels will be annotated when setting low thresholds, while some TP instances not be annotated when the thresholds are relatively high. To more intuitively see the impact of the threshold on pseudo labeling, we use the model annotate the samples of the ONCE validation for comparison with ground-truths.
A.3 Visualization Results of Pseudo Labels
Fig. 9 shows the visualization results of our final pseudo label results.
A.4 Details of Object Re-scaling
In detail, given bounding box 6 = (cx, Cy. Ca, h, 0x) and point clouds (pf P.. pf) within it, where (Cav Cy, C. (1, w, h) and 0,, denote the center, size and heading angle of the bounding box. We first transform points into the local coordinate with the following formula:
(p" Cr,p% - Cy. cx) R,
cos On
sin On
0
(6)
R
sin Or.
cos On. 0
where is matrix multiplication. Then, to derive the scaled object, the point coordinates inside the box and the bounding box size are scaled to be a< P." p.) and al,, w, h), where a the scaling factor. Finally, the points inside the scaled box are transformed back to the ego-car coordinate system and shifted to the center (cz, Cy. C2) as
p.
RJ + (cr,
A.5 Taxonomy difference between different datasets
As shown in Tab. 14, there exists huge taxonomy difference between some fine-tuning datasets and the pre-training dataset. result, some foreground instances may be regarded as background only using pseudo label as supervision.
15

---

(a)
(b)
(c)
2010
(e)
(f)
(h)
Figure 8: Visualization results under different pseudo label thresholds. (a-c): annotations with low thresholds (iee 0.6, 0.5, 0.5 for vehicle, pedestrian and cyclist, respectively). (d-f): the thresholds used in our methods. (g-i): high thresholds (i.e., 0.9, 0.8, 0.8 for vehicle, pedestrian and cyclist, respectively). The green and red bounding boxes represent ground-truths and detector predictions, respectively.
(b)
(c)
(d)
Figure 9: Pseudo-labeled annotation results on unlabeled set.
B Detailed Dataset Description and Evaluation Metrics
B.1 Dataset Description
ONCE dataset. ONCE dataset [14] is large-scale dataset that is built to encourage the exploration of self-supervised and semi-supervised learning in the autonomous driving scenario. ONCE is collected by 40-beam LiDAR in multiple cities in China and contains diverse weather conditions
16

---

Table 14: Taxonomy difference between different datasets.
Dataset
classes
ONCE (Pre-train)
Car, Truck, Bus, Pedestrian, Cyclist Vehicle, Pedestrian, Cyclist
Waymo (Fine-tune)
Car, Truck, Construction vehicle, Bus, Trailer, Barrier, Motorcycle, Bicycle,
nuScenes (Fine-tune)
Pedestrian. Traffic cone
KITTI (Fine-tunc)
Car, Pedestrian, Cyclist
(e.g., sumny, cloudy, rainy), traffic conditions, time periods (e.g., morning, noon, afternoon, night) and areas (e.g,, downtown, suburbs, highway, tunnel, bridge).
Waymo Open Dataset. Waymo Open Dataset [20] is widely-used large-scale autonomous driving dataset that is composed of 1000 sequences and divided into a train set with 798 sequences (~ 1~150 samples) and validation set with 202 sequences (~40k samples). The Waymo dataset gathered in the USA by 64-beam LiDAR and 200-beam short-range LiDAR with annotations in full 360". use the 1.0 version Waymo Open Dataset.
nuScenes Dataset. NuSenenes dataset [1] provides point cloud data from 32-beam LiDAR collected from Singapore and Boston, USA. It consists of 28130 training samples and 6019 validation samples. The data is obtained during different times in the day, different weather conditions and diverse set of locations (e.g., urban, residential, nature and industrial).
KITTI Dataset. KITTI dataset [5] is common-used autonomous driving dataset that contains 7481 training samples and divided into a train set with 3712 samples and a validation set with 3769 samples. The point cloud data is collected by 64-beam LiDAR in Germany. KITTI dataset only provides the annotations for the objects within the field view of the front RGB camera.
B.2 Evaluation Metrics
ONCE evaluation metric. Following ONCE official evaluation metric, we merge the car, bus and truck class into super-class (i.e., vehicle). Ap is used to evaluate the performance of the ONCE dataset, which can obtained by the following formula:
AP = 100 f max max (p(r')/ 2r)drr
(8)
where r is recall rates from 0.02 to 1.00 at step 0.02 and p(r) denotes the precision- curve. Mean average precision (mAP) is the average of the scores of the three categories. The Intersection over Union (IoU) thresholds are set to 0.7, 0.3 and 0.5 for vehicle, pedestrian and cyclist, respectively.
Waymo evaluation metric. Two difficulty levels (i.e., LEVEL 1 and LEVEL 2) utilized to evaluate the detection accuracy of Waymo dataset and we mainly focus on more difficult L2 performance, Among each difficulty level, we report AP and APH which can be formulated as:
=100 100 [ maxlp(r')) 2 rd, AP -P00 100 [ max(h(r)))(' 2 rrdr,
AP =
where the different between h(r) and p(r) is h(r) is weighted by the accuracy of heading accuracy.
nuScenes evaluation metric. Following the official NuScenes Evaluation Metric, we report mAp and nuScenes detection score (NDS). AP is defined as matches by thresholding the 2D center distance d on the ground plane and the mAP can be calculated by:
1
ΣEΣA AP. AP.
MAP
(10)
CD
where C is the set classes and To is the set of thresholds (ie., (0.5,1,2,4)). We mainly focus on 10 classes. NDS is the weighted of mAP and five true positive metries, including Average Translation
17

---

Error (ATE), Average Scale Error (ASE), Average Orientation Error (AOE), Average Velocity Error (AVE) and Average Attribute Error (AAE). The NDS can be formulated as:
mTP =
1 E NDS
5mAP+ +
Σ
(1 - min(L,
(11)
10
DEC
MTPE
where TP is the set of true positive metrics.
KITTI evaluation metric. We report mAP with 40 recall positions to evaluate the detection performance and the 3D loU thresholds is set to 0.7 for cars and 0.5 for pedestrians and cyclists.
C More Implementation Details
As shown in Tab. 15, we list some details about pre-training and fine-tuning datasets, Note that the voxel size of nuScenes is set to [0.1, 0.1, 0.2] following [13]. may have different dimensions of input features (e.g. ONCE use 4 dimension features as input while Waymo and nuScenes use dimension features) causing the input dimension of the first layer network to be different. We simply do not load the parameters of the first layer when this happens while fine-tuning. In the pre-training phase, we merge the pscudo-labeled data and a small amount of labeled data (i.e., ONCE train set) the pre-training dataset. In the fine-tuning phase, we fine-tune 30 epochs for Waymo, 20 epochs for nuScenes and 80 epochs for KITTI.
be seen that different datasets
Table 15: Some implementation details about pre- pre-training and fine- mhe-tunnng datasets.
Dataset
Point cloud range
voxel size
input features
ONCE (Pre-train)
|-75.2, -75.2, 75.2, 75.2, 3.0]
[0.1, 0.1, 0.2]
[x, y, intensityl
Waymo (Fine-tune)
[-75,2. 75.2, 75.2, 4.0)
[0.1. 0.15]
[x. y. z, intensity, elongation]
nuScenes (Fine-tune)
[-51.2, -5.2,-551,,
$1.2. 3.0]
[0.1, 0.2]
[x. y, z, intensily, timestamp)
KITTI (Fine-tung
[0.0, -40.0, -3.0,, 70.4, 40.0, 1.0]
[0.05, 0.05, 0.1]
[x, y, z, intensity]
D
More Experimental Results
D.1 Ablation Studies on Unknown-aware Instance Learning Head
In this part, we conduct experiments to ablate the hyper- hyper-paramess parameters in unknown aware instance leamning head (i.e,, the number M of selected features and the distance threshold 7).
Tab. 16 shows the results using different numbers of selected features in unknown-awe aware instance learning head when pre-training. When M is small, some foreground instances with relatively low scores are ignored, while when M is large, the matched background regions are increased. Considering these factors, choose M to be 256.
Tab. 17 shows the performance under different distance thresholds in Eq. the main submission. The number of matched features is relatively small when using lower t, thus can not fully exploit the unknown foreground instances. When using a larger threshold, some mismatches may occur. Finally, we set to 0.3 as mentioned in our main submission.
Table 16: Ablation studies of the number M of selected features.
Waymo L2 AP/ APH
M
Overall
Vehicle
Pedestrian
Cyclist
67.71 64.98
67.91 67.45
68.54 61.87
66.67 65.63 68.00 / 67.00
256
68.33 65.69
68.17 / 67.70
68.82/ 62,39
512
67.93 65.24
68.04/ / 67.36
68.63 62.12
67.14 / 66.23
D.2 More Results of Pre-training Scalability
In this section, we show more results verify the pre-training scalability. We pre-train the model on the small, middle and large splits of the ONCE dataset and then fine-tune the model on 3% Waymo
18

---

Table 17: Ablation studies of the distance threshold т.
Waymo L2 AP/APH
7
Overall
Vehicle
Pedestrian
Cyclist
0.1
67.90 65.22
68.01 67.54
68.52 / 62.01
67.17 66.12
0.3
68.33 / 65.69
68.17 67.70
68.82 / 62.39
68.00 / 67.00
0.5
67.82 65.15
67.73 67.26
68.26 61.73
67.49 / 66.46
and 20% KITTI train data. As shown in Tab. 18, the scale of the pre-training dataset and the diversity of scenarios increases, the performance of fine-tuning on the downstream dataset will also improve.
Table 18: The pre-training scalability. We use ONCE to pre-train and Waymo and KITTI to fine-tune.
Waymo 1.2 APIAPH
KITTI Moderate mAP
Pre-training dataset
Overull
Vehicle
Pedestrian
Cyclist
Overall
Cur
Pedestrian
Cyclist
ONCE (~ 100k)
68.33/ 65.69
68.17 / 66..70
68.82 12.2.3
68.00
69.43
82.75
57.59
67.96
ONCE (~-00kk)
69.04/ 66.52
68.69 68.23
69.81 66..44
68.61 /6.66
71.36
83.17
58.14
72.78
ONCE (~1M)
69.63 / 67.08
69.03 68.57
70.54 64.34
69.33 68.33
72.37
83.47
59.84
73.81
D.3
Results of fine-tuning on ONCE.
(a)
(b)
(c)
(d)
(c)
(f)
Figure 10: Visualization of fine-tuning results. We visualize the results of three downstream datasets. (a-c): results of Waymo. (d-f): results of nuScenes. (g-i): results KITTI. The green and red bounding boxes represent ground-truths and detector predictions, respectively.
In our main submission, we report the fine-tuning performance on multiple datasets which are different from the pre-training dataset. Here, we show some fine-tuning performance on ONCE, As shown in Tab. 19, the performance can be largely improved when the baseline detectors are initialized by AD-PT. For example, when using SECOND as the baseline detector, the overall performance can be improved from 56.47% to 64.10% (+7.63%). We use the ONCE train set to fine-tune the model.
19

---

Table 19: The fine-tuning performance on ONCE validation set.
SECOND
CenterPoint
Overall
0-30m
30-50m
250m
Overall
0-30m
30-50m
550m
Random Initialization
56.47
65
51.05
36.44
64.94
74.52
59.47
44.28
Intializaaion
D.4 Visualization Results.
Fig. 10 shows the visualization results of three downstream datasets (i.e., Waymo, nuScenes, KITTI).
20


Reviews:
[{"review_id": "4sHqCuXr3L", "reviewer": "Reviewer_8so5", "summary": "This paper aims at large-scale point cloud dataset pre-training. The authors propose AD-PT method to build pre-training dataset  with diverse data distribution and learn generalizable representations. In details, they design a diversity-based pre-training data preparation procedure and unknown-aware instance learning. The extensive experimental results on Waymo, nuScenes and KITTI datasets verify the effectiveness of the proposed method.", "questions": "Please reply to the weakness part.", "limitations": "The paper does not have limitation section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Pre-training on the large-scale point cloud dataset is meaningful for autonomous driving. On the one hand, human annotation is expensive and how to use unlabeled data is an improtant direction. On the other hand, the point clouds of different LiDAR sensors have different patterns and we need consider these domain gaps.\n2. The proposed method is reasonable. Although some thechniques are not new, suach as contrastive learning, I think they are reasonable for large-scale point cloud pretraining. For example, the LiDAR beam re-sampling can increase the robustness of beam domain gap.\n3. The paper is well-written and easy to read.", "weaknesses": " \n1. The results in Table 3 cannot show the superiority of the proposed method. The performance gain with SS-PT methods such as BEV-MAE is marginal.\n\n 2. I think a more useful setting is that we have some datasets A,B and C collected from different LiDAR sensors. We can use the labels in all three datasets and we want to improve the detector's performance on C dataset. Can the proposed method be applied to this setting? I hope the authors can give some insights.\n\n 3. The authors should give an analysis between the detector's performance on finetuning dataset and the labels amount in pre-training datasets. As an extreme case, if we do not have labels in pre-training dataset, will the performance drop largely?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 8so5\n\nDear Reviewer XpxU,\n\nThanks for your review, we provide more experimental results and explanations of your question.\n\n**Q1: Table 3 cannot show the superiority of the proposed method. The performance gain with SS-PT methods such as BEV-MAE is marginal.**\n\n**A1**: \n- Different from previous pre-training methods like BEV-MAE, which use **the same** dataset as the pre-training and fine-tuning dataset. AD-PT mainly focuses on designing a general pre-training pipeline that can pre-train backbones with strong generalization ability which can improve the performance on **multiple downstream datasets** (e.g., nuScenes, Waymo).\n\n- Here we list the percentage of our performance improvement over BEV-MAE compared to training from scratch ($(ADPT - BEVMAE) / (BEVMAE - Scratch)$). As shown in the following table, the percentage of Waymo's improvement is more than 45% compared with BEV-MAE under multiple detectors.\n\n  |Detector|Percentage (AP / APH) |\n  | --- | --- |\n  |SECOND|  56.1% / 88.6%   |\n  |CenterPoint|  55.6% / 45.4%  |\n  |PV-RCNN++| 198.1% /  211.3%  |\n\n - Meanwhile, using the same AD-PT pre-trained checkpoint, the performance on multiple datasets can be improved (e.g., nuScenes). However, under cross-dataset settings (i.e., Waymo pre-train, nuScenes fine-tune), performance can not be improved when initialized by BEV-MAE, as shown in the following table.\n  |Method|mAP|NDS|\n  | ---  | ---| --- |\n  |From scratch (CenterPoint) | 56.2 | 64.5|\n  |BEV-MAE (CenterPoint) | 56.30| 64.62|\n  |AD-PT (CenterPoint) | 57.17 | 65.48 |\n   \n- Further, AD-PT achieves continuous performance gains, by expanding the scale of the pre-training dataset. When pre-training on a more large-scale dataset (i.e., 500K ONCE samples), the performance can be further improved as shown in the following table.\n\n  |Method| Overall L2 AP / APH | Vehicle| Pedestrian | Cyclist|\n  | --- | --- | --- | --- | --- |\n  |BEV-MAE (SECOND) | 61.03 / 57.30 | 64.42 / 63.87| 59.97 / 50.65| 58.69 / 57.39 |\n  |AD-PT (SECOND, 100K pre-trained data) | 61.26 / 57.69| 64.54 / 64.00 | 60.25 / 51.21 | 59.  00 / 57.86 |\n  |AD-PT (SECOND, 500K pre-trained data) | 62.34 / 58.74| 65.20 / 64.66| 61.26 / 52.20 | 60. 56 / 59.36 | \n\n- When fine-tuning on a small amount of data, AD-PT shows better performance compared to previous methods.\n\n  |Method | Data amount | Overall L2 AP / APH | Vehicle| Pedestrian | Cyclist|\n  | ---|---|---|---|---|---|\n  |From scratch (PV-RCNN++) | 3% | 63.81 / 61.10 | 64.42 / 63.93 | 64.33 / 57.79 | 62.69 / 61.59 |\n  |BEV-MAE (PV-RCNN++) | 3% | 64.87 / 62.05 | 65.54 / 65.04 | 65.46 / 58.98 | 63.62 / 62.15|\n  |AD-PT (PV-RCNN++) | 3% | 68.33 / 65.69 | 68.17 / 67.70 | 68.82 / 62.39 | 68.00 / 67.00|\n\n**Q2: ...We have some datasets A,B and C collected from different LiDAR sensors. ... Can the proposed method be applied to this setting?** \n\n**A2**: Thanks for your constructive suggestion, data from different sensors can greatly improve the diversity of the pre-training dataset. AD-PT proposes a method to increase the diversity of data with almost zero cost and an effective pre-training method. When obtaining more datasets with different sensors, the AD-PT can also effectively increase the diversity and our pre-trained method can easily be combined with different datasets.\n\nWe conduct simple experiments to show that our method can be used in **multi-dataset pretraining scenarios**. We use KITTI dataset, ONCE labeled dataset and our pseudo-labeled dataset. KITTI dataset is collected by a 64-beam LiDAR and ONCE dataset is collected by a 40-beam LiDAR. Inspired by Uni3D[1], we align the point cloud range. The following table shows the fine-tuning results on 3% Waymo train set.\n\n  |Method|Fune-tuning data amount|Pre-training data |Overall L2 AP / APH |Vehicle|Pedestrian |Cyclist|\n  | ---|---|---|---|---|---| ---|\n  |AD-PT (PV-RCNN++) | 3% | once labeled dataset, pseudo-labeled dataset |68.33/65.69|68.17/67.70|68.82/62.39|68.00/67.00|\n  |AD-PT (PV-RCNN++) | 3% | KITTI dataset, once labeled dataset, pseudo-labeled dataset |68.67/66.00|68.47/67.98|69.12/62.63|68.41/67.39|\n\n\n[1] Zhang B, Yuan J, Shi B, et al. Uni3d: A unified baseline for multi-dataset 3d object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 9253-9262.\n\n**Q3:  Give an analysis between the detector's performance on finetuning dataset and label amount in pre-training datasets.**\n\n**A3**: \n- AD-PT uses the semi-supervised manner to pseudo a large amount of unlabeled data. Here, we conduct experiments on reducing the labeled data, and the results are shown in the following table. The ONCE labeled set consists of a total of 6 sequences, and we reduce the amount of data for labeling by extracting random sequences.\n\n  |Method| Labeled data | Overall L2 AP / APH | Vehicle| Pedestrian | Cyclist|\n  | --- | --- | --- | --- | --- | --- |\n  |AD-PT|  1 sequence from labeled set | 67.11/64.72 | 66.98/66.48 | 67.93/61.29 | 66.44/65.41 |\n  |AD-PT| 3 sequences from labeled set | 67.47 / 64.78 | 67.52/67.03 |68.35/61.80|66.54/65.52|\n  |AD-PT| 6 sequences from labeled set | 68.33 / 65.69 | 68.17/67.70 | 68.82/62.39 | 68.00/67.00 |\n\nIt can be seen that performance degrades as the amount of annotated data decreases. Such a degradation is mainly due to that the accuracy of pseudo-labels will drop and such phenomenon is consistent with Figure 6 of the main text. Besides, Table 8 in the main text also shows the effect when using different amounts of unlabeled data.\n\n**Limitation**: Although the AD-PT pre-trained backbone can improve the performance on multiple downstream datasets, it needs to be verified in more actual road scenarios. Meanwhile, training a backbone with more generalization capabilities through data from different sensors is also a future direction.\nWe will add a limitation section to our next version.\n"}, {"review_id": "Glxt38wSN9", "reviewer": "Reviewer_hHpC", "summary": "The paper proposes an autonomous driving pre-training method, where a large-scale pre-training dataset is built and a generalizable representation is created. To further improve the performances on the down-stream tasks, an advanced pre-training technique of semi-supervised method is proposed. Moreover, an unknown-aware instance learning head is proposed to learn the open-set detection. Various experiments are conducted to show the effectiveness of the method.", "questions": "None", "limitations": "The limitation discussion seems to be missing. Possible memory consumption or time efficiency could be discussed to show a more comprehensive comparison with other SOTA works.,", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and clearly written.\n2. Thorough experiments are conducted on various benchmarks.", "weaknesses": "1. The novelty is a bit limited, since the proposed techniques look like a combination of existing approaches, such as data augmentation and semi-supervised learning.\n2. Although the improvements on various benchmarks are promising, the results are inferior compared to SOTA Lidar based object detection approaches such as VISTA [A] and MGTANet [B] on the nuScenes dataset. Hence, it would be great to also test on current SOTA approaches, which could further verify the generalization ability and robustness of the method. \n3. Eq. 2 seems to be incorrect since x and y share the same calculation. It would be further clearer if a figure is drawn to show the calculation.\n4. Line 293-294: Should the improvements for Waymo and nuScenes be 0.65%/0.54% and 7.37%, respectively?\n\n[A]: Deng, Shengheng, et al. \"Vista: Boosting 3d object detection via dual cross-view spatial attention.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[B]: Koh, Junho, et al. \"MGTANet: Encoding Sequential LiDAR Points Using Long Short-Term Motion-Guided Temporal Attention for 3D Object Detection.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 1. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer hHpC\n\nWe sincerely appreciate the Reviewer's efforts and comments. We have also tried our best to clarify the novelty of the proposed method and supplement more experimental results and discussions here.\n\n**Q1: Novelty is a bit limited, since the proposed techniques look like a combination of existing approaches, such as data augmentation and semi-supervised learning.**\n\n**A1**:\n\n- Our study provides the autonomous driving community with **knowledge** that scaling up the amount of pre-training samples can boost the performance on **multiple downstream datasets** simultaneously. Such an insight can inspire the pre-training study in the autonomous driving community, with the purpose of fully leveraging the constantly increasing autonomous driving data.\n- Our proposed method mainly considers the generalization ability of the model while improving the performance of downstream tasks.   \n  - The re-scaling and re-sampling methods aim to increase the diversity of pre-trained data which improves the backbone's generalization ability in **data-level**.\n  - Unknown-aware instance learning head fully considers the difference of pre-training and fine-tuning dataset which improves the backbone's generalization ability in **algorithm-level**.\n\n  As shown in Tables 8, 9, 10 of the main text, initialized by our pre-trained backbone (**a single checkpoint**), the performance of multiple downstream datasets can be improved and even surpass the baselines which pre-train and fine-tune on the same dataset.\n\n- Further, *as the amount of pre-trained data increases, the performance of downstream tasks will be continuously improved as shown in Table 8* of the main text. Such a phenomenon **cannot be observed** by previous pre-training methods (*e.g.* ProposalContrast, BEV-MAE, etc.), due to that the scale of the pre-training dataset is difficult to be continuously expanded, only using previous pre-training methods (they perform the pre-training and fine-tuning on the same benchmark). Besides, we conduct experiments on more fine-tuning data (*i.e.* Waymo 20%), as shown in the following table. Overall, different from previous semi-supervised learning and 3D pre-training methods (*e.g.* ProposalContrast, BEV-MAE, etc.), which aim to improve the performance **on the same dataset**, our work aims to perform the pre-training process and downstream fine-tuning process **across different datasets**, reducing the model retraining cost and achieving the fine-tuning performance scalability. This is the first work to focus on pre-training 3D backbone that can be verified to be effective in many autonomous driving datasets such Waymo, nuScenes, KITTI.\n\n  |Method| Overall L2 AP / APH | Vehicle| Pedestrian | Cyclist|\n  | --- | --- | --- | --- | --- |\n  |AD-PT (SECOND, 100K pre-trained data) | 61.26 / 57.69 | 64.54 / 64.00 | 60.25 / 51.21 | 59.  00 / 57.86 |\n  |AD-PT (SECOND, **500K** pre-trained data) | 62.34 / 58.74 | 65.20 / 64.66| 61.26 / 52.20 | 60. 56 / 59.36 | \n\n- Current semi-supervised learning methods such as MeanTeacher cannot achieve a significant performance gain towards the downstream dataset, as shown in Table 1 of the main text and the following table.\n\n  |Method| mAP| NDS|\n  | --- | --- |---|\n  |MeanTeacher | 55.46 | 63.93 |\n  |AD-PT | 57.17 | 65.48 |\n\n**Q2: Although the improvements on various benchmarks are promising, the results are inferior compared to SOTA Lidar based object detection approaches such as VISTA.**\n\n**A2**: Thanks for your valuable comment. According to this Reviewer's suggestion, we supplement the experiments of initializing VISTA [Ref-A] with our pre-trained model using AD-PT. The result of training VISTA from scratch is 60.8 / 68.1 (as reported in their original paper [Ref-A]), and the result of training VISTA using the AD-PT checkpoint is 61.12 / 68.38. We report the results on nuScenes validation set, and the experimental results demonstrate that VISTA can be further improved by pre-training using the AD-PT method, which further verifies the generalization ability of our method. Furthermore, we will add the comparison results with VISTA [Ref-A] **in the next version**.\n\n[Ref-A]  Deng, Shengheng, et al. \"Vista: Boosting 3d object detection via dual cross-view spatial attention.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n**Q3: Eq. 2 seems to be incorrect since x and y share the same calculation. It would be further clearer if a figure is drawn to show the calculation.**\n\n**A3**: We thank the Reviewer very much for pointing it out. The correct formula should be:\n\n$\nx=rcos(\\phi)cos(\\theta),  y=rcos(\\phi)sin(\\theta), z=rsin(\\phi)\n$\n\nWe will correct the formula in the next version and provide a figure to clearly show the calculation in the **one-page PDF**.\n\n\n**Q4: Line 293-294: Should the improvements for Waymo and nuScenes be 0.65%/0.54% and 7.37%, respectively?**\n\n**A4**: Thank the Reviewer very much for this comment, and we will correct it in the next version. \n\nLimitation: Although the AD-PT pre-trained backbone can improve the performance on multiple downstream datasets, it needs to be verified in more actual road scenarios. Meanwhile, training a backbone with more generalization capabilities through data from different sensors is also a future direction.\n\nWe will add a limitation section to our next version.\n"}, {"review_id": "NX4XSiBRlP", "reviewer": "Reviewer_nLKe", "summary": "This paper focuses on the point-cloud pre-training problem in the autonomous driving scenarios. Particularly, the authors regard the pre-training task as a semi-supervised learning problem and generate pseudo-labels for massive unlabeled data with few labeled frames. In the pseudo-label generation phase, they proposed two specific designs for this task, including diversity-based pre-training data preparation and unknown-aware instance learning. The experiments conducted on the KITTI, nuScenes, and Waymo datasets show the effectiveness of the proposed method.", "questions": "See Weaknesses. ", "limitations": "See Weaknesses.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The topic is meaningful. Different from the 2D vision task, the 3D detection community lacks effective pretraining methods and pre-trained backbones, and the strong pre-training backbones are helpful to the whole community.\n- The authors provide a pre-trained paradigm, and the experiments show the effectiveness of the pre-trained model, especially when training samples are limited.\n- Code will be publicly available. The authors provide source code for reproducibility.", "weaknesses": "- The work claims that it provides a general pre-trained backbone for the LiDAR-based 3D object detection task. However, different from 2D perception tasks which are dominated by a few kinds of backbones,  there are several popular detection pipelines with different backbone nets in this field. Some 3D detectors even use different input data representations, e.g. points, pillars, voxels, range images, etc. It is hard to build unified 3D pre-trained models for these works, and what this work provided is a voxel-based pre-trained backbone. Based on this, I think this work is over-claimed.\n\n- The 3D-based backbone is highly customized. Take the voxel-based backbone as an example, we need to pre-define the voxel size, and I think the pre-trained voxel size used in pretraining is the same as that used in the downstream task. I would like to know what would happen if different voxel sizes were used for pre-training and downstream tasks. If we must re-pretrain each backbone with different resolutions, then the significance of this work is not very important.\n\n- The improvement in performance is limited. I found that the pre-trained model has a relatively small impact on the final performance, especially in the full dataset setting. Considering the extra pre-training cost, performance improvement is limited, and I  want to know whether simply extending the training schedule can achieve comparable performance or not.\n\n- Novelty is limited. This work regards the pre-training task as a semi-supervised learning problem with a pseudo-labeling scheme. However, using a few labeled data to generate pseudo-labels for LiDAR-based 3D detection has been investigated in [1], which achieves better performance.\n\n[1] Pseudo-labeling for Scalable 3D Object Detection, https://arxiv.org/abs/2103.02093\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: More results of using the points, pillars, voxel-based methods to demonstrate the generalization ability of AD-PT.**\n\n**A1**:\n- As mentioned by the reviewer, there are several different types of 3D object detection backbones, *e.g.* pillar-based, point-based, and voxel-based. We verify the generalization ability of AD-PT based on the voxel-based method, due to it being the most commonly-used backbone currently.\n- Furthermore, we conduct experiments on different types of backbones, *e.g.* pillar- and point-based. The results are shown in the following tables.\n  |Methods (PointPillar)|Data amount (Waymo)|Overall L2 AP/APH|Veh.|Ped.|Cyc.|\n  |-|-|-|-|-|-|\n  |Scratch|3%|48.56/39.30|54.28/53.51|47.11/25.50|44.29/38.89|\n  |AD-PT|3%|52.01/43.99|58.51/57.85|50.22/32.52|47.31/41.59|\n  |Scratch|20%|57.85/50.69|62.18/61.64|58.18/40.64|53.18/49.80|\n  |AD-PT|20%|59.71/53.49|64.10/63.54|59.00/43.13|56.04/53.80|\n\n  |Methods (PointRCNN)|Data amount (KITTI)|Overall|Car|Ped.|Cyc.|\n  |-|-|-|-|-|-|\n  |Scratch|20%|64.12|75.30|52.52|69.55|\n  |AD-PT|20%|67.67|77.20|54.16|71.86|\n  |Scratch|100%|68.40|78.70|54.41|72.11|\n  |AD-PT|100%|70.47|80.25|57.13|74.04|\n\n  We conduct point-based experiments on KITTI dataset for a fair comparison due to the lack of config for Waymo in OpenPCDet.\n  The fine-tuning performance of point- and pillar-based methods can be further improved, when the models are initialized by our pre-trained checkpoint, showing that our method can combine with multiple types of 3D detectors.\n- We agree with the reviewer's suggestion that we do not provide a general backbone but provide a general pre-training pipeline that can be used on various types of 3D detectors. **We will revise it in the next version**.\n\n**Q2: About the voxel size used in pre-training.**\n\n**A2**:\n- The voxel size used in the experiments.\n  |Dataset (setting)|Voxel size|\n  |-|-|\n  |ONCE (pre-training)|[0.1,0.1,0.2]|\n  |Waymo (Fine-tuning)|[0.1,0.1,0.15]|\n  |nuScenes (Fine-tuning)|[0.1,0.1,0.2]|\n  |KITTI (Fine-tuning)|[0.05,0.05,0.1]|\n  As shown in the table, the voxel size used in pre-training is different from that used in the downstream task (fine-tuning stage). Especially for KITTI dataset, the table shows the flexibility of loading the backbone with different voxel sizes.\n- For fair comparison to previous methods like BEV-MAE, we use [0.1,0.1,0.2] on nuScenes in the main text. Further, we also supplement the results, by conducting the experiment on nuScenes using [0.075,0.075,0.2] in the following table.\n  |Methods (CenterPoint)|mAP|NDS|\n  |-|-|-|\n  |Scratch|58.07|66.00|\n  |AD-PT|59.30|66.72|\n\n**Q3: I want to know whether extending the training schedule can achieve comparable performance or not.**\n\n**A3**:\n- First, we conduct experiments on extending the training schedule, and the results are shown as follows.\n|Methods (PV-RCNN++)|Data amount (Waymo)|Epochs|Overall L2 AP/APH|Veh.|Ped.|Cyc.|time cost|\n|-|-|-|-|-|-|-|-|\n|Scratch|20%|150|70.95/68.51|70.25/69.81|72.18/66.35|70.43/69.38|5x time|\n|AD-PT|20%|30|71.55/69.2|70.62/70.19|72.36/66.82|71.69/70.70|1x time|\n|Scratch|3%|150|68.76/66.18|68.10/67.62|70.08/63.88|68.11/67.03|5x time|\n|AD-PT|3%|150|69.72/67.14|69.00/68.52 |71.11/64.93|69.04/67.96|5x time|\n\nInitialized by our pre-trained model, the results of only training 30 epochs can exceed the results of 150 epochs of training from scratch. Besides, the performance using our pre-trained model can consistently outperform train from scratch when extending the training schedule (e.g., 3% fine-tuning data for 150 epochs).\n- Further, our method can easily scale up the pre-training data compared with previous methods, and the performance of the downstream dataset can be further improved as shown in table 8 and the following table.\n  |Methods (SECOND)|Overall L2 AP/APH| Veh.|Ped.|Cyc.|\n  |-|-|-|-|-|\n  |Scratch|60.62/56.86|64.26/63.73|59.72/50.38|57.87/56.48|\n  |AD-PT (100K pre-training data)|61.26/57.69|64.54/64.00|60.25/51.21|59. 00/57.86|\n  |AD-PT (500K pre-training data)|62.34/58.74|65.20/64.66|61.26/52.20|60. 56/59.36|\n  |improvement|+1.72/+1.88|+0.94/+0.93|+1.54/+1.82|+2.79/+2.88|\n\n**Q4: Difference between our proposed AD-PT and pseudo-labels based 3D semi-supervised learning.**\n\n**A4**：\n- *Task-level differences*: Different from pseudo-labeling and semi-supervised learning mentioned by the reviewer which aims to find a better pseudo-labeling method, our work aims to propose a *pre-training* method that can learn unified 3D representations that can improve the performance on multiple downstream datasets. Meanwhile, previous 3D pre-training methods mainly focus on pre-training and fine-tuning on the same dataset. AD-PT is the first work to focus on pre-training and evaluating on different downstream datasets (Waymo, KITTI, nuScenes) in this community.\n- *Algorithm-level differences*: \n  - Pseudo-labeling mentioned by the reviewer tries to generate more useful pseudo labels using semi-supervised learning methods. As mentioned in our Method Section, we use the semi-supervised learning method to improve the quality of pseudo labels, which is **only a part** of our pre-training pipeline.\n  - Our proposed method mainly considers the generalization of the model while improving the performance of downstream tasks which is totally different from previous pre-training methods and semi-supervised learning methods.   \n    - The re-scaling and re-sampling methods aim to increase the diversity of pre-trained data which improves the backbone's generalization ability at data-level.\n    - Unknown-aware instance learning head considers the difference between pre-training and fine-tuning dataset which improves the backbone's generalization ability at algorithm-level.\n- *Application-level differences*: As the amount of pre-training data increases, the performance of downstream tasks will be continuously improved as shown in Table 8 and table in **A3**. Such a phenomenon **cannot be observed** by previous 3D AD pre-training methods. "}, {"review_id": "NSM4UlUp9b", "reviewer": "Reviewer_5kGB", "summary": "This paper studies model pre-training for autonomous driving by utilizing a large-scale point cloud dataset, which largely facilitates the generalization performance of pre-trained models. To reconstruct such a dataset, scene- and instance-level distribution diversity are carefully enhanced. The overall pretraining is conducted in a semi-supervised manner, and its effectiveness is validated on several benchmark datasets with different baseline models. ", "questions": "Na", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compared to the traditional pre-training setting for point cloud, the AD-PT pretraining setting is more practical since it is expected to learn a generalized pre-trained model for all downstream tasks. \n2. Compared to existing pre-trained methods trained in a fully unsupervised manner, this paper adopts a semi-supervised strategy, which is somewhat novel.\n3. The data diversity enhancement with re-sampling strategies is reasonable. So does the unknown aware instance learning head.", "weaknesses": "1. Although the AD-PT presented an improved performance on downstream tasks with limited training data (e.g., 20% data amount on Waymo), it is interesting to illustrate the results with more training samples in downstream tasks (e.g., 100% data amount on Waymo).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer 5kGB\n\nThanks for your review and suggestions.\n\n**Q1: Although the AD-PT presented an improved performance on downstream tasks with limited training data (e.g., 20% data amount on Waymo), it is interesting to illustrate the results with more training samples in downstream tasks (e.g., 100% data amount on Waymo).**\n\n**A1**: For a fair comparison with the previous works like ProposalContrast and BEV-MAE, we conduct experiments on fine-tuning using 20% Waymo data in the main text. Here, we show the results fine-tuned on 100% Waymo data.\n\n  |Method|Data amount | Overall L2 AP / APH | Vehicle| Pedestrian | Cyclist|\n  | --- | --- | --- | --- | --- | --- |\n  |From scratch (PV-RCNN++) | 100%| 71.66 / 69.45 | 70.61 / 70.18 | 73.17 / 68.00 | 71.21 / 70.19 |\n  |AD-PT (PV-RCNN++) | 100% | 72.39 / 70.10 | 71.01 / 70.60 | 74.84 / 69.46 | 71.32 / 70.25 | \n\n  It can be seen that the performance can be consistently improved using more fine-tuning data.\n"}]

Author rebuttal:
## General Response:

Dear AC and reviewers,

Many thanks for your valuable comments and constructive suggestions to improve the quality of our work. Here is a summary of what we have done in the rebuttal phase.

- We conduct several new experiments to cover the concerns of the reviewers.
    
  - Verifying the effectiveness of our pre-training methods on different types of 3D backbones (e.g., Pillar-based backbone, point-based backbone)

  - Compare our results with the result of simply extending the training schedule
  
  - Applying our pre-trained backbone to more SOTA works (i.e., VISTA)
    
  - More experiments to show the sensitivity of voxel size

  - A simple attempt at a fusion of multiple datasets
  
   - Reducing the labeled data

  - Fine-tuning on 100% Waymo data

  - More experiments to show the generalization ability of pre-trained backbone obtained by AD-PT

  - More experiments to show the fine-tuning performance when scaling pre-training data

- We further provide more discussion on our insight and novelty, where we provide the autonomous driving community with the knowledge that scaling up the amount of pre-training samples can boost the performance on multiple downstream datasets simultaneously. Such an insight can inspire the pre-training study in the autonomous driving community, with the purpose of fully leveraging the constantly increasing autonomous driving data.

- We provide a discussion on the performance of our methods and previous methods.

- We provide a discussion on how to train on the dataset with multiple sensor types.

- We provide the limitation of our methods.

Thank you again for your precious time on the review. We hope that our response has addressed your concerns. We are happy to have further discussion on anything unclear about our paper.

Best regards,
Authors of paper 4632



Meta-review: All reviewers agree that the paper tackles an important problem and proposes a solution with demonstrated effectiveness. There are some concerns that individual components of the approach is not novel, but reviewers agree that the paper should be accepted nevertheless because of strong results.
Decision: Accept (poster)