SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: 3D object detection performance comparisons on the ONCE val split. We report mAP and AP per class. Our method achieves the best performance among these state-of-the-art methods.
Methods	PreTrain	PreTrain Data	mAP	Orientation-aware AP
Vehicle	Pedestrian	Cyclist
PointRCNN (38)	X	-	28.74	52.09	4.28	29.84
PointPillars (20)	X	-	44.34	68.57	17.63	46.81
SECOND (54)	X	-	51.89	71.19	26.44	58.04
PV-RCNN (36)	X	-	53.55	77.77	23.50	59.37
IA-SSD (63)	X	-	57.43	70.30	39.82	62.17
CenterPoint (60)	X	-	60.05	66.79	49.90	63.45
DepthContrast (64)	X	-	51.89	71.19	26.44	58.04
✓	medium	52.81+0.92	71.92+0.73	29.01+2.57	57.51-0.53
PointContrast (52)	X	-	51.89	71.19	26.44	58.04
✓	large	53.59+1.70	71.87+0.68	28.03+1.59	60.88+2.84
SLidR (34)	X	-	28.80	52.10	4.17	30.13
✓‡	large	30.72+1.92	53.19+1.09	6.74+2.57	32.22+2.09
ProposalContrast (59)	X	-	64.24	75.26	51.65	65.79
✓	large	66.32+2.08	77.22+1.96	54.01+2.36	67.73+1.94
GD-MAE (55)	X	-	62.62	75.64	45.92	66.30
✓	large	64.92+2.30	76.79+1.15	48.84+2.92	69.14+2.84
Ours (Second)	X	-	52.95	74.93	24.33	59.58
✓†	large	56.10+3.15	77.11+2.18	27.51+3.18	63.69+4.11
Ours (PV-RCNN)	X	-	54.35	78.65	24.71	59.70
✓†	large	57.45+3.10	80.99+2.34	27.93+3.22	63.44+3.74
Ours (CenterPoint)	X	-	64.28	76.82	49.99	66.02
✓†	small	65.87+1.59	78.56+1.74	51.50+1.51	67.55+1.53
✓†	medium	66.72+2.44	79.27+2.45	52.53+2.54	68.36+2.34
✓†	large	67.41+3.13	79.50+2.68	53.36+3.37	69.36+3.34
$\dagger$  Use pixel semantics during pretraining.
$\ddagger$  Use super-pixel and pixel features during pretraining.
divided into three subsets: small (100k scenes), medium (500k scenes), and large (1M scenes) for pre-training purposes. The official evaluation metric is the mean Average Precision (mAP).
Implementation Details. We use DSVT-P (47) as our default encoder, given its flexibility. For the decoder, we use a 4-layer MLP with 256 hidden channels, and Softplus is chosen as the activation function. For image segmentation, we utilize DeepLabV3 (7) with MobileNets (18) as the backbone. During pre-training, we train the model using the AdamW (25) optimizer and the one-cycle policy, with a maximum learning rate of  $3e^{-4}$ . We pre-train the model for 45 epochs on the nuScenes dataset, 20 epochs on the ONCE small, 5 epochs on the ONCE medium, and 3 epochs on the ONCE large. During fine-tuning, we employ random flipping, scaling, rotation, and copy-n-paste as data augmentations, with a maximum learning rate of  $3e^{-3}$ . All experiments are conducted on NVIDIA V100 GPUs. For more implementation details, please refer to Appendix.
Overlap of labels between pixel semantics and downstream tasks. Our pre-training phase utilizes pixel semantic labels that include 19 classes such as road, sidewalk, building, wall, fence, pole, traffic light, sign, vegetation, terrain, sky, person, rider, car, truck, bus, train, motorcycle, and bicycle. In the nuScenes object detection task, the labels include car, truck, construction vehicle, bus, trailer, barrier, motorcycle, bicycle, pedestrian, and traffic cone. For the ONCE object detection task, the labels are limited to vehicle, pedestrian, and cyclist. The nuScenes BEV map segmentation task uses labels such as drivable, pedestrian crossing, walkway, stop line, car park, and divider. There's an overlap in labels like 'car', 'bicycle', and 'pedestrian', and our approach is more in line with the field of weakly supervised learning.
Table 3: Performance comparisons on nuScenes val BEV map segmentation. We report mIoU and IoU per class. Our method quantitatively outperforms prior works.
Method	PreTrain	mIoU	Drivable	Ped. Cross.	Walkway	Stop Line	Carpark	Divider
PointPillars (20)	X	43.8	72.0	43.1	53.1	29.7	27.7	37.5
CenterPoint (60)	X	48.6	75.6	48.4	57.5	36.5	31.7	41.9
DSVT (47)	X	51.6	79.7	51.8	61.1	38.2	33.8	45.3
Ours	✓	55.0+3.4	82.5	54.4	65.2	40.5	39.4	48.1
Table 4: Variations in Encoder Types. We maintain the same components, merely swapping the 3D backbone.
Encoder	PreTrain	mAP	NDS
2D Conv (20)	X	50.0	60.7
✓	53.0+3.0	62.6+1.9
VoxelNet (60)	X	56.2	64.5
✓	59.0+2.8	66.3+1.8
DSVT-P (47)	X	61.5	68.0
✓	64.2+2.7	69.7+1.7
Table 5: Neural Rendering. Our rendering-based method outperforms direct projection, showcasing its effectiveness.
Case	PreTrain	mAP	NDS
baseline	X	61.5	68.0
projection	✓	62.0+0.5	68.3+0.3
projection-filt	✓	62.8+1.3	68.8+0.8
rendering w/o depth	✓	63.5+2.0	69.2+1.2
rendering w/ depth	✓	64.2+2.7	69.7+1.7
4.2 3D Object Detection
In this section, we investigate pre-training in an autonomous driving context. We benchmark the performance of our model against previous methods on the nuScenes and ONCE datasets.
NuScenes. We employ Second (54), CenterPoint (60), and DSVT (47) as our baseline and present the performance on the validation and test sets of nuScenes in Table 1. Our pre-trained model consistently improves the performance of all baselines compared to models trained from scratch. On the validation set, our method boosts the Second, CenterPoint, and DSVT by  $+3.0 / 2.0$  mAP/NDS,  $+2.7 / 1.7$  mAP/NDS, and  $+1.6 / 0.9$  mAP/NDS, respectively. With CenterPoint-Voxel as the detector (5-th row), our method surpasses previous pre-training methods (59; 52; 22) that also utilize CenterPoint-Voxel detector by a large margin. On the test set, our method boosts the CenterPoint and DSVT by  $+2.6 / 1.7$  mAP/NDS and  $+1.7 / 1.0$  mAP/NDS, respectively. Moreover, our method significantly outperforms previous state-of-the-art LiDAR detectors (20; 65; 47; 1; 35).
ONCE. As shown in Table 2, our model surpasses all other methods on the ONCE validation split. When using CenterPoint (60) as the detector, our method enhances the baseline by  $3.13\mathrm{mAP}$ , outperforming the improvements made by ProposalContrast (59) and GD-MAE (55) by 1.05 and 0.83, respectively. While SLidR (34) also incorporates image information during pre-training, our method significantly outperforms SLidR, showcasing the effectiveness of our approach in utilizing image information. The distinction between the 'no-pretrain' versions of SLidR and our model originates from their foundational detection frameworks, where SLidR employs PointRCNN as its detector. As the scale of pre-training data continues to increase (from small to large), our method delivers continuous gains, indicating its scalability with respect to data size.
4.3 BEV map segmentation
We also demonstrate the versatility of our approach by evaluating its performance on the BEV Map Segmentation task of the nuScenes dataset (4). We provide the IoU results for six background classes in Table 3. Leveraging our PRED, we achieve a significant improvement of  $3.4\%$  over DSVT's performance, demonstrating the effectiveness of our approach in map segmentation tasks.
In this section, we conduct a series of ablation studies on the nuScenes val to explore the essential components of our methodology. Unless otherwise specified, we use CenterPoint (60) as the detector.
Figure 4: Masking strategy and masking ratio. Point-wise masking outperforms patch-wise masking and group-wise masking when a high masking ratio (95%) is applied.
Encoder type. To demonstrate the adaptability of our approach across various 3D backbone architectures, we experiment by substituting DSVT-P (47) with 2D Conv from PointPillar (20), and VoxelNet from CenterPoint-Voxel (60). The results, presented in Table 4, show that our approach can boost detector performance by  $3.0 / 1.9\mathrm{mAP} / \mathrm{NDS}$  and  $2.8 / 1.8\mathrm{mAP} / \mathrm{NDS}$  when employing 2D Conv and VoxelNet as backbones, respectively. These improvements are comparable to those achieved using DSVT-P. Due to DSVT-P's superior performance and flexibility, it was chosen as our default encoder. However, our method proves to be effective with a variety of encoder types.
Semantic rendering design. Here, we verify the efficacy of our semantic rendering pipeline as shown in Table 5. A basic approach to leveraging images is to directly project the point cloud onto the images. However, as indicated in the 2nd row, this method does not yield significant improvements due to the overlooking of occlusion effects during the projection process. A potential solution is to filter the point cloud based on the image depth, thereby excluding points that do not align with the image depth from the training batch. In the absence of ground-truth image depth, we employ BEVDepth (21) for depth estimation. The enhanced performance in the 3rd row underlines the importance of handling occlusion. However,
Table 6: Semantic loss. Both class balance sampling and confidence weight play an important role in calculation of semantic loss.
Weight	Cls-balance	mAP	NDS
62.5	68.5
✓	63.2	69.0
✓	63.7	69.3
✓	✓	64.2	69.7
the inaccuracy of depth estimation hinders further performance improvement. Our neural rendering approach, shown in the 4th row, surpasses the prior two methods by adaptively managing occlusion, where attributing a reduced weight to occluded points when accumulating semantics as in Equation 4. Ultimately, incorporating depth supervision yields the best results, as depicted in the final row.
Table 6 presents ablation experiments on class balance sampling and confidence weight in semantic supervision. Both components are crucial for semantic supervision; the confidence weight mitigates the effects of inaccurate semantic labels, and class balance sampling addresses the issue of category imbalance in point clouds. For depth supervision, We compared two different rendering views in Table 7. The lidar view outperforms the image view, as the depth of points might be inaccurate in the image view, but not in the lidar view.
Table 7: LiDAR view. For depth supervision, the lidar view proves superior to the image view.
Case	mAP	NDS
image view	63.6	69.3
lidar view	64.2	69.7
Masking strategy. In Figure 4, we explore various masking strategies to investigate their impact on our pre-training framework: 1) Patch-wise masking masks a certain fraction of non-empty tokens, with each token representing a non-overlapping patch of the BEV feature map. 2) Group-wise masking masks a certain proportion of point cloud groups, with each group selected using the Furthest Point Sampling (FPS) and  $k$ -Nearest Neighbor (kNN) algorithm, following (61). 3) Point-wise masking, as elaborated in Section 3.2. As illustrated in Figure 4, group-wise and patch-wise masking strategies perform better when the masking ratio is low. When the ratio reaches  $70\%$ , the impact of group-wise and patch-wise masking hits its peak. Intriguingly, the effectiveness of point-wise masking continues to ascend until the masking ratio reaches  $95\%$ . The results from point-wise masking surpass those of the other two methods, achieving a score of  $64.2 \mathrm{mAP}$ . Consequently, we opt for
the point-wise masking strategy with a  $95\%$  masking ratio, as it offers the advantage of accelerated processing while preserving good performance.
Data-efficient. A significant advantage of pre-training lies in its ability to enhance data efficiency for downstream tasks that have limited annotated data. In this study, we examine data-efficient 3D object detection by first conducting pre-training on the nuScenes (4) training data, then fine-tuning the pre-trained model with varying fractions of training data:  $10\%$ ,  $20\%$ ,  $50\%$ , and  $100\%$ . The results of these experiments are illustrated in Table 8. Overall, our pre-trained model consistently boosts detection performance, especially when the available labeled data is limited, i.e., improving  $4.8\mathrm{mAP}$  and 3.1 NDS with only  $10\%$  labeled data. Remarkably, even when utilizing only  $50\%$  of the annotated data, the pre-trained model achieves  $63.0\mathrm{mAP}$  and 68.5 NDS, surpassing the performance of a non-pre-trained version using  $100\%$  of the annotated data.
More experimental results are provided in Appendix.
Table 8: Data Efficiency. Our approach consistently enhances the detection performance, particularly when the labeled data is limited.
Dataset fraction	PreTrain	mAP	NDS
10%	×	49.6	58.9
✓	54.4+4.8	62.0+3.1
20%	×	55.1	62.9
✓	59.3+4.2	65.5+2.6
50%	×	59.7	66.2
✓	63.0+3.3	68.5+2.3
100%	×	61.5	68.0
✓	64.2+2.7	69.7+1.7
In this paper, we introduce PRED, an effective pre-training framework for outdoor point cloud data. Our framework mitigates the inherent incompleteness of point clouds by integrating image information through semantic rendering. It also allows for effective occlusion management by assigning a reduced weight to occluded points during volume rendering. A point-wise masking strategy, with a mask ratio of  $95\%$ , is adopted to optimize performance. Extensive experiments validate the effectiveness of our framework as it significantly improves various baselines on the large-scale nuScenes and ONCE datasets across various 3D perception tasks. We hope our PRED can serve as a powerful baseline to inspire future research on point cloud pre-training.
Limitations. PRED mainly focuses on point cloud pre-training with images as supervision. The integration of an image-point multi-modality pre-training could potentially improve performance. However, it's beyond this paper's scope and we intend to explore this possibility in our future research.
This work is supported by National Key R&D Program of China (2022ZD0114900) and National Science Foundation of China (NSFC62276005).
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR, 2022. 6, 8
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 1
[3] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021. 3
[4] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 1, 2, 6, 8, 10
[5] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021. 3
[6] Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang, Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. arXiv preprint arXiv:2303.08129, 2023. 1, 2, 3
[7] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017. 5, 7
[65] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv:1908.09492, 2019. 6, 8