PRED: Pre-training via Semantic Rendering on LiDAR Point Clouds
Authors: Hao Yang
Haiyang Wang
Di Dai
Liwei Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Pre-Train
Autonomous Driving
LiDAR
3D Object Detection

Abstract: Pre-training is crucial in 3D-related fields such as autonomous driving where point cloud annotation is costly and challenging. Many recent studies on point cloud pre-training, however, have overlooked the issue of incompleteness, where only a fraction of the points are captured by LiDAR, leading to ambiguity during the training phase. On the other hand, images offer more comprehensive information and richer semantics that can bolster point cloud encoders in addressing the incompleteness issue inherent in point clouds. Yet, incorporating images into point cloud pre-training presents its own challenges due to occlusions, potentially causing misalignments between points and pixels. In this work, we propose PRED, a novel image-assisted pre-training framework for outdoor point clouds in an occlusion-aware manner. The main ingredient of our framework is a Birds-Eye-View (BEV) feature map conditioned semantic rendering, leveraging the semantics of images for supervision through neural rendering. We further enhance our model's performance by incorporating point-wise masking with a high mask ratio (95%). Extensive experiments demonstrate PRED's superiority over prior point cloud pre-training methods, providing significant improvements on various large-scale datasets for 3D perception tasks. Codes will be available at https://github.com/PRED4pc/PRED.

PRED: Pre-training via Semantic Rendering on LiDAR Point Clouds
Haiyang Wang' Di Dai2 Liwei Wang Center for Data Science, Peking University 2 National Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University "Pazhou Lab Chaoy@stu, vanghalyang@stu, didai0stu, wanglw@cis} pku. edu. cn
Hao Yang 1.3
Abstract
Pre-training is crucial in 3D-related fields such autonomous driving where point cloud annotation is costly and challenging. Many recent studies on point cloud pre-training, however, have overlooked the issue of incompleteness,, where only a fraction of the points are captured by LiDAR, leading to ambiguity dur- ing the training phase. On the other hand, images offer more comprehensive information and richer semantics that can bolster point cloud encoders in ad- dressing the incompleteness issue inherent in point clouds. Yer, incorporating images into point cloud pre-training presents its own challenges due to occlusions, potentially causing misalignments between points and pixels. In this work, we propose PRED, a novel imagc-assisted pre-training framework for outdoor point clouds in an occlusion-awart manner. The main ingredient of our framework is Birds-Eye- Ereseew View (BEV) feature map conditioned semantic rendering, leverag- ing the semantics of images for supervision through neural rendering. We further enhance our model's performance by incorporating point-wise masking with a high mask ratio (95%). Extensive experiments demonstrate PRED's superiority over prior point cloud pre-training methods, providing significant improvements on various large-scale datasets for 3D perception tasks. Codes will be available http://ggihub.coo/PREDDeog
Introduction
Pre-training a fundamental task in deep learning, serving as a powerful tool harness the potential of copious unlabeled data and enhance the results of downstream tasks. This is especially critical in 3D-related fields, including autonomous driving, where the annotation of point clouds both labo- rious and expensive, contrasting with the relative ease of amassing large volumes of unlabeled point cloud data (59). To tackle this challenge, we present PRED (PRE-training via semantic renDering). novel framework designed to pre-train point cloud processors using multi-view images.
With the tremendous advancements in masked signal modeling within the realms of image and natural language processing (16; 12; 2), recent studies (55; 30; 62; 17; 61; 6) have delved into applying masked auto-encoders to pre-training in the field of point clouds. These works firstly apply patch masking to the input point clouds, then employ an encoder-decode the point clouds, depicted in Figure ((a). Apart from masked encoder,,,,,ss other researchers (52; 59; 64; 22; 34; 8) have explored point cloud pre- pre-training through contrastive learning. Here, positive pairs are generated by applying varied data augmentations to the point clouds, as demonstrated in Figure 1(b). Despite these significant strides, there remains a blind spot regarding the inherent incompleteness of point clouds, which is ubiquitous issue outdoor LiDAR datasets. For example in nuScenes (4), large-scale outdoor LiDAR dataset, over 30% of the labeled objects contain fewer
framework to reconstruct
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Masked Signal Modeling
(e) Ours
Semanik
Eneader
Contrastive Learning
Encoder
Decoder
Ornmatore
Encoder
Encuder
View
View2
map
Figure 1: Comparison of our framework with previous point cloud pre-train methods. Exist- ing pre-trate pre-train works can be broadly categorized into (a) mask signal modeling and (b) contrastive learning (c) We propose a novel framework that pre-trains point cloud encoder through semantic rendering.
than five points. As illustrated in Figure 2(a), this incompleteness introduces ambiguity into point cloud reconstruction, thereby potentially impacting the quality of the training process.
Images, compared point clouds, embody tics. Several studies (6; 8; 34) have sought to offset the limitations of outdoor point clouds by in- corporating images into point cloud pre-training. However, these works typically align point clouds with images through a straightforward point-to-pixel projection, neglecting to account for occlusion where objects in point clouds may be visible in the image due to misalignment between LiDAR and camera. This oversight can lead to mismatches between points and pixels, as depicted in Fig- ure 2(b), potentially hindering the effectiveness of pre-training. Hence the principal challenge we address in this paper is an occlusion- aware, image-assisted pre-training framework for outdoor point clouds.
comprehensive information and enriched seman-
In this study, we introduce novel pre-training framework, PRED), designed to tackle the problem of reconstruction ambiguity and occlusion by integrating image information through semantic render- ing, as illustrated in Figure (c)) Neural rendering has demonstrated substantial success in learning 3D implicit representations from 2D images within the sphere 3D generation (28; 31; 40). This inspires us to learn point cloud representations from image semantics via neural rendering. To ac- complish this, we first extract BEV feature map from the point cloud using an encoder. Then, we render the semantic maps from image views, conditioned on the BEV feature map. The entire pro- reconstruction cess is supervised by image semantics, thereby circumventing the introduction ambiguity and enabling effective occlusion management by assigning reduced weight to occluded points during volume rendering. Moreover, we find the critical role of point-wiss wise masking, employ- ing substantially higher mask ratio of 95% compared to the 75% utilized in previous patch- wiie masking methods (16; 55).
We conduct extensive experiments on several outdoor LiDAR datasets, applying a variety of base- lines and encoders to appraise the effectiveness of our pre-training method. The experimental results consistently showcase the superiority of our approach, outperforming both training from scratch and recent concurrent point cloud pre- trainini methods. Our contributions can be summarized follows: 1) We introduce PRED, a novel vre-training framework for outdoor point clouds via semantic ren- dering. 2) We incorporate point-wise masking with a high mask ratio enhance the performance of PRED. 3) Our approach significantly bolsters the performance of various baselines on large-scale nuScenes (4) and ONCE (26) datasets across various 3D perception tasks.
2 Related Work
3D Perception on Point Clouds. 3D perception methods generally fall into two categories: point- based and voxel-based. Point-based methods (9; 24; 32; 38; 48; 57; 58) directly extract features and

---

poir cinst
Masking
Musilig
poln clou
target 2
octadde poizie
pizza
porzas bacigmmnnd
(a) Reconstruction ambiguity.
(b) Mismatch between points and pixels.
Figure 2: (a) The two point clouds the left side of 2(a) display varied levels of incompleteness. Nevertheless, after masking, these two point clouds become strikingly similar, resulting in a single input corresponding dual targets during reconstruction, thereby instigating ambiguity in training. (b) We present the projection of the LiDAR point cloud from the image view. The red dots belong the background. However, due to occlusion, these points are mistakenly projected onto the car.
predict 3D objects from raw point clouds, while voxel-based methods (11; 13; 15; 36; 37; 39; 45; 54; 60) streamline the process by discretizing these points into regular voxels or pillars for processing with 3D or 2D convolutional networks. Recently, integrating the Transformer (43) into point cloud processing has yielded notable results (27; 14; 42; 47; 49). In this work, we show that wide range of modern 3D perception methods can benefit from our pre-training framework.
Pre-Training for Point Clouds. Mask-basce (61; 30; 62; 46) and contrast-based (22; 52; 64; 23; 33) methods currently the leading paradigms for point cloud pre-training. However, these tech- niques often neglect the inherent incompleteness of outdoor point clouds. To address this, some studies (6; 8; 34) try integrate images for a more comprehensive perspective. Though impressive, they overlook potential occlusion when aligning points with images. To counter these challenges, introduce PRED, an effective pre-training paradigm that capitalizes images through semantic rendering.
Neural Rendering. The implicit representations encoded by neural network (28; 50; 29; 3; 41; 5; 56; 51) have gained a lot of attention recently, where the geometry and appearance in 3D space encoded by implicit neural representation supervised from 2D images. NeuS (50) constrains the scene space as a signed distance function and applies volume rendering to train this representation. In this work, we apply a NeuS-like rendering method, but different from learning an implicit repre- sentation of a single scene, we aim to learn generalized point cloud representations for pre-training Volumetric Rendering for Perception Tasks. Recently, some works (53; 44) have attempted to introduce volume rendering into perception tasks. Ponder (19) shares similar spirit to our work. However, Ponder focuses on indoor scenes, where point clouds often contain color information, facilitating color-based pre-training supervision. In contrast, our work addresses outdoor environ- mentsspecifically, autonomous drivingwhere point clouds are typically LiDAR-derivar and color- less. Consequently, Ponder is not applicable due to the absence of color data. In this context, we propose semantic rendering. Unlike Ponder's color-based rendering, our approach capitalizes on the semantic consistency between point clouds and images, offering distinct strategy for point cloud pre-training.
Methodology
We introduce PRED, novel image-assisted pre-training framework designed for outdoor point clouds, taking into account occlusion. The comprehensive framework is depicted in Figure 3, with subsequent sections delving into the specifics of our method.
3.1 Revisiting NeuS
NeuS (50) neural rendering technique employed for single-scene 3D reconstruction. In this approach, the 3D scene is represented by two functions: signed distance function (SDF) d R
3

---

polna unthdding
Rentering
Doonlerr
Tubreee
Key
DEV Iniiie mep
Se' Indasz
pestict prhedding
Volume Rendering
Dacceder
Ca
Projection
Bay Eotoon
Garur
all
Aosmanttc
Squaani
Cltos Bataace Saupping
SocaRiC rRag
Figure 3: Pipeline of our PRED. Firstly, we apply point-wise masking to the input point cloud, sub- sequently feeding the remaining points into the encoder to generate the BEV feature map. Semantic rendering is then performed on the BEV feature map, supervised by both semantic and depth loss. For the semantic loss, we compute the cross-entropy loss between the rendered semantics and the pseudo labels, which are predicted by an image segmenter. During pre-training, batches are sam- pled from the point cloud projections on the image plane, utilizing class balance sampling. Upon the completion of pre-training, we employ the encoder for downstream tasks.
I. which maps each spatial position r E R to its distance from the object's surface, and a color function e: R XS2 3 encoding the color of each point in the scene based on specific viewing direction v E S² Given a pixel with a ray emitted from it denoted as {r(t) o + tv t 2 0j. where o is the camera center and is the unit direction vector of the ray, NeuS computes the color for this pixel by integrating the color values along the ray using the following equation:
C(o,v)= fo w())c(((() v)dl,
(1)
where C(o, v) the pixel's output color, w(() represents weight function for the point r((). To derive an unbiased weight function w(t), NeuS computes the weight function as follows:
(t) = exp (((o()dd)) p(t), p(t) = max
de (d(r(t)) di
.0)
T (d(r(t))
where (2))) is the sigmoid function 4 (x) (1 e and is trainable parameter. NeuS is then trained by minimizing the discrepancy between rendered prediction and ground truth colors.
3.2 Pre-Training via Semantic Rendering
We apply Equations 1 and as our rendering method. simple approach would use the image's RGB values supervision for neural rendering like NeuS (50), but it failed to converge due to point cloud's color absence. Consequently, we opt for semantic supervision. Unlike the lack of color information, the point cloud contains semantic details, which can be further enhanced through the image, resulting in better convergence during training. As demonstrated in Figure 3, our proposed method employs encoder transform input point clouds into BEV feature map, and a decoder to map spatial positions in the 3D space, represented as r E IR3 to their corresponding semantic predictions and signed distances objects, conditioned on the BEV feature map.
Specifically, let P E x33 and I N. X, x3 x3 represent the input point clouds with Np points and corresponding N, images with width W; and height H;, respectively. Firstly, we encode point clouds P into BEV feature map E [ with the encoder E > F, where H. W. and C signify the height, width, and number of channels of the feature map, respectively. Subsequently, we sample batch of pixels from the images I and carry out semantic rendering for pre-training.

---

Semantic Rendering, Here, we use the pixel u = [u, e] as an example to illustrate the process. We denote the camera intrinsic and extrinsic parameters respectively by K E R and E [R, t| e R2x4, A ray emitting from the pixel u can be characterized as a line R. = - R.K [u, v,
in world coordinates parametrized by 6. We then sample a sequence of points r, = R(6,) along the
ray R. For each point, we utilize a decoder D to predict its semanties and signed distance, given by (s(r), = D(Connat(FF(r PE(r)));
(3)
where s(r) E R. and d(r) denote the semantic prediction with N₂ classes and signed distance of point r, F(r) symbolizes the feature extracted from the BEV feature map where is projected, and
PE(r) is the cosine position embedding of r. Finally, we aggregate the semantic predictions along the ray R employing Equations and 2, but in a discrete format:
$ == E II (1- (1-aa))as(Ti), a max (ox (d(())))) D2 (d(ri+))) .0); 1=1 ==1
N. i-1
(4)
T. (d(r:))
where N₂ represents the number of sampled points. Throughout this process, occluded points will be allocated low weight (50), thus mitigating the adverse effect of occlusion.
Semantic supervision. Given the absence of ground-truth semantic labels for images, we resort to DeepLabv3 (7) trained on CityScape (10) for predicting pseudo semantic labels for pixel u, denoted as S E IR Ne. Intuitively, the accuracy of s is linked to maximum value: larger maximum values indicate more confident predictions with fewer errors. Hence, to offset the impact of inaccurate labels, this maximum value is incorporated as a weight in the loss calculation. Ultimately, our loss function adopts the form of cross-entropy loss:
N. Σ s, log(s:),
Lsemantic
Σ WEZ
max (3)
(5)
where Ut denotes the training batch of pixels, which are sampled from the projections of point clouds. Additionally, considering that the semantic categories in the point cloud are heavily unbalanced with vegetation constituting more than 30% while pedestrians constitute less than 1 1% employ class- balanced sampling. Here, the sampling probability of each pixel is inversely proportional to the number of point clouds of the category which that pixel belongs (see Appendix for more details).
Geometry supervision. We introduce depth rendering to bolster the model's understanding of geo- metric information, which can enhance and stabilize pre- trainiiinn performance. For depth rendering, we adhere to the same method as outlined in Equation 4:
N. 4-1
d= (-E
(6)
i=
where 5(r:) denotes the depth of point r,, and a; is as defined in Equation 4. During the training phase, we sample batches of pixels projected from the point cloud. The ground-truth depth is deter- mined by the distance between the point and the projection plane. We opt for the view centered on the LiDAR position as the rendering view (referred to as the lidar view henceforth) given that the depth of image views may not be accurate due to occlusion, as exemplified in Figure 2(b). Finally, we supervise the rendered depth using the L2 loss function:
1
Edepth
Σ
|dgt 10-dd di2,
M
uEV
where dye represents the ground-truth depth, and V denotes the batch of pixels sampled from the projection plane of the lidar views.
Point-wise Masking, Contrary to the patch-wise masking approach utilized in masked signal mod- cling, we found that point-wise masking with a high masking ratio (95%) performs more effectively within our pre- pre-training framework. Before inputting the point cloud into the encoder, we uniformly sample a subset of the point cloud without replacement and mask the remaining points. Point-wise masking is more proficient at maintaining the semantics of the scene. For smaller objects such as pedestrians, patch-wise masking could potentially obliterate them entirely, leading to a significant loss of semantics. Conversely, moint-wise masking tends to make the point cloud of the object more sparse, thus preserving its semantics. Our experimental results further validate the effectiveness of wise masking (refer to Section 4.4 for details).
5

---

3.3 Learning Objective
Following (50), we adopt coarse-to-fine sampling strategy during rendering, utilizing 96 sample points at the coarse stage and 128 sample points at the fine stage. To optimize the model, we min- imize the sum of the semantic loss Lsemantic and the depth loss Edepth. together with the Eikonal term regg resulting in a total loss L defined as follows:
Ssemantic + Eeepth + X
(8)
where À is
that defaults to 0.1, and Creg represents the Eikonal term defined by:
N. ΣΣ (r)dl WE
Lreg
N.V
While rendering semantics, we apply the stop-gradient to the signed distance in Equation 4. This is circumvent the potential unreliability of semantic labels, which could otherwise adversely affect the learning of the scene's 's geometric information.
Table I:
Comparisons of 3D object detection performance on the nuScenes validation (top) and test (bottom) sets. We present mAP, NDS, and AP for each class. Quantitatively, our method surpasses previous state-of- the-art approaches.
Method
PreTrrin
mAP
NDS
Car
Truck
CV.
Bus
Trailer
Barrier
Motor.
Bike
Ped.
TC.
ConterPoint (60)
56.2
64,5
84.8
53.9
16.8
67.0
35.9
64,8
55.8
36.4
83.1
63.4
PoiniContrast (52)
56.3.
64.4 0.1
GCC-3D (22)
57.3+1.1
65.04a.5
ProposaContrast (59)
57.4
65.14a.
85.0
53.8
18.5
67.2
38.5
64.9
58.1
41.7
82.5
63.7
Ours (ConterPoint')
59.0.2a
66.341.8
84,9
56.4
20.3
68.6
37.1
64,5
63.2
46.6
82.8
65.2
GD-MAE (55)
58.1
65.6
85.4
56.5
16.1
70.3
36.9
64.1
59.0
39.7
84.5
68.4
58.940.
66.14a.5
85.4
56.8
18.5
70.1
38.2
64.4
60.9
39.3
84.7
70.4
X
52.2
63.5
84.7
56.0
13.8
68.4
35.3
57.7
45.8
18.9
79.1
61.9
Ours (Second)
55.2
65.5
84.8
59.2
17.1
71.1
38.4
58.1
55.0
23.2
80.1
64.8
X
61.5
68.0
85.6
60.5
20.2
71.6
37.0
66.4
64.3
49.0
86,1
74.5
Ours (CenterPoint)
/
64.2
69.7
85.6
60.8
25.0
72.8
40.2
67.3
71.8
58.5
85.8
73.9
X
87.8
64.1
25.4
41.9
69.2
74.4
Ours (DSVT)
/
68.0.1.6
72.0,a.u
87.6
64.7
27.5
75.6
46.9
73.8
74.7
61.0
88.5
79.5
PointPillars (20)
30.5
45.3
68.4
23.0
4.1
28.2
23.4
38.9
27.4
1.1
59.7
30.8
CBGS (65)
52.8
63.3
81.1
48.5
10.5
54.9
42.9
65.7
51.5
22.3
80.1
70.9
TransFusion (1)
65.5
70,2
86.2
56.7
28.2
66.3
58.8
78.2
68.3
44.2
86.1
82.0
PillarNet-34 (35)
66.0
71.4
87.6
57.5
27.9
63.6
63.1
77.2
70.1
42.3
87.3
83.3
DSVT (47)
68.4
72.7
87.9
57.6
34,9
67.0
63.3
78.3
73.1
49.7
87.9
84.2
63.3
69.1
85.22
54.6
26.4
67.8
57.1
73.6
64.6
36.0
85.7
81.9
Ours (CenterPoint)
65.9
70.8
85.0
54.5
30.2
65.5
60.9
73.5
73.2
47.9
86.0
82.2
68.4
72.7
87.9
57.6
34.9
67.0
63.3
78.3
73.1
49.7
87.9
84.2
Ours (DSVT)
70.1,1.7
73.7
87.7
58.7
38.8
68.1
65.5
79.5
76.3
53.4
87.9
84.6
Use 3D sparse convolution netwerk (60) as the point cloud encoder.
Use pixei semantics during pretraining. Notion Constrution (C.VV. pedestrian (Ped.), (TCC..
4 Experiments
4.1 Experimental Settings
Dataset: nuScenes (4) is a challenging outdoor dataset providing diverse annotations for various tasks, such as 3D object detection and BEV map segmentation. It comprises approximately 40k keyframes, each equipped with six cameras and 32-beam LiDAR scan. For 3D object detection, we report the nuScenes detection score (NDS) and mean average precision (mAP). For map segmen- tation, we provide the mean Intersection over Union (IoU).
Dataset: ONCE (26) is a large-scale autonomous driving dataset, featuring 1M LiDAR scenes and 7M corresponding camera images. The dataset split into training, validation, and testing sets con- sisting of 5k, 3k, and 8k point clouds, respectively. The remaining unannotated point clouds

---

Table 2: 3D object detection performance comparisons the ONCE val split. We report mAP and AP per class. Our method achieves the best performance among these state-of-the-ee methods.
Orientation-aware AP Pedestrian
Methods
PreTTrai
PreTruin Data
mAP
Vehicle
Cyclist
PointRCNN (38)
28.74
52.09
4.28
29.84
PointPillars (20)
44.34
68.57
17.63
46.81
SECOND (54)
51.89
71.19
26.44
58.04
PV-RCNN (36)
X
53.55
77.77
23.50
59.37
IA-SSD (63)
57.43
70.30
39.82
62.17
CenterPoint (60)
60.05
66.79
49.90
63.45
51.89
71.19
26.44
58.04
DepthContrast (64)
medium
52.81+0.94
71.92+0.73
29.01
57.51-0.53
51.89
71.19
26.44
58.04
PointContrast (52)
large
53.59, 4-.70
71.87 00.68
28.03 1.559
60.88 2.84
X
28.80
52.10
4.17
30.13
SLidR (34)
/1
large
30.72, 1.92
53.19
6.74_
32.22
64.24
75.26
51.65
65.79
ProposalComtrust (59)
large
66.32,22.08
77.2241.00
54.01
67.73
62.62
75.64
45.92
66.30
GD-MAE (55)
large 64.92+2.30
76.79+1.15
48.84+2.92
69.14 19.14-2.80
52.95
74.93
24.33
59.58
Ours (Second)
large
56.10 33.15
42.11
27.51
63.69 4.11
54.35
78.65
24.71
59.70
Ours (PV-RCNN)
large
57.45
42.34
27.93
63.44 3.74
X
64.28
49.99
66.02
small
65.87
78.56
51.50 1.61
67.55 1.53
Ours (CenterPoint)
medium
66.72
42.27
52.53
68.36
large
63.41
79.50+2.66 68.8
53,36+3.37
Use pixel semantics during pretraining.
Use super-pixel and pixel features during pretraining.
divided into three subsets: small (100k scenes), medium (500k scenes), and large (IM scenes) for pre-training purposes. The official evaluation metric the mean Average Precision (mAP). Implementation Details. We use DSVT-P (47) our default encoder, given its flexibility. For the decoder, we use a 4-layer MLP with 256 hidden channels, and Softplus chosen as the activation function. For image segmentation, During pre- pre-training, we train the model using the Adam (25) optimizer and the one-cycle policy, with a maximum learning rate of 3e- We pre-train the model for 45 epochs dataset, 20 epochs on the ONCE small, 5 epochs on the ONCE medium, and 3 epochs on the ONCE large. During fine-tuning. augmentations, with a maximum learning rate of 3e- All experiments are conducted on NVIDIA V100 GPUs. For more implementation details, please refer Appendix.
utilize DeepLab (7) with MobileNets (18) as the backbone.
the nuScenes
employ random flipping. scaling, rotation, and copy-n-paste data
Overlap of labels between pixel semantics and downstream tasks. Our pre-training phase uti- lizes pixel semantic labels that include 19 classes such as road, sidewalk. building, wall, fence, pole, traffic light, sign, vegetation, terrain, sky, person, rider, car, truck, bus, train, motorcycle, and bicy- cle. In the nuScenes object detection task, the labels include car, truck, construction vehicle, bus, trailer, barrier, motorcycle, bicycle, pedestrian, and traffic cone. For the ONCE object detection task, labels are limited vehicle, pedestrian, and cyclist. The nuScenes BEV map segmentation task uses labels such drivable, pedestrian crossing, walkway, stop line, car park, and divider. There's an overlap in labels like 'car", "bicycle', and "pedestrian', and our approach more in line with the field of weakly supervised learning.
7

---

Table 3:
Performance comparisons nuScenes val BEV map segmentation. We report mloU and IoU per class. Our method quantitatively outperforms prior works.
Method
PreTrain
mloU
Drivable
Ped. Cross.
Walkway
Stop Line
Carpark
Divider
PointPillars (20)
43.8
72.0
43.1
53.1
29.7
27.7
37.5
CenterPoint (60)
48.6
75.6
48.4
57.5
36.5
31.7
41.9
DSVT (47)
X
51.6
79.7
51.8
61.1
38.2
33.8
45.3
Ours
55.0
82.5
54.4
65.2
40.5
39.4
48.1
Table 4: Variations in Encoder Types. We maintain the same components, merely swapping the 3D backbone.
Table 5: Neural Rendering. Our rendering-based method outperforms direct projection, showcasing its effectiveness.
Encoder
Prei
Case
PreTrain
mAP
NDS
2D Conv (20)
x
50.0
60.7
baseline
X
68.0
53.0.a.
62.6,1.9
VoxelNet (60)
X
59.0,
56.2
66.3+
64.5
projection
projection-filt
62.0+0.5
62.8+1.3
68.3--0.3
68.8--0.8
rendering w/o depth
63.5+2.0
69.2--122
61.5
68.0
DSVT-P (47)
rendering w/ depth
64.2.2.7
69.7--1.7
64.2.227
69.7..17
4.2 3D Object Detection
In this section, we investigate pre-training in an autonomous driving context. We benchmark the performance of our model against previous methods on the nuScenes and ONCE datasets.
NuScenes. We employ Second (54), CenterPoint (60), and DSVT (47) as our baseline and present the performance on the validation and test sets of nuScenes in Table 1. Our pre-trained model con- sistently improves the performance of all baselines compared to models trained from scratch. On the validation set, method boosts the Second, CenterPoint, and DSVT by +3.0/2.0 mAP/NDS, +2.7/1.7 mAP/NDS, and +1.6/0.9 mAP/NDS, respectively. With CenterPoint- Voxel as the detec- tor (5-th row), our method surpasses previous pre-training methods (59; 52; 22) that also utilize CenterPoint- Voxel detector by large margin. On the test set, our method boosts the CenterPoint and DSVT by +2.6/1.7 mAP/NDS and +1.7/1.0 mAP/NDS, respectively. Moreover, our method significantly outperforms previous state-of-the-arr LiDAR detectors (20; 65; 47; 1;35).
ONCE. As shown in Table 2, our model surpasses all other methods on the ONCE validation split. When using CenterPoint (60) as the detector, our method enhances the baseline by 3.13 mAP, out- performing the improvements made by ProposalContrast (59) and GD-MAE (55) by 1.05 and 0.83, respectively. While SLidR (34) also incorporates image information during pre- pre-training. our method significantly outperforms SLidR, showcasing the effectiveness of approach in utilizing image in- formation. The distinction between the 'no-pretrain' versions of SLidR and our model originates from their foundational detection frameworks, where SLidR employs PointRCNN its detector. As the scale of pre-training data continues to increase (from small to large), our method delivers continuous gains, indicating its scalability with respect to data size.
4.3 BEV map segmentation
We also demonstrate the versatility of our approach by evaluating its performance on the BEV Map Segmentation task of the nuScenes dataset (4). We provide the IoU results for six background classes in Table 3. Leveraging our PRED, we achieve a significant improvement of 3.4% over DSVT's performance, demonstrating the effectiveness of our approach in map segmentation tasks.
4.4
Ablation Studies
In this section, we conduct a series of ablation studies on the nuScenes val to explore the essen- tial components of our methodology. Unless otherwise specified, we use CenterPoint (60) as the detector.

---

64.0
64.0
mAp
03.0
62.5
Patch-wise
Gmup-wise
62.0
Point-wip
62.0
masking ratio (%)
Figure 4: Masking strategy and masking ratio. Point-wise masking outperforms patch-wise mask- ing and group-wise masking when high masking ratio (95%) is applied.
Encoder type. To demonstrate the adaptability of our approach across various 3D backbone ar- chitectures, we experiment by substituting DSVT-P (47) with 2D Conv from PointPillar (20), and VoxelNet from CenterPoint- Voxel (60). The results, presented in Table 4, show that our approach can boost detector performance by 3.0/1.9 mAP/NDS and 2.8/1.8 mAP/NDS when employing 2D Conv and VoxelNet as backbones, respectively. These improvements are comparable to those achieved using DSVT-P. Due to DSVT-P's superior performance and flexibility, it chosen as default encoder. However, our method proves to be effective with a variety of encoder types.
Semantic rendering design. Here, we verify the efficacy of our semantic rendering pipeline as shown in Table 5, A basic approach to leveraging images is to directly project the point cloud onto the images. However, as indicated the 2nd row, this method does not yield significant improvements due to the overlooking of occlusion effects during the projection pro- cess. A potential solution is to filter the point cloud based on the image depth, thereby excluding points that do not align with the image depth from the training batch. In the absence of ground-truth image depth, we employ BEVDepth (21) for depth estimation. The enhanced performance in the 3rd row underlines the importance of handling occlusion. However, the inaccuracy of depth estimation hinders further performance improvement. Our neural rendering approach, shown in the 4th row, surpasses the prior two methods by adaptively managing occlusion, where attributing reduced weight to occluded points when accumulating semantics as in Equa- tion 4. Ultimately, incorporating depth supervision yields the best results, as depicted in the final row.
Table 6: Semantic loss. Both class balance sampling and confidence weight play an important role in cal- culation of semantic loss.
Weight
Cls-balance
mAP
NDS
62.5
68.5
69.0
63.7
69.3
64.2
69.7
Table presents ablation experiments on class balance sam- pling and confidence weight in semantic supervision. Both components are crucial for semantic supervision; the confi- dence weight mitigates the effects of inaccurate semantic la- bels, and class balance sampling addresses the issue of cate- gory imbalance in point clouds. For depth supervision, We compared two different rendering views in Table 7. The lidar view outperforms the image view, as the depth points might be inaccurate in the image view, but not in the lidar view.
Table 7: LiDAR view. For depth su- pervision, the lidar view proves su- perior to the image view.
Case
mAP
NDS
image view
63.6
69.3
lidar view
64.2
69.7
Masking strategy. In Figure 4, we explore various masking strategies investigate their impact on our pre-training framework: 1) Patch-wise masking masks a certain fraction of non-empty to- kens, with each token representing non-overlapping patch of the BEV feature map. 2) Group-wise masking masks a certain proportion of point cloud groups, with each group selected using the Fur- thest Point Sampling (FPS) and k-Nearest Neighbor (kNN) algorithm, following (61). 3) Point- wise masking, as elaborated in Section 3.2. As illustrated in Figure 4, group-wise and patch-wise wise masking strategies perform better when the masking ratio is low. When the ratio reaches 70,, the impact of group-wis wise and patch-wise masking hits its peak. Intriguingly, the effectiveness of point-wise mask- ing continues to ascend until the masking ratio reaches 95%. The results from point-wise masking surpass those of the other two methods, achieving score of 64.2 mAP. Consequently, we opt for

---

the point-wise masking strategy with 95% masking ratio, as it offers the advantage of accelerated processing while preserving good performance.
Data-efficient. A significant advantage of pre-training lies in its ability to enhance data efficiency for down- stream tasks that have limited annotated data. In this study, we examine data-efficient 3D object detection by first conducting pre-training on the nuScenes (4) training data, then fine-tuning the pre-trained model with varying fractions of training data: 10%, 20%, 50%, and 100%. The results of these experiments are illustrated in Table 8. Overall, our pre-trained model consistently boosts detection performance, especially when the available labeled data is limited, i.e., improv- ing 4.8 mAP and 3.1 NDS with only 10% labeled data. Remarkably, even when utilizing only 50% of the an- notated data, the pre-trained model achieves 63.0 mAP and 68.5 NDS, surpassing the performance of a non- pre- pre-trained version using 100% of the annotated data.
Table 8: Data Efficiency. Our approach consistently enhances the detection perfor- mance, particularly when the labeled data limited.
Dataset fraction
Pre'Train
mAP
NDS
49.6
58.9
10%
54.4, 4.8
62.0
55.1
62.9
20%
/
49.3
65.5,2.
59.7
66.2
50%
63.0 3.3
68.5 12.3
61.5
68.0
100%
64.2.2.7
69.7.11
More experimental results are provided in Appendix.
5 Conclusion
In this paper, we introduce PRED, an effective pre-training framework for outdoor point cloud data. Our framework mitigates the inherent incompleteness of point clouds by integrating image informa- tion through semantic rendering. It also allows for effective occlusion management by assigning a reduced weight to occluded points during volume rendering. A point-wise masking strategy, with mask ratio 95%, is adopted to optimize performance. Extensive experiments validate the effec- tiveness of our framework as it significantly improves various baselines the large-scale nuScenes and ONCE datasets across various 3D perception tasks. We hope our PRED can serve as a powerful baseline to inspire future research point cloud pre-training.
Limitations. PRED mainly focuses on point cloud pre-training with images as supervision. The integration of an image-point multi-modality pre-training could potentially improve performance. However, it's beyond this paper's scope and we intend to explore this possibility in our future re- search.
Acknowledgements
This work is supported by National Key R&D Program of China (2022ZD0114900) and National Science Foundation of China (NSFC62276005).
References
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Trans- fusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR, 2022. 6,
[2] Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.00 08254, 2021. 1
[3] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV,
2021. 3
[4] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krish- nan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020. 2, 6, 8,
[5] Anpei Chen, Zexiang Xu. Fuqiang Zhao. Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021. 3
[6] Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang, Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. arXiv preprint 23033 03.03.08129, 2023. 3
[7] Liang- Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolu- tion for semantic image segmentation. arXiv preprint arXiv: 2017. 5.
10

---

[8] Runnan Chen, Youquan Liu, Lingdong Kong. Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao. and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. arXiv preprint arXiv:2301.099)
2023. 2.
[9] Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points
for voting-based 3d object detection point clouds. In CVPR, 2021.
[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benen- son, Uwe Franke. Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene
understanding. In CVPR, 2016. 5
[11] Jiajun Deng. Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-can: Towards high performance voxel-based 3d object detection. In AAAI, 2021. 3
[12] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirec- tional transformers for language understanding, arXiv preprint 180.04800 2018. 1
[13] Shaocong Dong, Haiyang Wang, Tingla Xu, Xinli Xu, Jie Wang, Ziyang Bian, Ying Wang, Jianan Li, et al. Mssvt: Mixed-scale sparse voxel transformer for 3d object detection point clouds. NeurIPS, 2022. 3
[14] Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In CVPR, 2022.
[15] Tianrui Guan, Jun Wang. Shiyi Lan, Rohan Chandra, Zuxuan Wu, Larry Davis, and Dinesh Manocha. M3detr: nulti-representationn multi-scale, mutual-relation 3d object detection with transformers. In WACV, 2022. 3
[16] Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autocn-
coders are scalable vision leamers. In CVPR, 2022. 1.2
[17] Georg Hess, Johan laxing, Elias Svensson, David Hagerman, Christoffer Petersson, and Lennart Svensson. Masked autoencoder for self supervised pre-training on lidar point clouds. In WACV, 2023. [18] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 174,00660 2017.
[19] Di Huang. Sida Peng, Tong He, Honghui Yang, Xiaowei Zhou, and Wanli Ouyang. Ponder: Point cloud pre-training via neural rendcring. In /CCV, 2023. 3
[20] Alex H Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019. 6, 7, 8,9
[21] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and 7em- ing Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint
urXiv:2206. 10092. 2022. 9
[22] Hanxue Liang. Chenhan Jiang, Dapeng Feng, Xin Chen, Hang Xu, Xiaodan Liang, Wei Zhang, Zhen- guo Li, and Luc Van Gool. Exploring gcometry- aware contrast and clustering harmonization for self- supervised object detection. In ICCV, 2021. 1, 3. 6,
[23] Kangcheng Liu, Aoran Xiao, Xiaogin Zhang, Shijian Lu, and Ling Shao. Fac: 3d representation learning via foreground aware feature contrast. arXiv preprint TTii:2303306388 2023. 3
[24] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free free 3d object detection via transformers.
In ICCV, 2021.
[25] Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization. arXiv preprint
17Xiv1711.051001 2017. 7
[26] Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo i,, et al. One million scenes for autonomous driving: Once dataset.
arXiv preprint arXiv: 2066 11077 2021. 2, 6
[27] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai. Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing
Xu. Voxel transformer for 3d object detection. In /CCV, 2021. 3
[28] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 2021. 2,
[29] Thomas Müller, Alex Evans, Christoph Sehied, and Alexander Keller. Instant neural graphics primitives
with a multiresolution hash encoding. ACM Trans. Graph., 2022. 3
[30] Yatian Pang, Wenxiao Wang. Francis EH Tay. Wei Liu, Yonghong Tian, and Yuan. Masked autoen- coders for point cloud self-supervised learning. In ECCV, 2022. 1,
[31] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d 3d using 2d diffu-
sion. arXiv preprint arXiv:2209. 14988, 2022. 2
[32] Charles R Qi. Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection
in point clouds. In /CCV, 2019. 2
[33] Yongming Rao, Benlin Liu, Yi Jiwen Lu, Cho-Jui Hsich, and Jie Zhou. Randomrooms: unsupervised
pre-training from synthetic shapes and randomized layouts for 3d object detection. In /CCV, 2021. 3
[34] Corentin Sautier. Gilles Puy. Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet.
Image-to- to-daar self-supervised distillation for autonomous driving data. In CVPR, 2022. 1, 2. 3, 7.8
[35] Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillamet: Real-time and high-performance pillar-hased 3d
object detection. In ECCV, 2022. 6, 8
[36] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.
Pv-renn: voxnt-voxel feature set abstraction for 3d object detection, In CVPR, 2020. 3,7 7
11

---

[37] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hong- sheng Li. Pv-renn++: Point-voxel feature set abstraction with local vector representation for 3d object
detection, I/CV, 2023. 3
[38] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detec-
tion from point cloud. CVPR, 2019. 2.
[39] Shaoshuai Shi. Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. TPAMI, 2020. 3
[40] Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin- Ying Lee, Peter Wonka, and Sergey
Tulyakov. 3d generation on imagenet. arXiv preprint TrXiiv2303.014/0 2023. 2
[41] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering. In
CVPR, 2022. 3
[42] Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV, 2022. 3 [43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 30, 2017. 3
[44] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic
segmentation of 3d scenes. arXiv preprint arXiv:2111.33266 2021. 3
[45]
Haiyang Wang. Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang, et al. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. NeurlPS, 2022.
[46]
Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud
pre- trainini via occlusion complction. In /CCV, 2021. 3
[47] Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang. Di He, Bernt Schiele, and Liwei Wang.
Devt: Dynamic sparse voxel transformer with rotated sets. arXiv preprint orXiv:2001006051, 2023. 3, 6, 7, 8, 9
[48]
Haiyang Wang, Shaoshuai Shi. Ze Yang. Rongyao Fang. Qi Qian, Hongsheng Li, Bernt Schiele. and
Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. In CVPR, 2022. 2
[49]
Haiyang Wang. Hao Tang, Shaoshuai Shi, Aoxue Li, Zhenguo Li, Bernt Schiele, and Liwei Wang. Unitr:
A unified and efficient multi-modal transformer for bird s-eye-view representation, In ICCV, 2023. 3
[50] Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: l.crming neural implicit surfaces hy volume rendering for multi-view reconstruction. arXiv preprint arXiv: 10w. 10689, 2021. 3, 4, 5,
[51] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron. Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-
based rendering. In CVPR, 2021. 3
[52] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsu- pervised pre-training for 3d point cloud understanding. In ECCV, 2020. 1, 7,8
[53] Chenfeng Xu, Bichen Wu. Ji Hou, Sam Tsai, Ruilong Li, Jialiang Wang. Wei Zhan, Zijian He, Peter Vajda, Kurt Keutzer, al. Nerf-det: Learning geometry-aware volumetric representation for multi-view
3d object detection. In ICCV, 2023. 3
[54] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018.
3, 77
[55] Honghui Yang, Tong He, Jiaheng Liu, Hua Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wanli Ouyang. Gd-mae: Generative decoder for mae pre-training on lidar point clouds. arXiv preprint arXiv:22/2.0.30
2022. 1, 2, 6, 7
[56] Hao Yang. Lanqing Hong, Aoxue Li, Tianyang Hu, Zhenguo Li, Gim Hee Lee, and Liwei Wang. Contran- erf: Generalizable neural radiance fields for toathetic-to-rall novel view synthesis via contrastive learning.
In CVPR, 2023. 3
[57] Hao Yang, Chen Shi, Yihong Chen, and Liwei Wang. Boosting 3d object detection via object-focused
image fusion. arXiv preprint arXiv:2207. 200.:2990.,,,, 2022. 2
[58] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In
CVPR, 2020. 2
[59] Junbo Yin, Dingfu Zhou, Liangjun Zhang, Jin Fang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang. Proposalcontrast: Unsupervised pre-training for lidar-based 3d object detection. ECCV, 2022.
1. 7,8
[60]
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center based 3d object detection and tracking. In CVPR, 2021. 3, 7, 7,8,9
[61]
Xumin Yu. Lulu Tang, Yongming Rao, Tiejun Huang. Jie Zhou, and Jiwen Lu. Point-bert: Pre-training
3d point cloud transformers with masked point modeling, In CVPR, 2022. 1, 9
[62]
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li.
Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. arXiv preprint
arXiv:2205.14401, 2022. 1,3
[63] Yifan Zhang. Qingyong Hu, Guoquan Xu, Yanxin Ma. Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-based based detectors for 3d lidar point clouds. In CVPR, 2022.
[64] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features
on any point-cloud, In /CCV, 2021. 1, 7.7
12

---

[65] Benjin Zhu, Zhengkai Jiang. Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv 09492, 2019. 6, 8
13


Reviews:
[{"review_id": "Pq3CcamyG6", "reviewer": "Reviewer_4EsX", "summary": "1. The paper proposes PRED, a novel pre-training framework for outdoor point clouds that leverages image semantics through neural rendering. The paper addresses the challenges of incompleteness and occlusion in point clouds, which are common in outdoor LiDAR datasets for autonomous driving.\n2. The paper uses an encoder-decoder architecture to extract a BEV feature map from the point cloud and render semantic maps from image views, supervised by image segmentation and depth estimation.\n3. The paper incorporates point-wise masking with a high mask ratio (95%) to enhance the pre-training performance and avoid losing semantics of small objects.\n4. The paper conducts extensive experiments on nuScenes and ONCE datasets, showing that PRED significantly improves various baselines and state-of-the-art methods on 3D object detection and BEV map segmentation tasks.", "questions": "Please see the comments on 'weaknesses'.", "limitations": "Please see the comments on 'weaknesses'.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel pre-training framework for outdoor point clouds that integrates image semantics through neural rendering. This is a creative and effective way to address the incompleteness and occlusion issues in point clouds, which are often overlooked by previous pre-training methods. The paper also introduces point-wise masking with a high mask ratio, which is different from the conventional patch-wise or group-wise masking strategies and preserves more semantics of small objects.\n2. The paper is technically sound and well-motivated. The paper provides a clear and detailed description of the proposed method, including the encoder-decoder architecture, the semantic rendering pipeline, the loss functions, and the masking strategy. The paper also conducts extensive experiments on two large-scale datasets, nuScenes and ONCE, and compares with various baselines and state-of-the-art methods on 3D object detection and BEV map segmentation tasks. The paper reports significant improvements over previous methods, demonstrating the effectiveness and generality of the proposed framework.", "weaknesses": "1. The paper does not conduct ablation studies on the choice of image segmentation model and its impact on the pre-training performance. The paper uses DeepLabv3 as the image segmenter but does not justify or evaluate this choice. It is unclear how the quality and accuracy of the image segmentation model affect the semantic rendering and supervision.\n2. This paper does not report the computational cost or time complexity of the pre-training framework. Semantic rendering involves sampling and aggregating points along multiple rays for each pixel, which may be computationally expensive and memory-intensive. It would be helpful to provide some statistics or benchmarks on the pre-training speed and resource consumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're gratified by your acknowledgment of our approach's novelty and efficacy. We also value the concerns you've raised and here's our detailed response:\n\n**Q1: Evaluating the Impact of Image Segmentation Model Choices on Pre-training Performance.**\n\n**A1:** We acknowledge the importance of examining the choice of image segmentation model. To this end, we conducted ablation studies involving a range of renowned segmentation models, namely PSPNet [58], DeepLabv3 (MobileNet), DeepLabv3 (ResNet101), and SegFormer [59]. Keeping other settings consistent with Section 4.4, our results (as presented below) demonstrated robustness in pre-training performance across these models. DeepLabv3 ultimately emerged as our preferred choice, owing to its impressive performance and user-friendliness. These results will be incorporated into our revision.\n\n\n| Image Segmenter           | PreTrain | mAP      | NDS      |\n|:-------------------------:|:--------:|:--------:|:--------:|\n| baseline                  | ❌       | 61.5     | 68.0     |\n| PSPNet                    | ✔️       | 63.9$_{+2.4}$ | 69.5$_{+1.5}$ |\n| DeepLabv3 (MobileNet)     | ✔️       | 64.2$_{+2.7}$ | 69.7$_{+1.7}$ |\n| DeepLabv3 (ResNet101)     | ✔️       | 64.4$_{+2.9}$ | 69.7$_{+1.7}$ |\n| SegFormer                 | ✔️       | 64.3$_{+2.8}$ | 69.9$_{+1.9}$ |\n\n[58] PSPNet: Pyramid Scene Parsing Network.\\\n[59] SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.\n\n---\n\n**Q2: Computational cost or time complexity of the pre-training framework.**\n\n**A2:** Sorry for overlooking these details earlier, we will incorporate this information into the revised version. For our pre-training scheme, we utilized eight V100 GPUs, each equipped with 32GB memory. Abiding by the configurations detailed in Section 4.1 and Appendix A, and capitalizing on mixed precision training, each GPU consumed approximately 28GB to 30GB of memory. A complete pre-training exercise spanning 45 epochs on the nuScenes dataset was accomplished in around 32 hours. Additionally, training across the ONCE datasets—20 epochs for the small variant, 5 for the medium, and 3 for the large—required approximately 30, 38, and 45 hours, respectively. Since we only sample 768 pixels per scene as a training batch in each iteration, the overall training time and resource consumption are acceptable."}, {"review_id": "ns6KdlsSB5", "reviewer": "Reviewer_4GhG", "summary": "This work proposes a new pretraining algorithm for outdoor 3D perception tasks, where images are utilized to provide comprehensive semantic information. The main algorithm is to leverage the semantics of images for supervision through neural rendering. The authors also apply point-wise masking with a high mask ratio to further enhance performance. The pretraining brings notable performance gains on multiple benchmarks.", "questions": "Please refer to the weakness part, I would raise my score if the two concerns can be addressed.", "limitations": "Please refer above.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors provide a novel insight into the exploitation of image semantics by combing off-the-shelf image segmenter and neural rendering, which I think is well-motivated. \n2.\tExtensive experiments on multiple benchmarks demonstrate the effectiveness of the proposed method.\n3.\tThe paper is well-written and easy to follow. The illustrations are straightforward and helpful.", "weaknesses": "1. I acknowledge the motivation and the method design of this paper. Different from neural rendering for RGB as NeRF does, the high-level idea of rendering semantics from point-cloud is quite novel. However, I have heavy concerns about the use of Cityscape pre-trained segmenter as it is not general enough, e.g., Cityscape pre-trained segmenter can perform well on nuscenes and Once but can't on Waymo. Despite the authors explaining the reason Waymo experiments do not perform that well on Line101 of the appendix, I think the poor generalizability of pre-trained segmenter maybe also a reason. So I suggest replacing the pre-trained segmenter with SAM.\n\n2. This work shares a similar spirit to Ponder [1]. Despite [1] for indoor scenes, the paper aims at outdoor scenes, the authors should carefully discuss the intrinsic differences. \n\n[1] Ponder: point-cloud pre-training via neural rendering.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback regarding the motivation and results of our research. We understand your concerns about certain aspects of our paper and would like to provide some clarification.\n\n**Q1: On the selection of the pre-trained segmenter.**\n\n**A1:** We will definitely explore the possibility of replacing the segmenter with SAM. The selection of the pre-trained segmenter on Cityscape was motivated by its contextual relevance to our experimental datasets—all involve autonomous driving scenarios. The diverse array of scenes within Cityscape aligns with those found in other datasets, thus enhancing the segmenter's ability to generalize across a range of autonomous driving datasets. \n\nIn the case of Waymo, its point cloud exhibits greater density compared to other datasets, attributed to the utilization of 64-beam LiDAR scanning, whereas datasets like nuScenes employ 32-beam LiDARs. Moreover, Waymo only includes images captured from front and side views, whereas images in other datasets encompass a wider range of perspectives. As a result, the utility of these images in enhancing point cloud downstream tasks, such as object detection, remains limited on the Waymo dataset.  This observation also provides an explanation for the relatively modest improvements achieved through image-derived semantic information during the pre-training phase.\n\nNonetheless, we agree with you that SAM might exhibit superior generalization performance compared to the segmenter trained on Cityscape. While SAM exclusively predicts object masks without providing semantic labels, an intriguing strategy could involve substituting semantic rendering with semantic feature rendering. In this scenario, SAM could be employed to extract semantic features from images to serve as the supervision. This endeavor indeed holds great promise, and we deeply value your insightful suggestion. In future work, we will consider employing Vision Foundation Models like SAM to enhance the generalizability of our pre-training framework.\n\n---\n\n**Q2: The intrinsic differences between our work and Ponder.**\n\n**A2:** Several fundamental differences indeed exist between our work and Ponder, and it's crucial to emphasize these. \n\n- First and foremost, the application domains differ: Ponder focuses on indoor scenes, where point clouds often contain color information, facilitating color-based pre-training supervision. In contrast, our work addresses outdoor environments—specifically, autonomous driving—where point clouds are typically LiDAR-derived and colorless. Consequently, Ponder is not applicable due to the absence of color data. In this context, we propose semantic rendering. Unlike Ponder's color-based rendering, our approach capitalizes on the semantic consistency between point clouds and images, offering a distinct strategy for point cloud pre-training.\n- Second, the pixel sampling strategies differ: Unlike the one-to-one point cloud-to-pixel correspondence found in densely scanned indoor point clouds, outdoor point clouds exhibit only partial semantic information from the image due to their sparsity. Furthermore, the semantic imbalance prevails within the point clouds. Our method addresses these challenges by sampling pixels projected from point clouds with a class-balanced strategy. This stands in contrast to Ponder's approach of random sampling from images.\n- Lastly, Ponder employs a 3D voxelized feature volume for rendering, whereas we favor a bird's eye view (BEV) feature-based rendering strategy better suited to the unique traits of outdoor settings.\n\nWe will discuss Ponder in our revision."}, {"review_id": "dXpsQGNEoR", "reviewer": "Reviewer_8P8s", "summary": "This paper proposed a novel point cloud pre-training framework, PRED, which leverages the semantic information consistency between the LiDAR point clouds and the camera images to improve the point cloud pre-training performance. The author proposed (1) a novel semantic rendering module for decoding the semantics from the BEV feature maps and (2) a point-wise masking mechanism to alleviate the reconstruction ambiguity.\n\nThe proposed pre-training method demonstrates superior performance according to the experiments section.", "questions": "(1), The proposed method relies on well-trained 2D segmentation models, which limits its generalization ability.", "limitations": "(1) Is there anything special or novel for handling occlusion compared to [46]? If not, then this point should not be highlighted.\n(2) How would the proposed method perform if the 2D segmenter fails?", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1), To the best of my knowledge, the proposed point cloud pre-training framework with semantic rendering is novel and reasonable.\n(2), point cloud pre-training is an important task for academia and industry.\n(3), According to the experiment section, the proposed framework PRED has achieved superior performance on multiple benchmarks.", "weaknesses": "(1), Dealing with occlusion is claimed as one of this paper's major contributions and advantages. However, why the occluded points will be allocated a lower weight is not illustrated clearly in the paper. The author could add more presentation, analysis, visualization, and evaluation. \n(2), The pre-trained semantic model is required for the proposed approach. This point should be marked and compared with other methods in Tables 1 and 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The recognition of our approach's potential significance in both academic and industrial circles is particularly encouraging. We acknowledge the concerns you've highlighted and would like to offer clarifications:\n\n**Q1: Is there anything special or novel for handling occlusion compared to [46]? Why the occluded points will be allocated a lower weight?**\n\n**A1:** Sorry for any confusion. The key point we aim to emphasize in our paper is the importance of addressing occlusion during point cloud pre-training, a factor that has been overlooked in prior works. In this regard, we apply neural rendering, drawing inspiration from and aligning with [46], to effectively handle occlusion with encouraging results. \n\nThe allocation of a lower weight to occluded points stems from our volume rendering computation, as outlined in Equation 2. For two depth values $t_1$ and $t_2$ that satisfy $d(t_1) = d(t_2)$ and $t_1 < t_2$, we have \n$$w(t_1) - w(t_2) = \\exp \\left(-\\int_0^{t_1} \\rho(u) \\mathrm{d} u\\right) \\rho(t_1) - \\exp \\left(-\\int_0^{t_2} \\rho(u) \\mathrm{d} u\\right) \\rho(t_2) = (\\exp \\left(-\\int_0^{t_1} \\rho(u) \\mathrm{d} u\\right) - \\exp \\left(-\\int_0^{t_2} \\rho(u) \\mathrm{d} u\\right)) \\rho(t_1) > 0.$$\nThis indicates that, for points with equivalent SDF values, those proximal to the viewpoint are awarded greater weight, resulting in occluded points receiving reduced weight during rendering. We will clarify this in the revised manuscript.\n\n---\n\n**Q2: The pre-trained semantic model is required for the proposed approach, which should be marked and compared with other methods in Tables 1 and 2.**\n\n**A2:** Your suggestion is well-received. We will address this point in Tables 1 and 2.\n\nHere are rough versions of how the updated tables might look:\n\nTable 1:\n\n| Method           | PreTrain | Pixel Signal Used | mAP      | NDS      |\n|:----------------:|:--------:|:-----------------:|:--------:|:--------:|\n| CenterPoint      | ❌       | -                 | 56.2     | 64.5     |\n| PointContrast    | ✔️       | -                 | 56.3$_{+0.1}$ | 64.4$_{−0.1}$ |\n| GCC-3D           | ✔️       | -                 | 57.3$_{+1.1}$ | 65.0$_{+0.5}$ |\n| ProposalContrast | ✔️       | -                 | 57.4$_{+1.2}$ | 65.1$_{+0.6}$ |\n| GD-MAE           | ❌       | -                 | 58.1     | 65.6     |\n| GD-MAE           | ✔️       | -                 | 58.9$_{+0.8}$ | 66.1$_{+0.5}$ |\n| Ours (CenterPoint) | ❌       | -   | 61.5 | 68.0 |\n| Ours (CenterPoint) | ✔️       | pixel semantics   | 64.2$_{+2.7}$ | 69.7$_{+1.7}$ |\n\n\nTable 2:\n\n| Method              | PreTrain | Pixel Signal Used        | mAP      |\n|:-------------------:|:--------:|:------------------------:|:--------:|\n| PointRCNN           | ❌       | -                        | 28.74    |\n| SECOND              | ❌       | -                        | 51.89    |\n| CenterPoint         | ❌       | -                        | 60.05    |\n| PointContrast       | ❌       | -                        | 51.89    |\n| PointContrast       | ✔️       | -                        | 53.59$_{+1.70}$ |\n| SLidR               | ❌       | - | 28.80    |\n| SLidR               | ✔️       | super-pixel & pixel feature | 30.72$_{+1.92}$ |\n| ProposalContrast    | ❌       | -                        | 64.24    |\n| ProposalContrast    | ✔️       | -                        | 66.32$_{+2.08}$ |\n| GD-MAE              | ❌       | -                        | 62.62    |\n| GD-MAE              | ✔️       | -                        | 64.92$_{+2.30}$ |\n| Ours (CenterPoint)  | ❌       | -           | 64.28    |\n| Ours (CenterPoint)  | ✔️       | pixel semantics           | 67.41$_{+3.13}$ |\n\nWe hope these amendments will help readers better understand the prerequisites of our method and its comparison with others.\n\n---\n\n**Q3: The generalizability of using 2D segmentation models.**\n\n**A3:** Our method leans on well-trained 2D segmentation models. However, given their widespread adoption spanning a diverse spectrum of applications, we believe this reliance is justified. \n\nNotably, with the recent advancements in Vision Foundation Models (VFMs), we envision leveraging these VFMs to further enhance our method's generalization capabilities. For instance, SAM [60], a current popular segmenter known for its strong generalization performance, could be integrated into our framework. While SAM's predictions exclude semantic labels, its semantic features present an attractive supervisory signal, marking an exciting trajectory for our ensuing endeavors.\n\nMoreover, our method has consistently demonstrated its efficacy across a range of datasets, benchmarks, and tasks, underscoring the inherent robustness ingrained within our approach. Nevertheless, we will try to explore the feasibility of utilizing VFMs to further enhance the generalization ability.\n\n[60] SAM: Segment Anything.\n\n---\n\n**Q4: Implications of potential 2D segmenter failures.**\n\n**A4:** In instances where the 2D segmenter fails to accurately predict semantics, a potential impact on the quality of pre-training arises due to the methodology's reliance on 2D semantic labels. However, our extensive empirical evaluations have consistently demonstrated the segmenter's resilience across a diverse array of datasets. This robustness can be attributed to its foundation on the Cityscape dataset, which includes scenes similar to those present in other datasets.\n\nNonetheless, we acknowledge there might be situations where the 2D segmenter could be unreliable. In such cases, our approach incorporates the maximum prediction score as a weighting factor within our loss function, as detailed in Lines 127-129 of our methodology section. This weighting scheme effectively assigns reduced significance to potentially erroneous semantic labels, thereby mitigating the potential consequences of segmentation errors on the overall pre-training process."}, {"review_id": "aPlbnnGLsJ", "reviewer": "Reviewer_73N2", "summary": "This paper investigates weakly-supervised representation learning for outdoor LiDAR point clouds. To start, the authors point out that the inherent incompleteness of outdoor LiDAR points would reduce the effectiveness of self-supervised representation learning approaches. To mitigate this, the authors propose to use synchronized images as additional signals to supervise the representation learning process. Observing the lack of color information in the point cloud and the mismatch between points and pixels due to occlusion, the authors propose to use neural rendering of pixel semantics as a pre-text task to circumvent the two problems. Specifically, with a slight modification to NEUS’s weighting function, the authors build an implicit neural representation on top of the BEV LiDAR features, positing that good BEV LiDAR features for implicit neural representations could be good representations for any downstream recognition tasks. Extensive experiments have been conducted to show the effectiveness of the approach.", "questions": "The reviewer believes this is a good paper. However, the reviewer thinks it is important to properly address the weaknesses (especially on the SLiDR baseline and presentation 2). For pre-rebuttal, the reviewer would give “borderline reject” to implore the authors to properly address the weaknesses. If the weaknesses are sufficiently addressed, the reviewer is more than happy to give this submission a “strong accept”. \n\nSuggestions:\n- To fix the first weakness in the presentation, the reviewer recommends adding a column in table 1 and 2 to indicate whether and the type of pixel signal used. Also, it would be great to emphasize this in section 4.2.\n- To fix the second weakness in the presentation, the reviewer recommends adding a paragraph indicating label overlap between the pixel semantics and downstream tasks in section 4.1.\n- For the baseline, the reviewer would like to know why the “no-pretrain” SLiDR baseline is much worse than the \"no-pretrain\" OURS baseline is table 2.", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- Novelty - Using neural rendering as a pre-text task for point cloud representation learning is a novel idea the reviewer has never seen in the literature. \n- Beautiful figures - Figure 1,2,3 are aesthetically pleasing, with figure 2 clearly illustrating the core problems in representation learning for outdoor LiDAR point clouds. \n- Strong performance improvement over baselines that do not use images as additional signals. \n- Thorough experiments - Experiments thoroughly demonstrate the strength of the approach.", "weaknesses": "Overall, this is a strong paper but the reviewer has to point out three weaknesses — two related to presentation and the other related to baselines. \n\n- (Presentation 1) One of the contributions of the paper is to show that images could be valuable signals for pre-training representation. However, it is unclear which baselines in table 1 and 2 actually use additional images as signals. Also, it is unclear what type of signals are being used (i.e. raw pixel values vs pixel semantics). \n- (Baseline) Another contribution of the paper is to show that pixel semantics is a useful signal for pre-training representation. Although the proposed approach did not converge when trying to render raw pixel value (line 101-103), SLidR (table 2) is a baseline that actually leverages color information to form superpixel for pre-training. Given the big difference between the no-pretrain variants for SLidR and ours, it is difficult to judge whether pixel-value is a weaker signal or there is some other difference (such as model architecture or optimization recipes) that is causing SLidR to underperform PRED. \n- (Presentation 2): Since using pixel semantics is part of the core contributions, it is important for the authors to acknowledge/mention whether there are any overlaps between the pixel semantic label space and the downstream task label space. Without this, it is difficult to tell whether the proposed approach is a self-supervised approach or a weakly-supervised approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are encouraged by your acknowledgment of our work's novelty and thoroughness in experimentation. We aim to address the concerns raised to offer clearer insights into our methodology.\n\n**Q1: Clarity on image signals in Tables 1 and 2.**\n\n**A1:** To offer greater clarity, we will introduce a dedicated column in Tables 1 and 2 that indicates which baselines utilize image signals and the specific type of those signals - be it raw pixel values or pixel semantics.\n\nHere are rough versions of how the updated tables might look:\n\nTable 1:\n\n| Method           | PreTrain | Pixel Signal Used | mAP      | NDS      |\n|:----------------:|:--------:|:-----------------:|:--------:|:--------:|\n| CenterPoint      | ❌       | -                 | 56.2     | 64.5     |\n| PointContrast    | ✔️       | -                 | 56.3$_{+0.1}$ | 64.4$_{−0.1}$ |\n| GCC-3D           | ✔️       | -                 | 57.3$_{+1.1}$ | 65.0$_{+0.5}$ |\n| ProposalContrast | ✔️       | -                 | 57.4$_{+1.2}$ | 65.1$_{+0.6}$ |\n| GD-MAE           | ❌       | -                 | 58.1     | 65.6     |\n| GD-MAE           | ✔️       | -                 | 58.9$_{+0.8}$ | 66.1$_{+0.5}$ |\n| Ours (CenterPoint) | ❌       | -   | 61.5 | 68.0 |\n| Ours (CenterPoint) | ✔️       | pixel semantics   | 64.2$_{+2.7}$ | 69.7$_{+1.7}$ |\n\n\nTable 2:\n\n| Method              | PreTrain | Pixel Signal Used        | mAP      |\n|:-------------------:|:--------:|:------------------------:|:--------:|\n| PointRCNN           | ❌       | -                        | 28.74    |\n| SECOND              | ❌       | -                        | 51.89    |\n| CenterPoint         | ❌       | -                        | 60.05    |\n| PointContrast       | ❌       | -                        | 51.89    |\n| PointContrast       | ✔️       | -                        | 53.59$_{+1.70}$ |\n| SLidR               | ❌       | - | 28.80    |\n| SLidR               | ✔️       | super-pixel & pixel feature | 30.72$_{+1.92}$ |\n| ProposalContrast    | ❌       | -                        | 64.24    |\n| ProposalContrast    | ✔️       | -                        | 66.32$_{+2.08}$ |\n| GD-MAE              | ❌       | -                        | 62.62    |\n| GD-MAE              | ✔️       | -                        | 64.92$_{+2.30}$ |\n| Ours (CenterPoint)  | ❌       | -           | 64.28    |\n| Ours (CenterPoint)  | ✔️       | pixel semantics           | 67.41$_{+3.13}$ |\n\nAdditionally, we'll underscore the significance of harnessing image signals in Section 4.2, elaborating how distinct pixel signals like raw pixel values and pixel semantics can profoundly affect model efficacy.\n\n---\n\n**Q2: Performance discrepancy between 'no-pretrain' SLiDR and OURS baselines in Table 2.**\n\n**A2:** The distinction between the 'no-pretrain' versions of SLidR and our model originates from their foundational detection frameworks. SLidR employs PointRCNN as its detector, a methodology grounded in point-based detection. In contrast, our approach is designed for a BEV-based detector, which typically yields a superior baseline performance. Even if our baseline has less room for improvement, the boost our method offers over the baseline surpasses that of SLidR. For instance, our method (CenterPoint) achieves a remarkable +3.13mAP improvement compared to SLidR's +1.92mAP gain.\n\nOne more clarification, SLidR's methodology extends beyond the utilization of color information; it incorporates image features for point cloud contrastive learning supervision. These image features are extracted through a pre-trained ResNet employing MoCov2. In contrast, our strategy only uses pixel semantics.\n\nWe will clarify these points in the revised version of our paper.\n\n---\n\n**Q3: Overlap of labels between pixel semantics and downstream tasks.**\n\n**A3:** In our revised manuscript's Section 4.1, we will incorporate a detailed overview indicating the label overlaps between pixel semantics used in our pre-training and those of our downstream tasks.\n\nHere is a preliminary look at the label overlap:\n\n1. Pixel Semantics: Our pre-training phase utilizes pixel semantic labels that include 19 classes such as road, sidewalk, building, wall, fence, pole, traffic light, sign, vegetation, terrain, sky, person, rider, car, truck, bus, train, motorcycle, and bicycle.\n2. Downstream tasks:\n   - In the nuScenes object detection task, the labels include car, truck, construction vehicle, bus, trailer, barrier, motorcycle, bicycle, pedestrian, and traffic cone.\n   - For the ONCE object detection task, the labels are limited to vehicle, pedestrian, and cyclist.\n   - The nuScenes BEV map segmentation task uses labels such as drivable, pedestrian crossing, walkway, stop line, car park, and divider.\n\nThere's an overlap in labels like 'car', 'bicycle', and 'pedestrian', and our approach is more in line with the field of weakly supervised learning.\n\nFurthermore, we have the prospect of integrating Vision Foundation Models (VFMs), like SAM [60], into our framework, capitalizing on their semantic features as the supervision. These models exhibit enhanced generalization capabilities, which could potentially result in further performance improvements for our method. However, given the relatively brief existence of VFMs, there remains some ambiguity regarding whether pre-training with VFMs falls within the realm of self-supervised or weakly-supervised methods. We will discuss these in the revision.\n\n[60] SAM: Segment Anything."}, {"review_id": "oImhEAv5z7", "reviewer": "Reviewer_sbFA", "summary": "This work incorporates images into point cloud pretraining since images contain richer semantic information. Instead of adopting the back propagation strategy which can not handle the misalignment between camera and LiDAR, this paper leverages the neural rendering technique to injecting the semantics into the representation learning process.", "questions": "My major concern lies in the superiority of this rendering process compared to the point painting process. While the author address the occlusion issue, I firmly believe that it can be effectively resolved by accurately determining the object boundaries. By precisely locating the boundaries of objects, the occlusion challenge can be easily mitigated, potentially diminishing the need for a complex rendering approach.", "limitations": "The author address the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "(1) This paper is well-written and presents a clear and concise idea.\n(2) The motivation of leveraging the neural rendering to overcome the mismatch of pixel and point is compelling.\n(3) The experimental results demonstrate consistent improvement on different datasets with varying baselines. And the ablation studies validate the contribution of each component in the pipeline.", "weaknesses": "(1) The technical contribution is limited. For example, the BEV conditioned semantic neural rendering strategy, and the masking strategy have been studied in previous works.\n(2) Although the occlusion problem can be alleviated by assigning a reduced weight to occluded points, the rendering process will introduce additional noise since it assigns multiple semantic labels to each point, leading to the semantic ambiguity.\n(3) The rendering process allow every points in the ray receive the gradients which leads to many irrelevant points being optimized.\n(4) The authors claims that the neural rendering is superior to the point-to-pixel projection, but there is no performance comparison between these two methods in the experiments. I think the latter may also bring an obvious improvement since the inambiguous projections.\n(5) This work takes the semantic labels as supervision for neural rendering. I think it would be better to use the 2D feature vectors as the supervision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback regarding the paper's clarity, motivation, and experimental results. We understand your concerns and would like to address your points one by one.\n\n**Q1: Concerns regarding technical contribution.**\n\n**A1:** Our main contribution lies in a novel pre-training framework for outdoor point clouds. By combining neural rendering with point-wise masking and using 2D semantic labels as supervision, we provide a comprehensive solution that effectively addresses the challenges of reconstruction ambiguity and occlusion in point cloud pre-training. The effectiveness of our approach has been demonstrated through improved performance on several downstream tasks. Moreover, to the best of our knowledge, we are the first to introduce a BEV-conditioned semantic neural rendering strategy for point cloud pre-training.\n\n---\n\n**Q2: Semantic ambiguity arising from the rendering process.**\n\n**A2:** Assigning multiple labels to a single point can be a source of noise. However, our dataset's emphasis on autonomous driving scenarios mitigates this concern. Given the car-centric viewpoint of our images, overlaps between different perspectives remain minimal. For instance, in the nuScenes dataset, the overlap between two views is less than 10%. Consequently, multi-label rendering affects a limited set of points. Future research can further mitigate this by selectively sampling non-overlapping regions, a refinement we'll underscore in our revision.\n\n---\n\n**Q3: The rendering process allows every point in the ray to receive the gradients which leads to many irrelevant points being optimized.**\n\n**A3:** The rendering process allows for the geometric understanding of the scene. When rendering semantics, we apply the stop-gradient to the signed distance as explained in Line 162. This means that only a small number of significant points with larger weights are primarily optimized by semantic loss. In our depth rendering, while we don't curtail gradients to seemingly 'irrelevant' points, these contribute substantially to the model's geometric understanding of the scene. This geometry perception proves pivotal for downstream tasks, like object detection.\n\n---\n\n**Q4: Contrasting neural rendering with point-to-pixel projection.**\n\n**A4:** Sorry for any oversight. Kindly direct your attention to Table 5 for a comprehensive comparison between neural rendering and point-to-pixel projection. The results illustrate the substantial superiority of our neural rendering approach over the point-to-pixel projection method.\n\n---\n\n**Q5: Embracing 2D feature vectors as supervision.**\n\n**A5:** We have explored the option of supervising the model with feature vectors extracted from the segmenter's backbone. Through comparative results (shown below) under the experimental settings as detailed in Section 4.4, we find that both supervision signals are effective.\n\n| Supervision       | PreTrain | mAP      | NDS      |\n|:-----------------:|:--------:|:--------:|:--------:|\n| baseline          | ❌       | 61.5     | 68.0     |\n| semantic label    | ✔️       | 64.2$_{+2.7}$ | 69.7$_{+1.7}$ |\n| feature vector    | ✔️       | 64.3$_{+2.8}$ | 69.4$_{+1.4}$ |\n\nWe ultimately chose to use supervision from semantic labels as it simplifies the computation of the loss function. Specifically, semantic labels are utilized for class-balanced sampling and loss weights, as outlined in Lines 128-135. \n\nNonetheless, the potential of 2D feature vectors remains promise, particularly when sourced from advanced Vision Foundation Models such as SAM [60]. Even though SAM doesn't predict semantic labels, its impressive generalization capabilities could enhance the effectiveness of our method—a prospect worth deeper exploration.\n\n[60] SAM: Segment Anything.\n\n---\n\n**Q6: Comparing neural rendering to the point painting process.**\n\n**A6:** The idea of utilizing accurate object boundaries to address occlusion challenges is indeed intriguing. However, in the context of autonomous driving, which involves sparse and incomplete point clouds, accurately determining boundaries becomes challenging due to these constraints. Furthermore, this methodology would require pre-processing object segmentation and 3D boundary computations, potentially increasing system complexity. While we value your perspective, our research leverages neural rendering, effectively managing occlusion without resorting to explicit boundary detection. Nevertheless, exploiting the inductive bias of point clouds to handle occlusion presents a promising direction that we are eager to explore in future efforts."}]

Author rebuttal:

Meta-review: The paper proposes PRED, a novel pre-training framework for outdoor point clouds that leverages image semantics through neural rendering. All reviewers reach concensus that the paper should be accepted. AC read the paper, rebuttal, author response and discussions. The paper well addressed reviewer concerns and additionaly provide more experimental results as requested. 
Please revise the paper accordingly based reviewer comments for the camera-ready.
Decision: Accept (poster)