Title: PRED: Pre-training via Semantic Rendering on LiDAR Point Clouds
Venue: NeurIPS 2023
Accepted: yes
Keywords: Pre-Train

Autonomous Driving

LiDAR

3D Object Detection
Abstract: Pre-training is crucial in 3D-related fields such as autonomous driving where point cloud annotation is costly and challenging. Many recent studies on point cloud pre-training, however, have overlooked the issue of incompleteness, where only a fraction of the points are captured by LiDAR, leading to ambiguity during the training phase. On the other hand, images offer more comprehensive information and richer semantics that can bolster point cloud encoders in addressing the incompleteness issue inherent in point clouds. Yet, incorporating images into point cloud pre-training presents its own challenges due to occlusions, potentially causing misalignments between points and pixels. In this work, we propose PRED, a novel image-assisted pre-training framework for outdoor point clouds in an occlusion-aware manner. The main ingredient of our framework is a Birds-Eye-View (BEV) feature map conditioned semantic rendering, leveraging the semantics of images for supervision through neural rendering. We further enhance our model's performance by incorporating point-wise masking with a high mask ratio (95%). Extensive experiments demonstrate PRED's superiority over prior point cloud pre-training methods, providing significant improvements on various large-scale datasets for 3D perception tasks. Codes will be available at https://github.com/PRED4pc/PRED.

PRED: Pre-training via Semantic Rendering on LiDAR Point Clouds

Abstract

Pre-training is crucial in 3D-related fields such as autonomous driving where point cloud annotation is costly and challenging. Many recent studies on point cloud pre-training, however, have overlooked the issue of incompleteness, where only a fraction of the points are captured by LiDAR, leading to ambiguity during the training phase. On the other hand, images offer more comprehensive information and richer semantics that can bolster point cloud encoders in addressing the incompleteness issue inherent in point clouds. Yet, incorporating images into point cloud pre-training presents its own challenges due to occlusions, potentially causing misalignments between points and pixels. In this work, we propose PRED, a novel image-assisted pre-training framework for outdoor point clouds in an occlusion-aware manner. The main ingredient of our framework is a Birds-Eye-View (BEV) feature map conditioned semantic rendering, leveraging the semantics of images for supervision through neural rendering. We further enhance our model's performance by incorporating point-wise masking with a high mask ratio (95%). Extensive experiments demonstrate PRED's superiority over prior point cloud pre-training methods, providing significant improvements on various large-scale datasets for 3D perception tasks. Codes will be available at https://github.com/PRED4pc/PRED.

1 Introduction

Pre-training is a fundamental task in deep learning, serving as a powerful tool to harness the potential of copious unlabeled data and enhance the results of downstream tasks. This is especially critical in 3D-related fields, including autonomous driving, where the annotation of point clouds is both laborious and expensive, contrasting with the relative ease of amassing large volumes of unlabeled point cloud data (59). To tackle this challenge, we present PRED ( PRE -training via semantic ren D ering), a novel framework designed to pre-train point cloud processors using multi-view images.

With the tremendous advancements in masked signal modeling within the realms of image and natural language processing (16; 12; 2), recent studies (55; 30; 62; 17; 61; 6) have delved into applying masked auto-encoders to pre-training in the field of point clouds. These works firstly apply patch masking to the input point clouds, then employ an encoder-decoder framework to reconstruct the point clouds, as depicted in Figure 1(a). Apart from masked auto-encoders, other researchers (52; 59; 64; 22; 34; 8) have explored point cloud pre-training through contrastive learning. Here, positive pairs are generated by applying varied data augmentations to the point clouds, as demonstrated in Figure 1(b). Despite these significant strides, there remains a blind spot regarding the inherent incompleteness of point clouds, which is a ubiquitous issue in outdoor LiDAR datasets. For example in nuScenes (4), a large-scale outdoor LiDAR dataset, over 30% of the labeled objects contain fewer

View1

View2

Contrastive

Loss

Encoder

Encode

r

Decoder

(a) Masked Signal Modeling

(b) Contrastive Learning

(c) Ours

BEV feature map

Position

&

BEV feature

Semantic maps

Decoder

Encoder

Encoder

↑

than five points. As illustrated in Figure 2(a), this incompleteness introduces ambiguity into point cloud reconstruction, thereby potentially impacting the quality of the training process.

Images, compared to point clouds, embody more comprehensive information and enriched semantics. Several studies (6; 8; 34) have sought to offset the limitations of outdoor point clouds by incorporating images into point cloud pre-training. However, these works typically align point clouds with images through a straightforward point-to-pixel projection, neglecting to account for occlusion where objects in point clouds may not be visible in the image due to misalignment between LiDAR and camera. This oversight can lead to mismatches between points and pixels, as depicted in Figure 2(b), potentially hindering the effectiveness of pre-training. Hence the principal challenge we address in this paper is an occlusion-aware, image-assisted pre-training framework for outdoor point clouds.

In this study, we introduce a novel pre-training framework, PRED, designed to tackle the problem of reconstruction ambiguity and occlusion by integrating image information through semantic rendering, as illustrated in Figure 1(c). Neural rendering has demonstrated substantial success in learning 3D implicit representations from 2D images within the sphere of 3D generation (28; 31; 40). This inspires us to learn point cloud representations from image semantics via neural rendering. To accomplish this, we first extract BEV feature map from the point cloud using an encoder. Then, we render the semantic maps from image views, conditioned on the BEV feature map. The entire process is supervised by image semantics, thereby circumventing the introduction of reconstruction ambiguity and enabling effective occlusion management by assigning a reduced weight to occluded points during volume rendering. Moreover, we find the critical role of point-wise masking, employing a substantially higher mask ratio of 95% compared to the 75% utilized in previous patch-wise masking methods (16; 55).

We conduct extensive experiments on several outdoor LiDAR datasets, applying a variety of baselines and encoders to appraise the effectiveness of our pre-training method. The experimental results consistently showcase the superiority of our approach, outperforming both training from scratch and recent concurrent point cloud pre-training methods. Our contributions can be summarized as follows: 1) We introduce PRED, a novel pre-training framework for outdoor point clouds via semantic rendering. 2) We incorporate point-wise masking with a high mask ratio to enhance the performance of PRED. 3) Our approach significantly bolsters the performance of various baselines on large-scale nuScenes (4) and ONCE (26) datasets across various 3D perception tasks.

2 Related Work

3D Perception on Point Clouds. 3D perception methods generally fall into two categories: pointbased and voxel-based. Point-based methods (9; 24; 32; 38; 48; 57; 58) directly extract features and

Masking

Masking

Reconstruct

Reconstruct

？

point cloud 1

point cloud 2

target 2

target 1

masked point cloud

(a) Reconstruction ambiguity.

occluded points

points of car

points of background

(b) Mismatch between points and pixels.

predict 3D objects from raw point clouds, while voxel-based methods (11; 13; 15; 36; 37; 39; 45; 54; 60) streamline the process by discretizing these points into regular voxels or pillars for processing with 3D or 2D convolutional networks. Recently, integrating the Transformer (43) into point cloud processing has yielded notable results (27; 14; 42; 47; 49). In this work, we show that a wide range of modern 3D perception methods can benefit from our pre-training framework.

Pre-Training for Point Clouds. Mask-based (61; 30; 62; 46) and contrast-based (22; 52; 64; 23; 33) methods are currently the leading paradigms for point cloud pre-training. However, these techniques often neglect the inherent incompleteness of outdoor point clouds. To address this, some studies (6; 8; 34) try to integrate images for a more comprehensive perspective. Though impressive, they overlook potential occlusion when aligning points with images. To counter these challenges, we introduce PRED, an effective pre-training paradigm that capitalizes on images through semantic rendering.

Neural Rendering. The implicit representations encoded by a neural network (28; 50; 29; 3; 41; 5; 56; 51) have gained a lot of attention recently, where the geometry and appearance in 3D space are encoded by implicit neural representation supervised from 2D images. NeuS (50) constrains the scene space as a signed distance function and applies volume rendering to train this representation. In this work, we apply a NeuS-like rendering method, but different from learning an implicit representation of a single scene, we aim to learn generalized point cloud representations for pre-training. Volumetric Rendering for Perception Tasks. Recently, some works (53; 44) have attempted to introduce volume rendering into perception tasks. Ponder (19) shares a similar spirit to our work. However, Ponder focuses on indoor scenes, where point clouds often contain color information, facilitating color-based pre-training supervision. In contrast, our work addresses outdoor environmentsspecifically, autonomous drivingwhere point clouds are typically LiDAR-derived and colorless. Consequently, Ponder is not applicable due to the absence of color data. In this context, we propose semantic rendering. Unlike Ponder's color-based rendering, our approach capitalizes on the semantic consistency between point clouds and images, offering a distinct strategy for point cloud pre-training.

3 Methodology

We introduce PRED , a novel image-assisted pre-training framework designed for outdoor point clouds, taking into account occlusion. The comprehensive framework is depicted in Figure 3, with subsequent sections delving into the specifics of our method.

3.1 Revisiting NeuS

NeuS (50) is a neural rendering technique employed for single-scene 3D reconstruction. In this approach, the 3D scene is represented by two functions: a signed distance function (SDF) d : R 3 →

Point-wise

Masking

Segmenter

Class

Balance

Sampling

Decoder

Cat

Decoder

Cat

𝜎

𝜎

𝐿

!"#$%

𝐿

&"'()$*+

point cloud

Projection

images

semantic maps

semantic maps

BEV feature

BEV feature

position

embedding

position

embedding

image view

lidar view

sdf

sdf (stop gradient)

&

semantics

Volume Rendering

Volume Rendering

BEV feature map

BEV feature map

Encoder

unmasked point cloud

share parameter

R , which maps each spatial position r ∈ R 3 to its distance from the object's surface, and a color function c : R 3 × S 2 → R 3 , encoding the color of each point in the scene based on a specific viewing direction v ∈ S 2 . Given a pixel with a ray emitted from it denoted as { r ( t ) = o + t v | t ≥ 0 } , where o is the camera center and v is the unit direction vector of the ray, NeuS computes the color for this pixel by integrating the color values along the ray using the following equation:

where C ( o , v ) is the pixel's output color, w ( t ) represents a weight function for the point r ( t ) . To derive an unbiased weight function w ( t ) , NeuS computes the weight function as follows:

where Φ s ( x ) is the sigmoid function Φ s ( x ) = (1 + e -sx ) -1 , and s is a trainable parameter. NeuS is then trained by minimizing the discrepancy between rendered prediction and ground truth colors.

3.2 Pre-Training via Semantic Rendering

We apply Equations 1 and 2 as our rendering method. A simple approach would be to use the image's RGB values as supervision for neural rendering like NeuS (50), but it failed to converge due to the point cloud's color absence. Consequently, we opt for semantic supervision. Unlike the lack of color information, the point cloud contains semantic details, which can be further enhanced through the image, resulting in better convergence during training. As demonstrated in Figure 3, our proposed method employs an encoder to transform input point clouds into a BEV feature map, and a decoder to map spatial positions in the 3D space, represented as r ∈ R 3 , to their corresponding semantic predictions and signed distances to objects, conditioned on the BEV feature map.

Specifically, let P ∈ R N p × 3 and I ∈ R N i × H i × W i × 3 represent the input point clouds with N p points and corresponding N i images with width W i and height H i , respectively. Firstly, we encode the point clouds P into a BEV feature map F ∈ R H × W × C with the encoder E : P → F , where H , W , and C signify the height, width, and number of channels of the feature map, respectively. Subsequently, we sample a batch of pixels from the images I and carry out semantic rendering for pre-training.

Semantic Rendering. Here, we use the pixel u = [ u, v ] as an example to illustrate the process. We denote the camera intrinsic and extrinsic parameters respectively by K ∈ R 3 × 3 and E = [ R , t ] ∈ R 3 × 4 . A ray emitting from the pixel u can be characterized as a line R = t + δ RK -1 [ u, v, 1] ⊤ in world coordinates parametrized by δ . We then sample a sequence of points r i = R ( δ i ) along the ray R . For each point, we utilize a decoder D to predict its semantics and signed distance, given by ( s ( r ) , d ( r )) = D ( Concat ( F ( r ) , PE ( r ))) , (3)

where s ( r ) ∈ R N c and d ( r ) denote the semantic prediction with N c classes and signed distance of point r , F ( r ) symbolizes the feature extracted from the BEV feature map where r is projected, and PE ( r ) is the cosine position embedding of r . Finally, we aggregate the semantic predictions along the ray R employing Equations 1 and 2, but in a discrete format:

where N s represents the number of sampled points. Throughout this process, occluded points will be allocated a low weight (50), thus mitigating the adverse effect of occlusion.

Semantic supervision. Given the absence of ground-truth semantic labels for images, we resort to DeepLabv3 (7) trained on CityScape (10) for predicting pseudo semantic labels for pixel u , denoted as s ∈ R N c . Intuitively, the accuracy of s is linked to its maximum value: larger maximum values indicate more confident predictions with fewer errors. Hence, to offset the impact of inaccurate labels, this maximum value is incorporated as a weight in the loss calculation. Ultimately, our loss function adopts the form of cross-entropy loss:

where U denotes the training batch of pixels, which are sampled from the projections of point clouds. Additionally, considering that the semantic categories in the point cloud are heavily unbalanced with vegetation constituting more than 30% while pedestrians constitute less than 1% we employ classbalanced sampling. Here, the sampling probability of each pixel is inversely proportional to the number of point clouds of the category to which that pixel belongs (see Appendix for more details).

Geometry supervision. Weintroduce depth rendering to bolster the model's understanding of geometric information, which can enhance and stabilize pre-training performance. For depth rendering, we adhere to the same method as outlined in Equation 4:

where δ ( r i ) denotes the depth of point r i , and α i is as defined in Equation 4. During the training phase, we sample batches of pixels projected from the point cloud. The ground-truth depth is determined by the distance between the point and the projection plane. We opt for the view centered on the LiDAR position as the rendering view (referred to as the lidar view henceforth) given that the depth of image views may not be as accurate due to occlusion, as exemplified in Figure 2(b). Finally, we supervise the rendered depth using the L2 loss function:

where d gt represents the ground-truth depth, and V denotes the batch of pixels sampled from the projection plane of the lidar views.

Point-wise Masking. Contrary to the patch-wise masking approach utilized in masked signal modeling, we found that point-wise masking with a high masking ratio (95%) performs more effectively within our pre-training framework. Before inputting the point cloud into the encoder, we uniformly sample a subset of the point cloud without replacement and mask the remaining points. Point-wise masking is more proficient at maintaining the semantics of the scene. For smaller objects such as pedestrians, patch-wise masking could potentially obliterate them entirely, leading to a significant loss of semantics. Conversely, point-wise masking tends to make the point cloud of the object more sparse, thus preserving its semantics. Our experimental results further validate the effectiveness of point-wise masking (refer to Section 4.4 for details).

3.3 Learning Objective

Following (50), we adopt a coarse-to-fine sampling strategy during rendering, utilizing 96 sample points at the coarse stage and 128 sample points at the fine stage. To optimize the model, we minimize the sum of the semantic loss L semantic and the depth loss L depth , together with the Eikonal term L reg , resulting in a total loss L defined as follows:

where λ is a hyper-parameter that defaults to 0.1, and L reg represents the Eikonal term defined by:

While rendering semantics, we apply the stop-gradient to the signed distance in Equation 4. This is to circumvent the potential unreliability of semantic labels, which could otherwise adversely affect the learning of the scene's geometric information.

4 Experiments

4.1 Experimental Settings

Dataset: nuScenes (4) is a challenging outdoor dataset providing diverse annotations for various tasks, such as 3D object detection and BEV map segmentation. It comprises approximately 40k keyframes, each equipped with six cameras and a 32-beam LiDAR scan. For 3D object detection, we report the nuScenes detection score (NDS) and mean average precision (mAP). For map segmentation, we provide the mean Intersection over Union (IoU).

Dataset: ONCE (26) is a large-scale autonomous driving dataset, featuring 1M LiDAR scenes and 7M corresponding camera images. The dataset is split into training, validation, and testing sets consisting of 5k, 3k, and 8k point clouds, respectively. The remaining unannotated point clouds are

divided into three subsets: small (100k scenes), medium (500k scenes), and large (1M scenes) for pre-training purposes. The official evaluation metric is the mean Average Precision (mAP).

Implementation Details. We use DSVT-P (47) as our default encoder, given its flexibility. For the decoder, we use a 4-layer MLP with 256 hidden channels, and Softplus is chosen as the activation function. For image segmentation, we utilize DeepLabV3 (7) with MobileNets (18) as the backbone. During pre-training, we train the model using the AdamW (25) optimizer and the one-cycle policy, with a maximum learning rate of 3 e -4 . We pre-train the model for 45 epochs on the nuScenes dataset, 20 epochs on the ONCE small , 5 epochs on the ONCE medium , and 3 epochs on the ONCE large . During fine-tuning, we employ random flipping, scaling, rotation, and copy-n-paste as data augmentations, with a maximum learning rate of 3 e -3 . All experiments are conducted on NVIDIA V100 GPUs. For more implementation details, please refer to Appendix.

Overlap of labels between pixel semantics and downstream tasks. Our pre-training phase utilizes pixel semantic labels that include 19 classes such as road, sidewalk, building, wall, fence, pole, traffic light, sign, vegetation, terrain, sky, person, rider, car, truck, bus, train, motorcycle, and bicycle. In the nuScenes object detection task, the labels include car, truck, construction vehicle, bus, trailer, barrier, motorcycle, bicycle, pedestrian, and traffic cone. For the ONCE object detection task, the labels are limited to vehicle, pedestrian, and cyclist. The nuScenes BEV map segmentation task uses labels such as drivable, pedestrian crossing, walkway, stop line, car park, and divider. There's an overlap in labels like 'car', 'bicycle', and 'pedestrian', and our approach is more in line with the field of weakly supervised learning.

4.2 3D Object Detection

In this section, we investigate pre-training in an autonomous driving context. We benchmark the performance of our model against previous methods on the nuScenes and ONCE datasets.

NuScenes. We employ Second (54), CenterPoint (60), and DSVT (47) as our baseline and present the performance on the validation and test sets of nuScenes in Table 1. Our pre-trained model consistently improves the performance of all baselines compared to models trained from scratch. On the validation set, our method boosts the Second, CenterPoint, and DSVT by +3.0/2.0 mAP/NDS, +2.7/1.7 mAP/NDS, and +1.6/0.9 mAP/NDS, respectively. With CenterPoint-Voxel as the detector (5-th row), our method surpasses previous pre-training methods (59; 52; 22) that also utilize CenterPoint-Voxel detector by a large margin. On the test set, our method boosts the CenterPoint and DSVT by +2.6/1.7 mAP/NDS and +1.7/1.0 mAP/NDS, respectively. Moreover, our method significantly outperforms previous state-of-the-art LiDAR detectors (20; 65; 47; 1; 35).

ONCE. As shown in Table 2, our model surpasses all other methods on the ONCE validation split. When using CenterPoint (60) as the detector, our method enhances the baseline by 3.13 mAP, outperforming the improvements made by ProposalContrast (59) and GD-MAE (55) by 1.05 and 0.83, respectively. While SLidR (34) also incorporates image information during pre-training, our method significantly outperforms SLidR, showcasing the effectiveness of our approach in utilizing image information. The distinction between the 'no-pretrain' versions of SLidR and our model originates from their foundational detection frameworks, where SLidR employs PointRCNN as its detector. As the scale of pre-training data continues to increase (from small to large ), our method delivers continuous gains, indicating its scalability with respect to data size.

4.3 BEV map segmentation

We also demonstrate the versatility of our approach by evaluating its performance on the BEV Map Segmentation task of the nuScenes dataset (4). We provide the IoU results for six background classes in Table 3. Leveraging our PRED, we achieve a significant improvement of 3.4% over DSVT's performance, demonstrating the effectiveness of our approach in map segmentation tasks.

4.4 Ablation Studies

In this section, we conduct a series of ablation studies on the nuScenes val to explore the essential components of our methodology. Unless otherwise specified, we use CenterPoint (60) as the detector.

20

40

60

80

100

masking ratio (%)

62.0

62.5

63.0

63.5

64.0

mAP

90.0

92.5

95.0

97.5

100.0

62.0

62.5

63.0

63.5

64.0

Patch-wise

Group-wise

Point-wise

Encoder type. To demonstrate the adaptability of our approach across various 3D backbone architectures, we experiment by substituting DSVT-P (47) with 2D Conv from PointPillar (20), and VoxelNet from CenterPoint-Voxel (60). The results, presented in Table 4, show that our approach can boost detector performance by 3.0/1.9 mAP/NDS and 2.8/1.8 mAP/NDS when employing 2D Conv and VoxelNet as backbones, respectively. These improvements are comparable to those achieved using DSVT-P. Due to DSVT-P's superior performance and flexibility, it was chosen as our default encoder. However, our method proves to be effective with a variety of encoder types.

Semantic rendering design. Here, we verify the efficacy of our semantic rendering pipeline as shown in Table 5. A basic approach to leveraging images is to directly project the point cloud onto the images. However, as indicated in the 2nd row, this method does not yield significant improvements due to the overlooking of occlusion effects during the projection process. A potential solution is to filter the point cloud based on the image depth, thereby excluding points that do not align with the image depth from the training batch. In the absence of ground-truth image depth, we employ BEVDepth (21) for depth estimation. The enhanced performance in the 3rd row underlines the importance of handling occlusion. However,

the inaccuracy of depth estimation hinders further performance improvement. Our neural rendering approach, shown in the 4th row, surpasses the prior two methods by adaptively managing occlusion, where attributing a reduced weight to occluded points when accumulating semantics as in Equation 4. Ultimately, incorporating depth supervision yields the best results, as depicted in the final row.

Table 6 presents ablation experiments on class balance sampling and confidence weight in semantic supervision. Both components are crucial for semantic supervision; the confidence weight mitigates the effects of inaccurate semantic labels, and class balance sampling addresses the issue of category imbalance in point clouds. For depth supervision, We compared two different rendering views in Table 7. The lidar view outperforms the image view, as the depth of points might be inaccurate in the image view, but not in the lidar view.

Masking strategy. In Figure 4, we explore various masking strategies to investigate their impact on our pre-training framework: 1) Patch-wise masking masks a certain fraction of non-empty tokens, with each token representing a non-overlapping patch of the BEV feature map. 2) Group-wise masking masks a certain proportion of point cloud groups, with each group selected using the Furthest Point Sampling (FPS) and k -Nearest Neighbor (kNN) algorithm, following (61). 3) Point-wise masking, as elaborated in Section 3.2. As illustrated in Figure 4, group-wise and patch-wise masking strategies perform better when the masking ratio is low. When the ratio reaches 70%, the impact of group-wise and patch-wise masking hits its peak. Intriguingly, the effectiveness of point-wise masking continues to ascend until the masking ratio reaches 95%. The results from point-wise masking surpass those of the other two methods, achieving a score of 64.2 mAP. Consequently, we opt for

the point-wise masking strategy with a 95% masking ratio, as it offers the advantage of accelerated processing while preserving good performance.

Data-efficient. Asignificant advantage of pre-training lies in its ability to enhance data efficiency for downstream tasks that have limited annotated data. In this study, we examine data-efficient 3D object detection by first conducting pre-training on the nuScenes (4) training data, then fine-tuning the pre-trained model with varying fractions of training data: 10%, 20%, 50%, and 100%. The results of these experiments are illustrated in Table 8. Overall, our pre-trained model consistently boosts detection performance, especially when the available labeled data is limited, i.e., improving 4.8 mAP and 3.1 NDS with only 10% labeled data. Remarkably, even when utilizing only 50% of the annotated data, the pre-trained model achieves 63.0 mAP and 68.5 NDS, surpassing the performance of a nonpre-trained version using 100% of the annotated data.

More experimental results are provided in Appendix.

5 Conclusion

In this paper, we introduce PRED, an effective pre-training framework for outdoor point cloud data. Our framework mitigates the inherent incompleteness of point clouds by integrating image information through semantic rendering. It also allows for effective occlusion management by assigning a reduced weight to occluded points during volume rendering. A point-wise masking strategy, with a mask ratio of 95%, is adopted to optimize performance. Extensive experiments validate the effectiveness of our framework as it significantly improves various baselines on the large-scale nuScenes and ONCE datasets across various 3D perception tasks. We hope our PRED can serve as a powerful baseline to inspire future research on point cloud pre-training.

Limitations. PRED mainly focuses on point cloud pre-training with images as supervision. The integration of an image-point multi-modality pre-training could potentially improve performance. However, it's beyond this paper's scope and we intend to explore this possibility in our future research.

References

Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR , 2022. 6, 8

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021. 1

Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV , 2021. 3

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , 2020. 1, 2, 6, 8, 10

Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV , 2021. 3

Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang, Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. arXiv preprint arXiv:2303.08129 , 2023. 1, 2, 3

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017. 5, 7

Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. arXiv preprint arXiv:2301.04926 , 2023. 1, 2, 3

Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points for voting-based 3d object detection in point clouds. In CVPR , 2021. 2

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR , 2016. 5

Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In AAAI , 2021. 3

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. 1

Shaocong Dong, Haiyang Wang, Tingfa Xu, Xinli Xu, Jie Wang, Ziyang Bian, Ying Wang, Jianan Li, et al. Mssvt: Mixed-scale sparse voxel transformer for 3d object detection on point clouds. NeurIPS , 2022. 3

Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In CVPR , 2022. 3

Tianrui Guan, Jun Wang, Shiyi Lan, Rohan Chandra, Zuxuan Wu, Larry Davis, and Dinesh Manocha. M3detr: Multi-representation, multi-scale, mutual-relation 3d object detection with transformers. In WACV , 2022. 3

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022. 1, 2

Georg Hess, Johan Jaxing, Elias Svensson, David Hagerman, Christoffer Petersson, and Lennart Svensson. Masked autoencoder for self-supervised pre-training on lidar point clouds. In WACV , 2023. 1

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017. 7

Di Huang, Sida Peng, Tong He, Honghui Yang, Xiaowei Zhou, and Wanli Ouyang. Ponder: Point cloud pre-training via neural rendering. In ICCV , 2023. 3

Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR , 2019. 6, 7, 8, 9

Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint arXiv:2206.10092 , 2022. 9

Hanxue Liang, Chenhan Jiang, Dapeng Feng, Xin Chen, Hang Xu, Xiaodan Liang, Wei Zhang, Zhenguo Li, and Luc Van Gool. Exploring geometry-aware contrast and clustering harmonization for selfsupervised 3d object detection. In ICCV , 2021. 1, 3, 6, 8

Kangcheng Liu, Aoran Xiao, Xiaoqin Zhang, Shijian Lu, and Ling Shao. Fac: 3d representation learning via foreground aware feature contrast. arXiv preprint arXiv:2303.06388 , 2023. 3

Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In ICCV , 2021. 2

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. 7

Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, et al. One million scenes for autonomous driving: Once dataset. arXiv preprint arXiv:2106.11037 , 2021. 2, 6

Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In ICCV , 2021. 3

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 2021. 2, 3

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. , 2022. 3

Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In ECCV , 2022. 1, 3

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022. 2

Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV , 2019. 2

Yongming Rao, Benlin Liu, Yi Wei, Jiwen Lu, Cho-Jui Hsieh, and Jie Zhou. Randomrooms: unsupervised pre-training from synthetic shapes and randomized layouts for 3d object detection. In ICCV , 2021. 3

Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self-supervised distillation for autonomous driving data. In CVPR , 2022. 1, 2, 3, 7, 8

Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: Real-time and high-performance pillar-based 3d object detection. In ECCV , 2022. 6, 8

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In CVPR , 2020. 3, 7

Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. IJCV , 2023. 3

Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In CVPR , 2019. 2, 7

Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. TPAMI , 2020. 3

Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, and Sergey Tulyakov. 3d generation on imagenet. arXiv preprint arXiv:2303.01416 , 2023. 2

Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering. In CVPR , 2022. 3

Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV , 2022. 3

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 30, 2017. 3

Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. arXiv preprint arXiv:2111.13260 , 2021. 3

Haiyang Wang, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang, et al. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. NeurIPS , 2022. 3

Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In ICCV , 2021. 3

Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, and Liwei Wang. Dsvt: Dynamic sparse voxel transformer with rotated sets. arXiv preprint arXiv:2301.06051 , 2023. 3, 6, 7, 8, 9

Haiyang Wang, Shaoshuai Shi, Ze Yang, Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. In CVPR , 2022. 2

Haiyang Wang, Hao Tang, Shaoshuai Shi, Aoxue Li, Zhenguo Li, Bernt Schiele, and Liwei Wang. Unitr: A unified and efficient multi-modal transformer for bird's-eye-view representation. In ICCV , 2023. 3

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021. 3, 4, 5, 6

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view imagebased rendering. In CVPR , 2021. 3

Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In ECCV , 2020. 1, 3, 6, 7, 8

Chenfeng Xu, Bichen Wu, Ji Hou, Sam Tsai, Ruilong Li, Jialiang Wang, Wei Zhan, Zijian He, Peter Vajda, Kurt Keutzer, et al. Nerf-det: Learning geometry-aware volumetric representation for multi-view 3d object detection. In ICCV , 2023. 3

Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors , 2018. 3, 7, 8

Honghui Yang, Tong He, Jiaheng Liu, Hua Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wanli Ouyang. Gd-mae: Generative decoder for mae pre-training on lidar point clouds. arXiv preprint arXiv:2212.03010 , 2022. 1, 2, 6, 7, 8

Hao Yang, Lanqing Hong, Aoxue Li, Tianyang Hu, Zhenguo Li, Gim Hee Lee, and Liwei Wang. Contranerf: Generalizable neural radiance fields for synthetic-to-real novel view synthesis via contrastive learning. In CVPR , 2023. 3

Hao Yang, Chen Shi, Yihong Chen, and Liwei Wang. Boosting 3d object detection via object-focused image fusion. arXiv preprint arXiv:2207.10589 , 2022. 2

Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR , 2020. 2

Junbo Yin, Dingfu Zhou, Liangjun Zhang, Jin Fang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang. Proposalcontrast: Unsupervised pre-training for lidar-based 3d object detection. In ECCV , 2022. 1, 6, 7, 8

Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR , 2021. 3, 6, 7, 8, 9

Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In CVPR , 2022. 1, 3, 9

Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. arXiv preprint arXiv:2205.14401 , 2022. 1, 3

Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds. In CVPR , 2022. 7

Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. In ICCV , 2021. 1, 3, 7

Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv:1908.09492 , 2019. 6, 8

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work incorporates images into point cloud pretraining since images contain richer semantic information. Instead of adopting the back propagation strategy which can not handle the misalignment between camera and LiDAR, this paper leverages the neural rendering technique to injecting the semantics into the representation learning process.\n\nSTRENGTHS:\n(1) This paper is well-written and presents a clear and concise idea.\n(2) The motivation of leveraging the neural rendering to overcome the mismatch of pixel and point is compelling.\n(3) The experimental results demonstrate consistent improvement on different datasets with varying baselines. And the ablation studies validate the contribution of each component in the pipeline.\n\nWEAKNESSES:\n(1) The technical contribution is limited. For example, the BEV conditioned semantic neural rendering strategy, and the masking strategy have been studied in previous works.\n(2) Although the occlusion problem can be alleviated by assigning a reduced weight to occluded points, the rendering process will introduce additional noise since it assigns multiple semantic labels to each point, leading to the semantic ambiguity.\n(3) The rendering process allow every points in the ray receive the gradients which leads to many irrelevant points being optimized.\n(4) The authors claims that the neural rendering is superior to the point-to-pixel projection, but there is no performance comparison between these two methods in the experiments. I think the latter may also bring an obvious improvement since the inambiguous projections.\n(5) This work takes the semantic labels as supervision for neural rendering. I think it would be better to use the 2D feature vectors as the supervision.\n\nQUESTIONS:\nMy major concern lies in the superiority of this rendering process compared to the point painting process. While the author address the occlusion issue, I firmly believe that it can be effectively resolved by accurately determining the object boundaries. By precisely locating the boundaries of objects, the occlusion challenge can be easily mitigated, potentially diminishing the need for a complex rendering approach.\n\nLIMITATIONS:\nThe author address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates weakly-supervised representation learning for outdoor LiDAR point clouds. To start, the authors point out that the inherent incompleteness of outdoor LiDAR points would reduce the effectiveness of self-supervised representation learning approaches. To mitigate this, the authors propose to use synchronized images as additional signals to supervise the representation learning process. Observing the lack of color information in the point cloud and the mismatch between points and pixels due to occlusion, the authors propose to use neural rendering of pixel semantics as a pre-text task to circumvent the two problems. Specifically, with a slight modification to NEUS’s weighting function, the authors build an implicit neural representation on top of the BEV LiDAR features, positing that good BEV LiDAR features for implicit neural representations could be good representations for any downstream recognition tasks. Extensive experiments have been conducted to show the effectiveness of the approach.\n\nSTRENGTHS:\n- Novelty - Using neural rendering as a pre-text task for point cloud representation learning is a novel idea the reviewer has never seen in the literature. \n- Beautiful figures - Figure 1,2,3 are aesthetically pleasing, with figure 2 clearly illustrating the core problems in representation learning for outdoor LiDAR point clouds. \n- Strong performance improvement over baselines that do not use images as additional signals. \n- Thorough experiments - Experiments thoroughly demonstrate the strength of the approach.\n\nWEAKNESSES:\nOverall, this is a strong paper but the reviewer has to point out three weaknesses — two related to presentation and the other related to baselines. \n\n- (Presentation 1) One of the contributions of the paper is to show that images could be valuable signals for pre-training representation. However, it is unclear which baselines in table 1 and 2 actually use additional images as signals. Also, it is unclear what type of signals are being used (i.e. raw pixel values vs pixel semantics). \n- (Baseline) Another contribution of the paper is to show that pixel semantics is a useful signal for pre-training representation. Although the proposed approach did not converge when trying to render raw pixel value (line 101-103), SLidR (table 2) is a baseline that actually leverages color information to form superpixel for pre-training. Given the big difference between the no-pretrain variants for SLidR and ours, it is difficult to judge whether pixel-value is a weaker signal or there is some other difference (such as model architecture or optimization recipes) that is causing SLidR to underperform PRED. \n- (Presentation 2): Since using pixel semantics is part of the core contributions, it is important for the authors to acknowledge/mention whether there are any overlaps between the pixel semantic label space and the downstream task label space. Without this, it is difficult to tell whether the proposed approach is a self-supervised approach or a weakly-supervised approach.\n\nQUESTIONS:\nThe reviewer believes this is a good paper. However, the reviewer thinks it is important to properly address the weaknesses (especially on the SLiDR baseline and presentation 2). For pre-rebuttal, the reviewer would give “borderline reject” to implore the authors to properly address the weaknesses. If the weaknesses are sufficiently addressed, the reviewer is more than happy to give this submission a “strong accept”. \n\nSuggestions:\n- To fix the first weakness in the presentation, the reviewer recommends adding a column in table 1 and 2 to indicate whether and the type of pixel signal used. Also, it would be great to emphasize this in section 4.2.\n- To fix the second weakness in the presentation, the reviewer recommends adding a paragraph indicating label overlap between the pixel semantics and downstream tasks in section 4.1.\n- For the baseline, the reviewer would like to know why the “no-pretrain” SLiDR baseline is much worse than the \"no-pretrain\" OURS baseline is table 2.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposed a novel point cloud pre-training framework, PRED, which leverages the semantic information consistency between the LiDAR point clouds and the camera images to improve the point cloud pre-training performance. The author proposed (1) a novel semantic rendering module for decoding the semantics from the BEV feature maps and (2) a point-wise masking mechanism to alleviate the reconstruction ambiguity.

The proposed pre-training method demonstrates superior performance according to the experiments section.

STRENGTHS:
(1), To the best of my knowledge, the proposed point cloud pre-training framework with semantic rendering is novel and reasonable.
(2), point cloud pre-training is an important task for academia and industry.
(3), According to the experiment section, the proposed framework PRED has achieved superior performance on multiple benchmarks.

WEAKNESSES:
(1), Dealing with occlusion is claimed as one of this paper's major contributions and advantages. However, why the occluded points will be allocated a lower weight is not illustrated clearly in the paper. The author could add more presentation, analysis, visualization, and evaluation. 
(2), The pre-trained semantic model is required for the proposed approach. This point should be marked and compared with other methods in Tables 1 and 2.

QUESTIONS:
(1), The proposed method relies on well-trained 2D segmentation models, which limits its generalization ability.

LIMITATIONS:
(1) Is there anything special or novel for handling occlusion compared to [46]? If not, then this point should not be highlighted.
(2) How would the proposed method perform if the 2D segmenter fails?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new pretraining algorithm for outdoor 3D perception tasks, where images are utilized to provide comprehensive semantic information. The main algorithm is to leverage the semantics of images for supervision through neural rendering. The authors also apply point-wise masking with a high mask ratio to further enhance performance. The pretraining brings notable performance gains on multiple benchmarks.\n\nSTRENGTHS:\n1.\tThe authors provide a novel insight into the exploitation of image semantics by combing off-the-shelf image segmenter and neural rendering, which I think is well-motivated. \n2.\tExtensive experiments on multiple benchmarks demonstrate the effectiveness of the proposed method.\n3.\tThe paper is well-written and easy to follow. The illustrations are straightforward and helpful.\n\nWEAKNESSES:\n1. I acknowledge the motivation and the method design of this paper. Different from neural rendering for RGB as NeRF does, the high-level idea of rendering semantics from point-cloud is quite novel. However, I have heavy concerns about the use of Cityscape pre-trained segmenter as it is not general enough, e.g., Cityscape pre-trained segmenter can perform well on nuscenes and Once but can't on Waymo. Despite the authors explaining the reason Waymo experiments do not perform that well on Line101 of the appendix, I think the poor generalizability of pre-trained segmenter maybe also a reason. So I suggest replacing the pre-trained segmenter with SAM.\n\n2. This work shares a similar spirit to Ponder [1]. Despite [1] for indoor scenes, the paper aims at outdoor scenes, the authors should carefully discuss the intrinsic differences. \n\n[1] Ponder: point-cloud pre-training via neural rendering.\n\nQUESTIONS:\nPlease refer to the weakness part, I would raise my score if the two concerns can be addressed.\n\nLIMITATIONS:\nPlease refer above.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. The paper proposes PRED, a novel pre-training framework for outdoor point clouds that leverages image semantics through neural rendering. The paper addresses the challenges of incompleteness and occlusion in point clouds, which are common in outdoor LiDAR datasets for autonomous driving.\n2. The paper uses an encoder-decoder architecture to extract a BEV feature map from the point cloud and render semantic maps from image views, supervised by image segmentation and depth estimation.\n3. The paper incorporates point-wise masking with a high mask ratio (95%) to enhance the pre-training performance and avoid losing semantics of small objects.\n4. The paper conducts extensive experiments on nuScenes and ONCE datasets, showing that PRED significantly improves various baselines and state-of-the-art methods on 3D object detection and BEV map segmentation tasks.\n\nSTRENGTHS:\n1. The paper presents a novel pre-training framework for outdoor point clouds that integrates image semantics through neural rendering. This is a creative and effective way to address the incompleteness and occlusion issues in point clouds, which are often overlooked by previous pre-training methods. The paper also introduces point-wise masking with a high mask ratio, which is different from the conventional patch-wise or group-wise masking strategies and preserves more semantics of small objects.\n2. The paper is technically sound and well-motivated. The paper provides a clear and detailed description of the proposed method, including the encoder-decoder architecture, the semantic rendering pipeline, the loss functions, and the masking strategy. The paper also conducts extensive experiments on two large-scale datasets, nuScenes and ONCE, and compares with various baselines and state-of-the-art methods on 3D object detection and BEV map segmentation tasks. The paper reports significant improvements over previous methods, demonstrating the effectiveness and generality of the proposed framework.\n\nWEAKNESSES:\n1. The paper does not conduct ablation studies on the choice of image segmentation model and its impact on the pre-training performance. The paper uses DeepLabv3 as the image segmenter but does not justify or evaluate this choice. It is unclear how the quality and accuracy of the image segmentation model affect the semantic rendering and supervision.\n2. This paper does not report the computational cost or time complexity of the pre-training framework. Semantic rendering involves sampling and aggregating points along multiple rays for each pixel, which may be computationally expensive and memory-intensive. It would be helpful to provide some statistics or benchmarks on the pre-training speed and resource consumption.\n\nQUESTIONS:\nPlease see the comments on 'weaknesses'.\n\nLIMITATIONS:\nPlease see the comments on 'weaknesses'.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}