Point Cloud Reconstruction Is Insufficient to Learn 3D Representations

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 04:30:18

Abstract: This paper revisits the development of generative self-supervised learning in 2D images and 3D point clouds in autonomous driving. In 2D images, the pretext task has evolved from low-level to high-level features. Inspired by this, through explore model analysis, we find that the gap in weight distribution between self-supervised learning and supervised learning is substantial when employing only low-level features as the pretext task in 3D point clouds. Low-level features represented by PoInt Cloud reconsTruction are insUfficient to learn 3D REpresentations (dubbed PICTURE). To advance the development of pretext tasks, we propose a unified generative self-supervised framework. Firstly, high-level features represented by the Seal features are demonstrated to exhibit semantic consistency with downstream tasks. We utilize the Seal voxel features as an additional pretext task to enhance the understanding of semantic information during the pre-training. Next, we propose inter-class and intra-class discrimination-guided masking (I$^2$Mask) based on the attributes of the Seal voxel features, adaptively setting the masking ratio for each superclass. On Waymo and nuScenes datasets, we achieve 75.13\% mAP and 72.69\% mAPH for 3D object detection, 79.4\% mIoU for 3D semantic segmentation, and 18.4\% mIoU for occupancy prediction. Extensive experiments have demonstrated the effectiveness and necessity of high-level features. The project page is available at https://anonymous-picture.github.io/.

Main paper: 
 introduction: 
 LiDAR has received widespread attention for its ability to simulate the depth and spatial distribution of the surrounding environment with high quality. In outdoor autonomous driving (AD), many works are based on LiDAR to achieve 3D perception, such as Point-Pillars [20] , PV-RCNN [32] , and SphereFormer [19] . Advanced 3D perception algorithms can significantly improve the safety of vehicles. However, a large amount of carefully annotated 3D data, such as 3D boxes or point cloud semantic categories, is costly and timeconsuming [26] . This leads to advanced models being trapped in fitting specific scenarios, and performance deteriorates significantly when migrating to other scenarios [17] . Learning the universal representation of 3D point clouds from a large amount of unlabeled data is a promising solution to alleviate the problems above. Generative self-supervised learning (SSL), represented by masked autoencoders [13] , has attracted attention in autonomous driving community. Occupancy-MAE [28] predicted whether each masked voxel was occupied. This promoted scene understanding by inferring the composition information of 3D scenes. The reconstruction target of GD-MAE [42] was to reconstruct the 3D coordinates of the point clouds in each masked voxel. GeoMAE [36] further introduced geometric features, e.g., normal, to enhance spatial understanding. MV-JAR [40] combined jigsaw and masked autoencoders to understand the spatial relationship between voxels. However, these selfsupervised methods did not perform as expected on downstream tasks. Let's revisit the development process of representation learning in 2D images, as shown in Fig. 1 . Low-level features are defined as unimodal, simple physical properties or spatial composition information, whereas the opposite is defined as high-level features. At the beginning, MAE [13] explicitly reconstructed missing raw pixels in low-level. Subsequently, MaskFeat [39] and MR-MAE [11] et al. reconstructed high-level features such as histogram of oriented gradient (HOG) and CLIP features. This has been proven to facilitate the interpretation of the image. However, current generative self-supervised learning methods for 3D point clouds still reconstruct explicit spatial information in low-level such as 3D coordinates. We argue that the representation learning in 3D point clouds is equivalent to the primitive stage of 2D images.Downstream tasks, such as semantic segmentation and occupancy prediction, require encoders derived from self-supervised learning to be proficient in extracting semantic information. This includes not only the multimodal features of single point cloud, such as images and text, but also the collective information from contextual point clouds. However, reconstructing spatial information focuses only on the 3D coordinates of the single point cloud, which is detrimental to downstream tasks that require semantic information. We believe that PoInt Cloud reconsTruction is insUfficient to learn 3D REpresentations (dubbed PICTURE).The question we ask is this: Why did representation learning in 3D point clouds not transition from explicit low-level features to implicit high-level features like in 2D images? Are implicit highlevel features not applicable in 3D point clouds, or have appropriate methods not been found yet?Recently, contrastive learning-based methods such as SLidR [31] and Seal [24] have explored collaborative learning on point cloudimage pairs obtained from superpixels [1] or vision foundation model SAM [16] . Through Seal, information is aggregated not only about the point cloud itself but also about the color of contextual points. From a qualitative perspective, we have demonstrated the semantic consistency between the Seal feature heatmap and the ground truth in downstream tasks through visualization. From a quantitative perspective, an exploratory model analysis based on weight distribution has been employed to demonstrate the promotion of using Seal features as pretext tasks on downstream tasks. Based on these observations, we propose a unified framework for generative self-supervised learning, PICTURE. We consider reconstructing Seal voxel features as an additional pretext task. Specifically, MinkUNet [7] pre-trained by Seal acts on raw point clouds. The point cloud features within the same voxel are aggregated to obtain Seal voxel features of the masked voxel. Reconstructing Seal voxel features implicitly benefits from image supervision and encourages the model to learn high-level vision concepts. More efficiently, we complete this process offline to accelerate the training.The mask sampling strategy is essential to the generative SSL. Previous random masking treated all non-empty voxels equally. This neglected the varying difficulties caused by the discrimination of voxel features. Moreover, it dispersed the focus evenly across all regions instead of the categories of interest for downstream tasks. Inspired by Focal Loss [22] , AttMask [15] , etc., we develop a mask sampling strategy that is tightly coupled with the target Seal voxel features to adjust the difficulty. Specifically, we propose inter-class and intra-class discrimination-guided masking (I 2 Mask). Firstly, eight superclasses that can represent autonomous driving scenarios are obtained unsupervised. On the one hand, for interclass, we propose Fastest Class Sampling (FCS), which divides eight superclasses into three groups based on discriminative cues and sets base mask ratios. On the other hand, we define the intra-class consistency coefficient for each superclass and modulate the base mask ratio. As a result, the visualization of the mask ratio reveals that the reconstruction is more focused on regions like vehicles.We conduct experiments on Waymo [34] and nuScenes [4] to verify promotion for downstream tasks. Compared to other selfsupervised methods, our method achieves state-of-the-art results.The main contributions of this paper are as follows:• We propose a unified framework for generative self-supervised learning, PICTURE, in autonomous driving. Seal features with strong semantic information serve as additional reconstruction targets. 3D representation learning has been pushed forward following the developmental trajectory of 2D images. • We propose inter-class and intra-class discrimination-guided masking. We have demonstrated that, compared to random masking, I 2 Mask tightly coupled with target features achieves significant improvements. • We achieve improvements of 30.4%, 26.1%, and 40.0% compared to advanced self-supervised methods in downstream tasks such as object detection, semantic segmentation, and occupancy prediction. We demonstrate the benefits through ablation studies. 
related work 2.1 self-supervised learning in ad: 
 Learning universal features from large-scale point clouds in a selfsupervised manner attracts attention in the autonomous driving community. Similar to 2D images, contrastive and generative selfsupervised learning are two mainstream approaches. In contrastive self-supervised learning, ProposalContrast [45] applied random geometric transformations under two views to construct training samples. SLidR [31] and Seal [24] performed contrastive learning between corresponding images and point clouds through SLIC [1] or SAM [16] . CLIP 2 [47] followed the architecture of CLIP [30] and performed triple contrastive learning between images, point clouds, and text. TARL [29] required the encoder to extract equivalent features at different timesteps. In generative self-supervised learning, point cloud reconstruction is considered a crucial avenue to understanding point cloud scenes. 3D coordinates [40, 42] and other physical properties [14, 18, 36] were used as reconstruction targets.This method encourages the model to understand the details and distribution of 3D point clouds, which is the focus of this paper.Recently, drawing inspiration from language models and content generation, ViDAR [44] forced the model to predict the future from history and supervised the scene flow. UniPAD [43] rendered 2D RGB-D images using raw sparse point clouds.Pretext tasks are exploited by generative self-supervised methods to extract information from unlabeled datasets. The most common pretext task was to reconstruct the 3D coordinates of the point clouds in each masked voxel [14, 23, 40, 42] . Some work [3, 27, 28, 36] predicted whether each masked voxel was occupied. In addition, some other spatial information was used for pretext tasks, such as geometric features [36] , jigsaw [40] , occupation type [14, 18] , and number of point clouds [14, 23] . However, the current pretext task in autonomous driving scenes is still focused on reconstructing explicit spatial information in low-level. Implicit semantic features in highlevel have never been introduced as pretext tasks. We believe that explicit spatial information in low-level is insufficient.The mask sampling strategy determines where pretext tasks are applied. For 3D point clouds in autonomous driving, most of the work [3, 14, 36, 40] used random masking. MAELi [18] proposed that the masking ratio should decrease with the distance from ego, which not only reduced the difficulty of reconstructing far-range point clouds, but also ensured a consistent sparsity among both near-range and far-range points to enhance generalization ability of the model. BEV-MAE [23] applied a lower masking ratio in sparse areas to ensure a more stable reconstruction process. GD-MAE [42] implemented separate masking strategies at different granularities to maintain a consistent masking scope. However, mask sampling strategies based on feature attributes are less discussed in 3D generative SSL. In 2D images, AttMask [15] employed the attention map to establish the masking map. Areas that were highly attended were masked, making the reconstruction more efficient. SemMAE [21] and AMT [25] utilized iteration and throwing to make the attention map focus more on objects, respectively. Inspired by this, we develop a mask sampling strategy based on the attributes of the target Seal voxel features to adjust the difficulty and focus. 
proposed method: 
 In this section, we first introduce the unified self-supervised learning framework PICTURE in Fig. 2 and the process of exploiting Seal features in 3D high-level voxel feature generation module (Sec.3.1). We then propose inter-class (Sec. 3.2) and intra-class (Sec. 3.3) discrimination-guided masking based on the feature attribute.Voxelization and Masking. The point cloud can be defined as P = 𝒄 ℓ , 𝒇 ℓ ℓ = 1, . . . , 𝑁 p , where 𝒄 ℓ and 𝒇 ℓ represent coordinates and point cloud features. We use the Voxel Feature Encoding (VFE) [41] for voxelization. The coordinates and features of all non-empty voxels are defined as C all ∈ R 𝑁 v ×3 , F all ∈ R 𝑁 v ×𝐶 . The masking sampling strategy will be described in Sec. 3.2 and 3.3. The coordinates of the mask voxels are denoted as C m . The coordinates and features of unmasked voxels are denoted as C um , F um .Sparse Encoder and Decoder. To reduce the computational complexity of the attention, SST [9] and DSVT [37] specifically designed grouping mechanisms for sparse voxels. The coordinates C um and feature F um of unmasked voxels are input into DSVT to extract features. The features of masked voxels are replaced by mask tokens and input into the decoder together with the output of the encoder. We also use DSVT as the decoder. In this way, the features of the masked voxels can be reconstructed.Reconstruction Target. Reconstructing spatial features is not sufficient for downstream tasks that require semantic information. Seal [24] transferred image features from pre-trained vision foundation models to 3D networks, progressively aggregating contextual point clouds as it delves deeper into layers. This process can provide favorable semantic feature reconstruction signals for generative self-supervised learning. The Seal feature heatmaps [24] for example point cloud scene in nuScenes [4] are shown in Fig. 3(a) . At the same time, the ground truth of the same scene in three downstream tasks are visualized in Fig. 3(b ). It can be seen Seal features and the real scene have a high semantic consistency. The Seal feature is strong for road-related objects. Besides, the feature decreases with distance from ego. Based on this observation, we consider reconstructing Seal voxel features as an additional pretext task. Specifically, all raw point clouds P that have not been voxelized are fed into MinkUNet (Res16UNet34C) [7] pre-trained by Seal. Each point cloud aggregates spatial and feature information from other point clouds. We obtain the Seal point features P s . Subsequently, voxelization and average pooling are employed to aggregate sparse features that are located in the same voxel. Finally, we obtain the Seal voxel features S all = 𝒔 𝑗 𝑗 = 1, . . . , 𝑁 v of all non-empty voxels. This process can be completed offline in advance to accelerate training. For masked voxels, the loss between reconstructed and target Seal voxel features is defined as:EQUATIONFig. 3 (a) shows the heatmap extracted by the transformer encoder before and after introducing the pretext task. After introducing Seal voxel features, the features extracted by encoder have stronger semantic information, which is crucial for downstream tasks.We develop a mask sampling strategy based on the target Seal voxel features. For 22 categories in SemanticKITTI [2] , 17 categories in nuScenes Lidarseg [10] , and 22 categories in Waymo Semantic Segmentation [34] , we first classify them into eight semantic classes: ground-related, structures, vehicle, two-wheeled vehicle, nature, human, object, and outlier. Semantic labels are agnostic in self-supervised learning. Therefore, we cluster Seal voxel featuress 7:K = K \ K 𝑡 .8:𝑡 = 𝑡 + 1. 9: end while 10: return K 1 , K 2 , K 3 S all using the k-means [6] , and the number is set to 8. We obtain eight superclasses 𝑘 𝑖 . Although it is impossible to determine the correspondence between 𝑘 𝑖 and semantic classes, this partition has considered the semantic attributes of the Seal voxel features.We set the base masking ratio for each superclass based on the inter-class discrimination. First, we compute the cluster center 𝝁 𝑘 𝑖 , i.e., the mean of Seal voxel features:EQUATIONwhere 𝑁 𝑘 𝑖 is the number of voxels of 𝑘 𝑖 . Next, we perform Algorithm 1 Fastest Class Sampling on clustering centers 𝝁 𝑘 . We define the set K = {𝑘 𝑖 | 𝑖 = 1, 2, . . . , 8}. Then we select any 𝑛 1 superclass from K and calculate the distance between the cluster centers. The set D inter composed of average inter-class distances is:EQUATIONwhere 𝑑 (𝝁 𝑘 𝑚 , 𝝁 𝑘 𝑛 ) represents the distance between the clustering centers of two superclasses, e.g., cosine similarity. Next, find the 𝑛 1 superclass partition with the fastest average inter-class distance, denoted as K 1 . Subsequently, the set difference between the original set K and K 1 is used as the input for the next cycle. Select any 𝑛 2 superclass from the remaining set, repeat the above process, and finally obtain the 𝑛 2 superclass partition with the second fastest average inter-class distance, denoted as K 2 . The average inter-class distance of the remaining set is the smallest, denoted as K 3 .The average inter-class distance reflects inter-class discrimination. Larger inter-class differences facilitate more straightforward distinction and reconstruction, and are typically associated with dynamic objects such as vehicles. We set a higher base mask ratio to increase the difficulty and the focus on the reconstruction. Based on this principle, we set the base mask ratio 𝑟 1 b , 𝑟 2 b , 𝑟 3 b for each set.The intra-class discrimination reflects the difficulty of reconstructing masked voxel features from unmasked voxels. First, we define EQUATIONwhere 𝑑 (𝝁 𝑘 𝑖 , 𝒔 𝑘 𝑖 𝑗 ) represents the distance between the Seal features of a voxel and its clustering center. The 1{} stands for indicator function. The denominator of Eq. ( 4) represents the number of voxels whose distance from the clustering center exceeds threshold 𝜆. The meaning of D intra is the average distance that is too far from the cluster center. This can reflect the consistency between Seal voxel features in a superclass.We define intra-class consistency coefficient 𝑟 c 𝑖 to reflect the intra-class discrimination:EQUATIONThe intra-class consistency coefficient 𝑟 c 𝑖 for a superclass has the opposite trend to the average intra-class distance D 𝑖 intra . It is easier to reconstruct masked voxel features from unmasked voxels in regions with high intra-class consistency coefficient 𝑟 c 𝑖 . Therefore, we use 𝑟 c 𝑖 to modulate the base masking ratio to obtain the unified masking ratio 𝑟 𝑘 𝑖 for each superclass:EQUATION) Each superclass calculates the unified mask ratio uniquely based on its inter-class and intra-class discrimination. 
experiments 4.1 experimental settings: 
 Dataset. We demonstrate the performance of PICTURE in 3D object detection on Waymo Open Dataset [34] , 3D semantic segmentation on nuScenes [4] , and occupancy prediction on OpenOccupancy [38] . To mitigate the risk of data leakage, we perform pre-training and fine-tuning on the training set, and report the results on the val and test set. For semantic segmentation and occupancy prediction, we only report the Intersection over Union (IoU) for a subset of key objects.Model. We utilize the popular frameworks OpenPCDet [35] and MMDetection3D [8] . We fine-tune the transformer-based encoders SST, DSVT, and Cylinder3D-SST. By default, we employ 4 DSVT (with 8 attention layers) as encoders and 2 DSVT (with 4 attention layers) as decoders. The feature dimension is set to 192. Regarding the number of encoders and decoder types, please refer to the Training Details. In inter-class discrimination-guided masking, we set the expected number of superclass partition (𝑛 1 , 𝑛 2 , 𝑛 3 ) as (3, 3, 2) and the base mask ratio (𝑟 1 b , 𝑟 2 b , 𝑟 3 b ) as (0.9, 0.45, 0), meaning that some superclasses are not masked. We set the distance threshold 𝜆 in intra-class discrimination-guided masking to 0.6. For detailed ablation experiments on hyperparameters, please refer to the supplementary material. During pre-training, we employ AdamW and train for 30 epochs on Waymo and 72 epochs on nuScenes. For fine-tuning, we train for 12 epochs on Waymo and 24 epochs on nuScenes. The loss weights for reconstructing low-level and highlevel features are set to 1.0 and 3.0, respectively. All experiments are conducted on 8 NVIDIA A100-SXM4-40GB GPUs.We first compare the performance of our method with other selfsupervised learning methods in 3D object detection. Tab. 1 and Tab. 5 present the performance on the Waymo val set and test set for the leaderboard, respectively. We employ two 3D transformer blocks, SST and DSVT, as encoders. The compared low-level features include 3D point cloud coordinates (Occupancy-MAE [28] , GD-MAE [42] ), geometric features (GeoMAE [36] ), and the jigsaw puzzle (MV-JAR [40] ). Our proposed PICTURE outperforms training from scratch, leading to improvements of 1.93% and 1.69% in L2 mAP on the Waymo val set. Meanwhile, compared to the best self-supervised learning method, GD-MAE, our proposed PICTURE achieves improvements of 0.45% and 0.47%. This is attributed to the semantic consistency of high-level features and the feature attribute-related mask sampling strategy I 2 Mask. Furthermore, it can be observed that as the pre-training data increases, the performance significantly improves from 73.84% to 75.13%, opening up possibilities for leveraging large-scale unlabeled point clouds.For 3D object detection, in addition to the Waymo dataset [34] , we have also reported results on the val set of the nuScenes [4] in Tab. 2. Despite utilizing the robust DSVT as encoder, our proposed method, PICTURE, still manages to achieve improvements of 1.7% in mAP and 1.5% in NDS, respectively. Compared to using only low-level features such as 3D point cloud coordinates and jigsaw, employing high-level features for reconstruction targets results in improvements of 0.4% and 0.5% in mAP, respectively. This indicates that our proposed 3D self-supervised reconstruction target exhibits strong generalization across various autonomous driving datasets.Tab. 3 provides a comparison of 3D semantic segmentation on the nuScenes val set. We consider low-level features such as geometric features and the occupation type specifically designed for segmentation (ALSO [3] ). Our proposed PICTURE outperforms them by 0.8% and 0.6%, respectively, and exhibits greater advantages in categories like bicycle, car, and motorcycle. This effectively demonstrates the beneficial impact of high-level features on segmentation tasks that rely on semantic information.Tab. 4 provides a comparison with other self-supervised methods in occupancy prediction. Since the Seal feature is aligned with both images and text, reconstructing the Seal voxel features can enhance the performance of occupancy prediction. Our proposed PICTURE surpasses GD-MAE by 0.6%. Furthermore, our proposed I 2 Mask places more attention on road-related objects, and improvement in occupancy prediction accuracy is crucial for driving safety.Tab. 6 presents the ablation studies on the reconstruction target and mask sampling strategy. Firstly, with random masking, supplementing with Seal voxel features as a pretext task results in an This indicates that the Seal feature exhibits a significant positive impact on downstream tasks owing to its high semantic consistency. Secondly, the performance further improves when replacing the random mask with I 2 Mask. The base mask ratio 𝑟 b and intra-class consistency coefficient 𝑟 c obtained from inter-class and intra-class discrimination-guided masking further contribute to an additional improvement of 0.08% and 0.11%, respectively. When both distance metrics are jointly considered, distinct mask ratios are assigned to each superclass, resulting in an ultimate improvement of 0.39%. In addition to Seal features, we also explore alternative highlevel features that could be used for pretext tasks in Tab. 7. It can be observed that compared to low-level features, all high-level features can yield a minimum improvement of 0.37%. This indicates that using low-level features as the pretext task in 3D point cloud self-supervised representation learning is insufficient. Various highlevel features can yield benefits, not just Seal features. On the other hand, although all point cloud encoders are aligned with the semantic information of images or texts through contrastive learning, Seal features can achieve the best alignment results by locating imagepoint cloud pairs via SAM. Seal features yield the best performance, with improvements ranging from 0.43% to 0.71% compared to other high-level features. We condense these observations and analyses into a concise implementation.Comparison with training from scratch on different data scales for fine-tuning is shown in Tab. 8. On the one hand, at all data scales, the encoder pre-trained with PICTURE outperforms no pre-training. Specifically, when there is less fine-tuning data, the advantage of pre-training becomes more pronounced. For instance, when finetuning with 20% of the data, there is a 4.41% increase in L2 mAPH.On the other hand, with only 50% fine-tuning data required, the performance of DSVT w/PICURE is close to training from scratch at 100% fine-tuning data, which is beneficial for autonomous driving communities that severely lack annotated data.The time cost of our proposed method, PICTURE, compared to other self-supervised learning methods, is presented in Tab. 9. Despite the superior performance of Seal features compared to low-level features, such as geometric features [36] , it requires approximately 5× time cost (215h vs. 48h). This represents a trade-off between performance and time cost. To accelerate the pre-training, we adopt an offline approach to pre-extract Seal features for the entire dataset and save them locally. Thus, Seal features can be reused throughout pre-training. Following offline processing, the time cost for our PICTURE is approximately 55 hours, which is comparable to the time required for low-level features (55h vs. 61h, 48h).Fig. 4 shows the semantic and mask ratio distribution for a certain scene. Compared to randomly masking all non-empty voxels, our proposed I 2 Mask assigns mask ratios with inverse trends for different regions based on their reconstruction difficulty. Moreover, it can be seen areas such as vehicles and pedestrians, that are highly focused on in downstream tasks, have a high masking ratio, which forces complex reconstruction. On the contrary, the masking ratio is lower for areas such as roads and constructions, which reduces the attention during the reconstruction. 
exploratory model analysis: 
 Pretext tasks play a pivotal role in self-supervised learning. Appropriately designed pretext tasks guide the network towards the generalized features from the feature space. Further, the feature extraction may be quantitatively represented as weights. Therefore, we explore the differences between pretext tasks from the perspective of weight distribution. We select the weights of the value nodes in all attention layers of the encoder as our research subject, including the weights from multi-heads.What is the promotion of high-level features in 2D images? We employ the reconstruction of raw pixels as a low-level pretext task and the reconstruction of CLIP feature as a high-level pretext task. To simplify the analysis, we assume the weights follow a Gaussian distribution [12] and compute the mean 𝜇 2d l , 𝜇 2d h and variance 𝜎 2 2d l , 𝜎 2 2d h using maximum likelihood estimation. We assume We use differences to measure disparity without loss of generality. It can be observed that when only reconstructing low-level features, the amplitude of the differences is erratic, indicating that low-level features in 2D images are insufficient to guide the network in learning a weight distribution that is beneficial for downstream tasks. When high-level features are supplemented, the weight distribution approaches that of supervised learning.The same analysis is applied to 3D point clouds. We use DSVT [37] as the encoder. We employ the reconstruction of 3D point cloud coordinates as a low-level pretext task, and the reconstruction of Seal voxel features as a high-level pretext task. The weight distribution derived from the downstream 3D object detection task 𝜇 3d s , 𝜎 2 3d s serves as the ideal weight distribution for self-supervised learning 𝜇 3d ss , 𝜎 2 3d ss . Fig. 5(c ) and (d) visualize the disparity from the ideal weight distribution when using only low-level features and when supplementing with high-level features. It can be observed low-level features are insufficient for learning 3D representations, as there are significant disparities in mean and variance. After supplementing with Seal voxel features, the parameters advance in a direction beneficial to downstream tasks. This quantitatively indicates the necessity and feasibility of introducing high-level features in self-supervised learning for 3D point clouds. 
limitation: 
 Firstly, compared to existing methods, the improvement is not sufficiently significant. On the one hand, the capability of the highlevel feature extractor, Seal, is restricted. Due to the lack of largescale point cloud-image-text datasets, current high-level feature extractors cannot leverage the powerful understanding abilities of large language models. On the other hand, open-source point cloud datasets in autonomous driving are significantly smaller compared to internet-scale datasets, which limits the full potential of selfsupervised learning. Secondly, the framework may perform poorly in areas far from ego, where the point clouds are sparse. The masked voxels fail to reconstruct at excessively far distances due to the lack of nearby unmasked voxels. In our subsequent work, our proposed I 2 Mask can consider not only the difficulty of the sample but also the distance from the ego. Finally, the time cost of extracting Seal features is significantly higher compared to low-level features such as geometric features. We pre-extracte Seal features for all point cloud scenes in the entire dataset. Loading these features offline during pre-training allows the time cost of PICTURE to be similar to other self-supervised methods. However, the process of pre-extracting Seal features incurs a significant time cost. 
conclusion: 
 Generative self-supervised learning in 3D point clouds is trapped in low-level features. We present the necessity of introducing highlevel features from four perspectives: field development, consistency visualization with downstream tasks, quantitative experiments, and exploratory model analysis from the perspective of weight distribution. Reconstructing Seal voxel features during pretraining brings benefits to downstream tasks that require semantic information. Furthermore, in contrast to random masking, we propose inter-class and intra-class discrimination-guided masking (I 2 Mask) to adaptively set the masking ratio for each superclass, which explores the potential of the Seal voxel feature. Extensive experiments confirm our contributions to advancing generative self-supervised learning in 3D point clouds.  
 
 Appendix: k : 0.12 (b) 


Decision: All reviewers lean to acceptance of this submission. Authors please include revisions to comments from reviewers, e.g., over-claim, comprehensive literature etc

Reviews: review: paper_summary: paper_summary: This paper discusses the limitations of generative self-supervised learning in the field of 3D point clouds, particularly in autonomous driving. It highlights the challenges associated with using "low-level features" for 3D point cloud reconstruction and proposes a new approach using "high-level features" in addition to "low-level features" to improve the learning of 3D representations.
strength_weakness: strength_weakness: 1. This paper improves the basic reconstruction-based method by introducing a "high-level" loss. The proposed method uses the SEAL feature to gain more information during the pre-training process. I think such idea makes sense to me. Simply using the reconstruction method is not enough for point cloud pre-training.

2. The usefulness of the proposed method has been validated on a number of downstream tasks. There is some performance gain shown in Table 1 - Table 5, compared with GD-MAE. Also, I can see the effectiveness of inter-class / intra-class guided masking from the ablation experiments.
limitations: limitations: ### Some flaws related with the paper
> current generative self-supervised learning methods for 3D point clouds still reconstruct explicit spatial information in low-level such as 3D coordinates 

Actually, there are some papers about 3D point cloud representation learning (though not used for automatic driving scenario) that uses the "high-level" features. 
[1] Xue, Le, et al. "Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding."
[2] Yao, Yuan, et al. "3d point cloud pre-training with knowledge distillation from 2d images."

> Low-level features are defined as unimodal, simple physical properties or spatial composition information, whereas the opposite is defined as high-level features

I think the high-level features are not well-defined here. The opposite of unimodal features is clear to me (multi-modal features). However,  what is the opposite of simple physical properties / spatial composition seems vague. You should clearly define the high-level features you are referring to.

> We argue that the representation learning in 3D point clouds is equivalent to the primitive stage of 2D images

This statement should be validated. Are there experiments / some arguments pointing to this?

### Some general questions

1. Why do you think the inter-class and intra-class features are high-level features? What is their substantial difference from the low-level physical features?

2. If I understand your method correctly, you don't use class labels during pre-training. Instead you use class labels generated by unsupervised learning. Since you don't use class labels during pre-training, why do you think pre-training with "classes" works?

3. Do you think the features of Seal is good enough? From the heat map it seems that the Seal feature still has some disagreement with the ground-truth labels.

### Some minor errors:
> Why did representation learning in 3D point clouds not transition from explicit low-level features to implicit high-level features like in 2D images?

I don't understand this sentence.

> Firstly, eight superclasses that can represent autonomous driving scenarios are obtained unsupervised.

Where is secondly, thirdly etc.?
paper_summary: paper_summary: To overcome the gap in weight distribution betweeen self-supervised learning and supervised learning when only low feature employed, the authors propose a unified generative self-supervised framework. The method proposed
by the authors mainly innovates in two ways:

1.Utilizing of the high-level features: high-level features are demonstrated to exhibit semantic consistency
with downstream tasks. The authors utilize the Seal voxel features as an additional pretext task to enhance the
understanding of semantic information during pre-training.

2. inter-class and intra-class discrimination-guided masking: based on the attributes of the Seal voxel
features, the inter-class and intra-class discrimination-guided masking adaptively setting the masking ratio
for each superclass.
strength_weakness: strength_weakness: 1.Unlike previous methods that cannot Learning the universal representation of 3D point
clouds from a large amount of unlabeled data. PICTURE propose a unified framework for
generative self-supervised learning with seal features served as additional
reconstruction targets.

2.Compared to random masking, inter-class and intra-class discrimination-guided masking
proposed by the authors tightly coupled with target features and achieves significant
improvements.

3.The model achieves significant improvements compared to advance self-supervised
methods in downstream tasks.
limitations: limitations: 1. The performance is a little weaker than the previous method in comparison of Cyclist.

2. in the part of the methods, the details for the theory explanation are limited.
paper_summary: paper_summary: This paper focus on pretraining a 3D network specifically for 3D point clouds in autonomous driving. It reconstructs the semantic features using the Seal pretrained network and integrates an existing spatial reconstruction loss into MAE-based pretraining method. This method enhances the MAE-based pretraining technique by facilitating the extraction of higher-level semantic features, addressing the limitations inherent in current generative self-supervised learning approaches for 3D point clouds. Traditional methods mainly focus on reconstructing explicit low-level spatial information, such as 3D coordinates, which can be limiting. This proposed enhancement seeks to overcome these constraints by improving the network’s ability to extract features at a higher semantic level. Meanwhile, it also introduces a feature-guided mask sampling strategy, inspired by recent works, such as Focal Loss, AttMask.
strength_weakness: strength_weakness: The strengths of this paper are threefold. First, the authors have made a significant stride in learning higher-level semantic features, which effectively address the limitations of spatial-based MAE that primarily learns low-level features. Additionally, the introduction of the farthest categories sampling is a noteworthy contribution. It overcomes the inefficacies of random masking strategies, which often retain unuseful points, making it difficult to reconstruct the raw point cloud accurately. Lastly, the numerical results demonstrate a significant improvement, underscoring the efficacy of the proposed methods.
limitations: limitations: The limitations of this study can be outlined in four main points. First, the high-level features reconstruction method presented is incremental and heavily relies on the existing pre-trained method, Seal. Although utilizing additional information from Seal can enhance performance, a novel method that does not depend on pre-existing pretrained features would be more innovative and is highly anticipated. Second, the paper lacks a direct comparison with Seal, despite utilizing its features. Providing a visual comparison of the features pre-trained with Seal would significantly enhance the credibility and depth of the analysis. Third, the ablation study is incomplete as it does not present results from experiments that exclusively use feature reconstruction, omitting those that also involve coordinates reconstruction. Finally, the masking strategy also depends on supplementary information derived from the Seal pretrained network, which may limit the generalizability of the approach.
paper_summary: paper_summary: The paper titled "Point Cloud Reconstruction Is Insufficient to Learn 3D Representations' proposes a unified generative self-supervised framework for masked point modeling, which reconstructs both the raw coordinates and high-level features (extracted by Seal). Experiments on autonomous driving datasets showcase the effectiveness of designed framework and discrimination-guided masking strategy.
strength_weakness: strength_weakness: The paper is well-written, with exquisite figures and tables. 

Both qualitative and quantitative experiments are comprehensive, particularly the creative visual analysis in Figure 5.

The article candidly analyzes the limitations of the proposed method, such as time consumption and accuracy improvement.
limitations: limitations: Main concerns:

1, The improvement is marginal, which is also pointed out by the authors in the Limitation section.

2, The discrimination-guided masking strategy in the paper is non-trainable, and its hyperparameters need to be manually specified. Moreover, Tables D.2, D.3, and D.4 in the Appendix indicate that different hyperparameters have a significant impact on the results.

Detailed explanations for the observations are welcomed:

3, When using only intra-class or inter-class distance metrics, the improvement is minimal in Table 6. However, when using both types of distance metrics, the improvement becomes significantly higher. 

4, In Table 7, only when using seal features as the high-level feature can the results surpass the existing SOTA method (i.e., GD-MAE). The introduction of other high-level features does not achieve this effect.

5, In Table 8, when fine-tuning with 10% of the data, the incremental gain from pre-training is not as significant as when fine-tuning with 20% of the data. This seems somewhat counterintuitive.