Point-GCC: Universal Self-supervised 3D Scene Pre-training via Geometry-Color Contrast

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:58:30

Abstract: Geometry and color information provided by the point clouds are both crucial for 3D scene understanding. Two pieces of information characterize the different aspects of point clouds, but existing methods lack an elaborate design for the discrimination and relevance. Hence we explore a 3D self-supervised paradigm that can better utilize the relations of point cloud information. Specifically, we propose a universal 3D scene pre-training framework via Geometry-Color Contrast (Point-GCC), which aligns geometry and color information using a Siamese network. To take care of actual application tasks, we design (i) hierarchical supervision with point-level contrast and reconstruct and object-level contrast based on the novel deep clustering module to close the gap between pre-training and downstream tasks; (ii) architecture-agnostic backbone to adapt for various downstream models. Benefiting from the object-level representation associated with downstream tasks, Point-GCC can directly evaluate model performance and the result demonstrates the effectiveness of our methods. Transfer learning results on a wide range of tasks also show consistent improvements across all datasets. e.g., new state-of-the-art object detection results on SUN RGB-D and S3DIS datasets. Codes will be released on Github.

Main paper: 
 introduction: 
 3D Self-supervised learning (SSL) has received abundant attention recently because of remarkable improvement on various downstream tasks. 3D scene datasets are tiny compared to the 2D field because 3D point cloud labeling is time-consuming and laborintensive, which dramatically impedes the improvements of supervised methods. Hence many works [23, 39, 55, 57, 67, 69] explore pre-training models out of 3D labeled data to transfer knowledge for downstream tasks. The goal of self-supervised learning can be summarized as learning rich representations from unlabeled data and helping to improve performance on downstream tasks with labeled data. Most existing works follow the paradigm in the previousFigure 1 : The visualization of unsupervised semantic segmentation results. For better visualization, we use the Hungarian matching alignment to project the pseudo-labels to groundtruth labels.2D field, such as contrastive learning [26, 27, 57, 70] and masked autoencoder (MAE) [36, 39, 63, 68] . After standing on the shoulders of giants in the 2D field, we could further see the particularity of 3D representation learning as follows:• Unique information. 3D scene point cloud contains various information such as geometry and color, which makes 3D point cloud data different from 2D image data. Most existing methods [26, 57, 70] treat all information of each point as an entirety in model architecture design. We argue that directly concatenating all information can not adapt the model to discriminately learn different aspects of point clouds. Although some works [52, 62] propose the two-stream architecture that encodes point cloud by 3D network and images by 2D network, it needs extra 2D data, and 3D network can not clearly learn the discrimination between different information. Considering these additional differences may be beneficial for effective representation learning. • Mismatch between pretraining and downstream tasks.Previous pre-training works [26, 36, 57, 68] design their selfsupervised point-level tasks, such as contrast and reconstructing between specific points. However, 3D scene downstream tasks mostly focus the object representations such as object detection and instance segmentation. The gap in supervision level between pre-training and downstream tasks may hinder the improvements of 3D self-supervised learning.• Architecture diversity. The 3D point cloud field has grown rapidly in recent years [16, 33, 38, 42, 47] , and the popular architecture appears changeable and specific for downstream tasks. Hence a universal pre-training framework is important that can implement various existing methods for all kinds of tasks and is easy to adapt for future architecture.To mitigate the aforementioned problems, we explore a 3D selfsupervised paradigm that can better utilize the relations of point cloud information. Most 3D scene datasets [2, 19, 21, 49, 72] provide geometry and color information, representing different aspects of the point cloud. Geometry information describes the outline of objects and can easily distinguish between them, while color information refines the internal characteristics of objects and gives a more accurate view of each object. What's more, different information has inherent relevance. For instance, we can roughly infer the geometric structure of the object from a color photo and vice versa. Motivated by the difference and relevance inherent in the information, we propose a self-supervised 3D scene pre-training framework via Geometry-Color Contrast (Point-GCC), which uses a Siamese network to extract representations and implements elaborate hierarchical supervision. To bridge the gap between pre-training and downstream tasks, the hierarchical supervision contains point-level supervision that aims to align point-wise features and object-level supervision based on a novel deep clustering module to provide better object-level representations strongly associated with downstream tasks. Additionally, the universal Siamese network is designed as an architecture-agnostic backbone so that various downstream models can easily be adapted in a plug-and-play way.In extensive experiments, we directly perform a fully unsupervised semantic segmentation task without fine-tuning to evaluate the quality of the pre-training model. The result outperforms the previous method with +7.8% mIoU on ScanNetV2, which proves that Point-GCC has learned rich object representations through our paradigm. Furthermore, we choose a broad downstream task to demonstrate our generality: object detection, semantic segmentation and instance segmentation on ScanNetV2 [19] , SUN RGB-D [49] and S3DIS [2] datasets. Remarkably, our results indicate general improvements across all tasks and datasets. For example, we achieves new state-of-the-art results with 69.7% AP 25 , 54.0% AP 50 on SUN RGB-D and 75.1% AP 25 , 56.7% AP 50 on S3DIS datasets. Compared with previous pre-training methods, our method achieves higher AP 50 by +3.1% on ScanNetV2 and +1.1% on SUN RGB-D. Our contributions can be summarized as follows:• We propose a new universal self-supervised 3D scene pre-training framework, called Point-GCC, which aligns geometry and color information via a Siamese network with hierarchical supervision. To the best of our knowledge, this is the first study to explore the alignment between geometry and color information of point cloud via the pre-training approach.  
related work 2.1 3d scene understanding: 
 Most 3D scene understanding works are still specially designed for downstream tasks, such as object detection [33, 35, 37, 44, 47, 48, 51, 52] , semantic segmentation [11, 31, 38, 45, 46, 53, 66] , and instance segmentation [13, 25, 29, 30, 50] . The model architecture can be summarized as a backbone module extracting the features of point clouds, and a downstream head adapting for the special task.According to the processing method, these works can be roughly divided into two categories: point-based methods and voxel-based methods. Point-based methods [11, 33, 37, 54, 71] are widely used in point clouds thanks to the effectiveness of PointNet++ [38] , which alternately use farthest point algorithm and multi-layer perceptron to sample and extract the features of point. Voxel-based methods [13, 29, 30, 47, 48, 50] is recently popular because of the better performance and efficiency on many downstream tasks than pointbased methods, which operate 3D sparse convolution on regular voxels transformed from irregular point clouds. We pre-train on both point-based PointNet++ and voxel-based 3D sparse convolution backbone and fine-tune on multiple downstream methods to give a comprehensive view of our work.Compared to 2D vision or natural language, 3D vision has a more serious problem of data scarcity [23] which limits the downstream performance of 3D tasks. To solve the raising problem, 3D selfsupervised learning (SSL) [1, 28, 55, 64, 69] has gotten more attention in recent years. The mainstream SSL methods can be roughly divided into two categories: contrastive learning and reconstructive learning. Contrastive learning is motivated to learn the invariant representation from different paired carriers such as view augmentation [14, 15, 57] or different data formats [43, 60] . Reconstructive learning is designed to reconstruct the disturbed data to learn geometry knowledge between patches [6, 22] . Motivated by the success of masked autoencoder in 2D [24, 59] , the MAE-style selfsupervised method became popular in point cloud [36, 41, 61, 68] .Recently, some works find that the pattern difference between the two methods in attention area [58] and scaling performance [39, 40] .Based on previous work, we consider the color and geometry of scene point clouds as two views for contrastive learning, and use a swapped reconstruct strategy for reconstructive learning. Therefore, Point-GCC achieves the integration of two methods and derives benefits from both of them.Deep Clustering [5, 7, 9, 12, 34, 56, 65] aims to learn better features and discover data labels using deep neural networks, which has been broadly applied in self-supervised and semi-supervised learning. DeepCluster [8] uses the off-the-shelf K-means algorithm pseudo-labels as supervision which learns comparative representations for self-supervised learning. SeLa [4] proposes a simultaneous clustering and representation learning method using the Sinkhorn-Knopp algorithm to generate pseudo-labels with equal partitions quickly. SwAV [10] combines contrastive learning and deep clustering, which enforces consistency between cluster assignments from different views of the same image. In this work, we attempt to apply deep clustering in 3D self-supervised learning field, which generates pseudo-labels based on the inherent feature consistency of the geometry and color information of the point cloud. 
point-gcc: pre-training via geometry-color contrast: 
 Existing methods mainly focus on geometric information, but our goal is to enhance the 3D representation capability by better utilizing all the information discriminately in scene point clouds. Therefore, a novel Geometry-Color Contrast method is proposed to address this motivation. Figure 2 illustrates the overall framework of Point-GCC. We first perform a Siamese backbone to extract the features of the geometry and color information respectively in Section 3.1. To carefully align the features belonging to different information, we propose the point-level supervision via combining the contrastive and reconstructive learning in Section 3.2, then we design an unsupervised deep clustering module to generate object pseudo-labels and perform object-level contrastive learning between high-confidence object samples in Section 3.3. The final hierarchical supervision is described in Section 3.4. In Section 3.5, we propose a new method directly evaluating the pre-training model on unsupervised semantic segmentation to demonstrate the effectiveness of our method.Information split and embedding. In 3D scene datasets, a point 𝒑 is usually associated geometry information represented by the coordinates 𝒑 𝑔𝑒𝑜 and color information represented by RGB value 𝒑 𝑐𝑜𝑙𝑜𝑟 . Different from previous pre-training methods regarding a single point as an atom unit, we split the point cloud into two parts, the geometry and color respectively. Then we project them to universal embedding space 𝒆 by Equation 1. Additionally, to distinguish similar colors in different coord, we add an extra weakly positional embedding 𝒆 𝑝𝑜𝑠 to the color embedding with the Euclidean norm of coord. Positional embedding helps the network distinguish different objects that share the same color and improves generalization in various scenarios. Note that we remove all embedding modules in fine-tuning stage to keep our framework plug-and-play in order that more existing methods can benefit from ours.Siamese architecture-agnostic backbone. We use a symmetric Siamese network F (•) to separately encode geometry features 𝒇 𝑔𝑒𝑜 and color features 𝒇 𝑐𝑜𝑙𝑜𝑟 . Since we attempt to help more existing architectures learn better representations from the combination of geometry and color information, we do not modify any backbone architecture. So that we can directly reuse the core module for standard segmentation with any backbone architecture. In other words, the backbone encodes input 𝒙 ∈ 𝑅 𝑁 ×𝐶 1 and extracts feature 𝒚 ∈ 𝑅 𝑁 ×𝐶 2 . To align the two information, Siamese backbone F (•) encodes the geometry embedding 𝒆 𝑔𝑒𝑜 and color embedding 𝒆 𝑐𝑜𝑙𝑜𝑟 with weakly positional embedding 𝒆 𝑝𝑜𝑠 to geometry features 𝒇 𝑔𝑒𝑜 and color features 𝒇 𝑐𝑜𝑙𝑜𝑟 respectively:EQUATIONEQUATIONwhere E is corresponding linear layer of each embedding, F (•) is the Siamese network.In our opinion, the Siamese backbone network is forced to apply the discriminative distributions and learn the relation and distinction of different attributions from the same points. The learned knowledge is critical for transferring to all discriminative inputs concatenated situations in most downstream tasks.Inspired by the success of associating contrastive learning and reconstructive learning in recent work [39] , We propose our pointlevel supervision elaborately designed for our Siamese architecture, which first contrasts and then swapped reconstruct the features to benefit from different paradigms. Contrastive learning. The geometry features 𝒇 𝑔𝑒𝑜 and color feature 𝒇 𝑐𝑜𝑙𝑜𝑟 are point-wise aligned because they are split from the same point cloud 𝒑 and extracted by the Siamese segmentationstyle backbone network. We apply the InfoNCE loss aiming to pull positive pairs close, and push negative pairs away across the geometry features and color features:EQUATION)where 𝜏 is the temperature hyper-parameter, we follow the previous works [57] to set it as 0. except 𝒛 𝑖 𝑐𝑜𝑙𝑜𝑟 represent negative pairs. Reconstructive learning. Based on our Siamese architecture, we apply the reconstructive learning by swapped reconstruct strategy instead of mask strategy, which solves the raising problem about the distribution mismatch between training and testing data in masked autoencoding for point cloud [32] . Specifically, we simply project the geometry features 𝒇 𝑔𝑒𝑜 and color features 𝒇 𝑐𝑜𝑙𝑜𝑟 to reconstruct color p𝑔𝑒𝑜 and geometry p𝑐𝑜𝑙𝑜𝑟 . The reconstructive loss is the mean squared error (MSE) between the reconstructed and original information of each point:EQUATION)where 𝑁 is the number of points, p𝑖 𝑔𝑒𝑜 and p𝑖 𝑐𝑜𝑙𝑜𝑟 represent the reconstruct prediction, 𝒑 𝑖′ 𝑔𝑒𝑜 and 𝒑 𝑖′ 𝑐𝑜𝑙𝑜𝑟 represent the reconstruct targets which both scale to between 0 and 1 for stability training loss.Point-level supervision is widely applied in 3D self-supervised learning, which provides rich representations for downstream tasks. However, the object representation strongly associated with downstream tasks hasn't been noticed before. We propose our objectlevel supervision driven by the novel unsupervised deep clustering module. The clustering module generates pseudo-label predictions P 𝑔𝑒𝑜 and P 𝑐𝑜𝑙𝑜𝑟 for the geometry features 𝒇 𝑔𝑒𝑜 and color features 𝒇 𝑐𝑜𝑙𝑜𝑟 respectively, and enforces consistent prediction between geometry prediction P 𝑔𝑒𝑜 and color prediction P 𝑐𝑜𝑙𝑜𝑟 of same point 𝒑. We argue that the pseudo-labels represent more various object features, which are not restricted by human annotations with fixed object classes. To achieve robust supervision among these objectlevel pseudo labels, we sample the high-confidence object features based on the prediction confidence score and apply object-level contrastive learning according to pseudo labels. Deep clustering via swapped prediction. We apply the swapped prediction [10] in 2D contrastive learning to our model, which predicts the pseudo label of an image from the clustering result of another view. In our framework, we swap the cluster target of different information features, and predict the pseudo label from the other information feature based on the inherent consistency of the two types of information as shown in Figure 3 (a). For pseudo label classes 𝐾, we use a learnable matrix C = [𝒄 1 , • • • , 𝒄 𝐾 ] to represent the cluster centroids, and calculate the similarity S between the ℓ 2 -normalized features 𝒇 and cluster centroids 𝒄. To avoid the degeneration problem that all features collapse into the same prediction, the Sinkhorn-Knopp algorithm [18] is used to generate the equal partition cluster distribution Q from the similarity S by converting pseudo-label generation to an optimal transport problem. And the learnable prediction P is computed by 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (S/𝜏), where 𝜏 is the temperature hyper-parameter. We set all hyper-parameter in swapped prediction same to the previous works [10] in 2D. Finally, The swapped prediction loss is the cross entropy losses between the learnable prediction P and swapped equal partition distribution Q: where ℓ is the cross-entropy loss between the prediction and target.EQUATIONObject-level contrastive learning. For the features 𝒇 with corresponding pseudo prediction P and confidence score from deep clustering, we pick features with confidence scores higher than the picking threshold to alleviate the noise from unsupervised clustering. Then we compute the mean features of high-confidence samples from geometry and color branches, respectively. We take the two types of mean features with the same pseudo-label as positive pairs, oppositely with different pseudo-label as negative pairs, and apply the InfoNCE loss at object-level:EQUATIONwhere 𝜏 is the temperature hyper-parameter, we set it to 0.4 following the above-mentioned setting. 𝒛 𝑖 is the ℓ 2 -normalized mean feature with pseudo-label 𝑖. 𝒛 𝑖 𝑔𝑒𝑜 and 𝒛 𝑖 𝑐𝑜𝑙𝑜𝑟 represent a pair of positive sample with same pseudo-label 𝑖. And 𝒛 𝑖 𝑔𝑒𝑜 with 𝒛 𝑗 𝑐𝑜𝑙𝑜𝑟 corresponding different pseudo-label 𝑗 represent negative samples. We believe the pseudo-labels represent various object features that are not restricted by human annotations with fixed object classes and are significant for learning rich representations.Our framework contains hierarchical supervision at point-level and object-level, and the final loss is a combination of the four losses above-mentioned:EQUATIONwhere 𝛼, 𝛽 and 𝛾 are the loss weight hyper-parameters, we set them to 100, 100 and 1 respectively to balance the magnitude of losses.Due to the pseudo-label from object-level supervision, Point-GCC can adapt to unsupervised downstream tasks without fine-tuning. Meanwhile, previous pre-training methods evaluate the performance by transfer learning on downstream tasks. The results can be greatly affected by the fine-tuning setting and are not intuitive between different baselines. As shown in Figure 3 (b), we generate the final pseudo-labels by utilizing cluster prediction from geometry and color branch. During the evaluation stage, we use the Hungarian matching alignment [65] to project the pseudo-labels to ground-truth labels because we are agnostic to the ground truth in pre-training. Although our method is not specifically designed for unsupervised downstream tasks, we find that the process is more intuitive and fair for evaluating the performance of pre-training methods. 
experiments: 
 To analyze the 3D representation learned by Point-GCC, we conduct extensive experiments on multiple datasets and tasks described in Section 4.1. First, we evaluate fully unsupervised semantic segmentation tasks to validate the effectiveness of object representation in Section 4.2. Then we expand experiments by transfer learning on multiple downstream tasks and datasets in Section 4.3.Dataset. We use three popular indoor scene datasets: ScanNetV2 [19] , SUN RGB-D [49] , S3DIS [2] in our experiments. ScanNetV2 is a 3D reconstruction dataset, which provides 1513 indoor scans with a total of 20 classes. SUN RGB-D is a monocular RGB-D image dataset, which provides 10335 RGB-D images from four different sensors with a total of 37 classes. S3DIS is another 3D indoor scene dataset, which provides 271 point cloud scenes across 6 areas with 13 classes. Implementation details. We implement Point-GCC built upon the MMDetection3D [17] framework. We use the AdamW optimizer with an initial learning rate of 0.001 and weight decay of 0.0001. Other implementation details follow the default scheme. To ensure fair comparability of results, we refer to selecting downstream models implemented by MMDetection3D. In downstream task experiments, we decay the learning rate by 0.5, and other settings follow the original implementation. The full detail settings are provided in the Appendix. We evaluate our pre-training model on fully unsupervised semantic segmentation tasks using the method in Section 3.5 to validate the effectiveness of object representation. As shown in Table 3 , our method surpasses previous unsupervised methods by a huge margin and is closer to the weakly-supervised method, despite Point-GCC being not specifically designed for unsupervised downstream tasks.With the same backbone PointNet++, Point-GCC surpasses previous work SL3D [11] by +9.8% mIoU, and +7.8% mIoU compared with more powerful Point Transformer on ScanNetV2 dataset. The result proves that Point-GCC has learned rich object representation in unsupervised pre-training. Fine-tuning semantic segmentation. Additionally, we fine-tune the pre-training model for semantic segmentation to verify the consistent improvement of our method. With supervised fine-tuning, the model gains significant improvements by +5.4% mIoU on the ScanNetV2 dataset, which proves that our method has learned intrinsic representations of the point cloud.3D Object detection. For 3D object detection task, we pre-train the PointNet++ [38] backbone for VoteNet [37] , VoteNet+FF [48] and GroupFree-3D [33] and the MinkResNet [16] backbone for TR3D [48] , TR3D+FF [48] respectively. Table 1 and 2 shows the results on ScanNetV2, SUN-RGBD, and S3DIS datasets. Our method gains stable and significant improvements for various settings. Compared with previous 3D self-supervised methods with the common baseline model VoteNet, our method achieves higher AP 50 than the previous highest model Ponder [27] by +3.1% on ScanNetV2 and +1.1% on SUN RGB-D. For more recent models, our model also significantly boosts VoteNet+FF, GroupFree-3D, TR3D, TR3D+FF on multiple datasets and achieves new state-of-the-art results with 69.7% AP 25 , 54.0% AP 50 on SUN RGB-D and 75.1% AP 25 , 56.7% AP 50 on S3DIS datasets. 3D Instance segmentation. For 3D instance segmentation task, we pre-train the MinkResNet backbone for TD3D [30] on ScanNet Interestingly, the improvements for the PointNet++ [38] backbone widely surpass the MinkResNet [16] backbone. We guess that sparse convolution architecture implicitly aligns the color information from features and the geometry information from fine-grained sparse voxel operation. It may be a kind of explanation for why 3D sparse convolution has better performance and efficiency on various tasks. Additional comparison Some baseline models [33, 37] do not use color information. Table 5 shows the additional baseline model with color information for a fair comparison from our reproduction and other works [27, 52] , which get slight improvement and even decrease. The results prove what we mentioned in the main paper: directly concatenating all information can not adapt the model to learn different aspects of point clouds discriminately, demonstrating our work's necessity.To analyze the effectiveness of our approach, we further explore additional experiments to measure the contribution of each component to the final representation quality. For efficiency, all ablation experiments are implemented with VoteNet setting on pre-training and object detection. Hierarchical supervision. To further explore the improvement of our hierarchical supervision, we conduct ablation studies with different components. Table 6 shows the unsupervised semantic segmentation results with pre-training and object detection results with fine-tuning. The results show that both contrastive learning and reconstructive learning at point-level contribute to the final results. Even though just with point-level supervision, our method has achieved higher AP 25 and AP 50 than the previous best model Ponder by +1.2% and +2.0%. Furthermore, the swapped prediction and object-level contrastive learning also provide remarkable improvements for AP 50 and AP 25 , especially AP 50 . Intuitively, the improvement of AP 50 is more significant than AP 25 , demonstrating that object-level supervision improves the model with a more precise view of objects.Weakly positional embedding The color information is vague for precise geometry reconstruction, and some backbone networks are translational equivariant and hard to predict a global coordinate only with color information embedding, i.e., some different objects partly share the same color can not be distinguished. The weakly positional embedding is designed to add to color information embedding to provide blurry global position information. Positional embedding provides implicit global information to help reconstruct precise geometry from vague color information, which balances the task difficulty between two branches. It also helps the network distinguish different objects that share the same color and improves generalization among various scenarios.We conduct an additional ablation study with different settings to explore the improvement of positional embedding further. Table 7 shows the object detection results with different positional embedding. no pos means without positional embedding. xy pos means with the positional embedding of x and y, and the corresponding task aims only to reconstruct z, i.e., height estimation task. The results show that our model has stable effects under different position encoding conditions. This may be because we remove the positional embedding and adjust the input channel in the downstream fine-tuning stage so that the impact on the downstream task is reduced. Geometry-Color Contrast. To verify the importance of Geometry-Color Contrast approach, we compare the results with a single reconstruction branch setting. Table 8 shows object detection results with different pre-training branches. The results show that the performance with a single branch of whether geometry or color reconstruction obviously declines, which proves our Geometry-Color Contrast plays an essential role in the significant performance. Object sampling strategy. The result in table 6 shows that objectlevel supervision provides the most obvious boost for AP 50 . We compare the results with different object sampling strategies to analyze the object samples used in object-level contrastive learning. The results in table 9 show that the more confident object samples are, the greater performance we achieve. However, the performance decays only using the maximum score sample because of overfitting. Unsupervised semantic segmentation visualization We provide visualization results of unsupervised semantic segmentation.Figure 1 shows that our method can clearly distinguish the main parts of different objects without supervision. However, for small or complex objects, the segment may be merged into others or ignored because of the equal partition cluster distribution from the Sinkhorn-Knopp algorithm [18] . The result demonstrates that our pre-training approach helps the model learn object representations to enhance performance on downstream tasks. Object sampling strategy visualization From the following experiments, we can find that the quality of points used for objectlevel contrastive learning is important for performance. We believe that the quality of object-level features depends on the trade-off between quantity and quality of point-level features. For a more intuitive perspective, we visualize points used after different thresholds in Figure 5 . Shrinking the threshold helps retrieve more accurate points that belong to the same semantic class, but the lack of quantity makes it difficult to reveal the essential object-level feature. However, simply relaxing the threshold will introduce more noise that belongs to different semantic classes, inevitably degrading object-level feature quality.Geometry and color reconstruction visualization Figure 4 shows the visualization of geometry and color reconstruction results. The results show that our method can consistently generate high-quality complements from one type of information in the point cloud. The method may contain potential applications such as depth estimation and texture generation.  
future work: 
 As mentioned above, most of the limitations come from our compromise to general adaptation. Global coordinates make it easy for our work to adapt to different model architectures and datasets but may also cause the distribution mismatch problem. The preferred aim of this work is to help improve various existing downstream methods.Our future exploration will consider more elaborate architectural designs for better point cloud understanding and performance, such as geometry and color information embedding and pre-training tasks. 
conclusions: 
 In this paper, we propose a new universal self-supervised 3D scene pre-training framework via Geometry-Color Contrast (Point-GCC), which utilizes an architecture-agnostic Siamese network with hierarchical supervision. Extensive experiments show that Point-GCC significantly improves performance on unsupervised tasks without fine-tuning and a wide range of downstream tasks, especially achieving new state-of-the-art results on multiple datasets.To the best of our knowledge, Point-GCC is the first study to explore the self-supervised paradigm that can better utilize the relations of different point cloud information, hence we elaborately design our plug-and-play pre-training framework to help improve various existing downstream methods, instead of directly designing a new architecture. We hope our work could attract more attention to the discriminative information of point clouds, which may inspire future point cloud representation learning works.  
 
 Appendix:  


Decision: This paper was reviewed by four experts in the field. The recommendations are (Borderline Accept x 4). Based on the reviewers' feedback, the decision is to recommend the acceptance of the paper. The reviewers did raise some valuable concerns (especially more detailed ablations experiments, numerical results for model efficiency and computational cost, more comparisons with related works, remove of overclaim statements, etc.) that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a self-supervised 3D scene pre-training framework via Geometry-Color Contrast (Point-GCC), which utilizes an architecture-agnostic Siamese network with hierarchical supervision.
strength_weakness: strength_weakness: The authors design a deep clustering module to generate object pseudo-labels based on the inherent feature consistency of the two pieces of information.
limitations: limitations: 1) The motivation for this work is not well discussed in both the abstract and introduction sections. What's more, The logic presented in the introduction is difficult to follow.
2) Several baseline methods selected for comparison are outdated and may not reflect recent advancements in the field.
3) The ablation study of the object sampling strategy is confusing. 
4) The layout of the paper appears disorderly and could benefit from better organization and visual presentation.
paper_summary: paper_summary: The paper proposes a novel self-supervised learning framework designed to improve 3D scene understanding by leveraging both geometric and color information in point clouds. The authors introduce a Siamese network that aligns these two types of data through hierarchical supervision, including point-level contrastive and reconstructive learning as well as object-level contrastive learning facilitated by a deep clustering module. The approach is architecture-agnostic, enabling it to adapt to various downstream models. The experimental results demonstrate significant improvements across multiple datasets and tasks, such as object detection and semantic segmentation, achieving new state-of-the-art performances.
strength_weakness: strength_weakness: 1.The introduction of a Siamese network to align geometric and color information is a novel approach that effectively utilizes the unique characteristics of 3D point cloud data.

2.The combination of point-level and object-level supervision ensures comprehensive learning of features, addressing the gap between pre-training and downstream tasks.
limitations: limitations: 1.The research content of this article is limited to indoor scenes and does not mention outdoor autonomous driving scenes, so naming it "3D Scene Pre training" is somewhat inappropriate.

2.In Point level Supervision, using the color and geometric features of each point as the reconstruction target, without fusing the features of other points, can this method be successfully trained?

3.The approach assumes the availability of both geometric and color information in point cloud datasets, which may not always be the case in real-world scenarios.

4.While the paper reports improvements in performance, it lacks a detailed analysis of the computational cost and efficiency of the proposed methods compared to baseline models.
paper_summary: paper_summary: The paper introduces a novel 3D self-supervised learning framework for pre-training on 3D scene understanding tasks. The authors propose Point-GCC, which utilizes a Siamese network to align geometry and color information from point clouds. The framework includes hierarchical supervision with both point-level and object-level contrast, and it is designed to be architecture-agnostic to accommodate various downstream models. Experiments demonstrate that Point-GCC achieves significant improvements on a range of tasks across different datasets, including new state-of-the-art results for object detection on SUN RGB-D and S3DIS datasets.
strength_weakness: strength_weakness: 1. Innovative Approach: The paper presents a unique paradigm for 3D self-supervised learning by contrasting geometry and color information, which is a novel contribution to the field.
2. Architecture-Agnostic Design: The proposed framework is designed to be adaptable to various downstream models, making it a versatile tool for different applications.
3. Hierarchical Supervision: The use of both point-level and object-level supervision provides a more nuanced approach to learning from unlabeled data.
4. State-of-the-Art Results: The framework demonstrates excellent performance, achieving new state-of-the-art results on standard benchmarks.
5. Extensive Evaluation: The authors have conducted a thorough set of experiments, including unsupervised tasks and transfer learning to downstream tasks, which substantiates the effectiveness of the framework.
limitations: limitations: 1. Complexity: The framework's design might introduce additional complexity compared to simpler models, which could potentially impact its scalability.
2. Generalization to Other Domains: While the paper shows strong results for indoor scenes, it's unclear how well the framework generalizes to other domains like outdoor scenes or different types of 3D data.
3. Computational Resources: The paper does not discuss the computational requirements or runtime, which could be a concern for researchers with limited resources.
4. Explainability and Transparency: The paper could benefit from a deeper analysis of what the model learns during the pre-training phase and how this knowledge transfers to downstream tasks.
paper_summary: paper_summary: This paper introduces a novel self-supervised 3D scene pre-training framework. It aligns geometry and color information using a Siamese network with point-level and object-level supervisions. Additionally, it proposes a deep clustering module to generate object pseudo-labels. The effectiveness of the framework is demonstrated through experiments on three datasets and three different downstream tasks.
strength_weakness: strength_weakness: 1. The paper is written clearly, making it easy for the reviewer to understand.
2. The experiments in Table 1 are impressive, as they compare the latest methods and test various backbones.
3. The ablation study is clear and effectively presents the contribution of each component.
limitations: limitations: 1. In the experiments outlined in Section 4.2, the comparisons appear to be insufficient. Specifically, for the unsupervised semantic segmentation task on ScanNet V2, the authors only compared with SL3D, which is not considered state-of-the-art. For instance, GrowSP [1] achieves an mIoU of 25.4 in this task, significantly higher than the reported result. Similarly, for the fully supervised fine-tuning task, no comparison with state-of-the-art methods was provided. It is recommended to include at least one state-of-the-art method for a fair comparison to demonstrate the effectiveness of this method.
2. Regarding the instance segmentation task discussed in Section 4.3, it is encouraged to include a comparison with at least one state-of-the-art method.
3. There are concerns about the fairness of the comparison to Ponder on the 3D detection task in Table 1. The VoteNet used in this paper achieves a much higher AP than the officially reported results. Is this higher performance due to a significantly higher baseline?

Ref:
[1] Z. Zhang, B. Yang, B. Wang, 和 B. Li，“Growsp: Unsupervised semantic segmentation of 3d point clouds,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 17619–17629.